# Self-hosted AI Stack Demo
This repo contains an example deployment of an AI stack using a Docker Compose 
file. 

The stack is designed to be self-hosted and can be deployed on a single machine.

TODO: Make the Dockerfile suitable for deployment via ECR.

## Stack Architecture
![Stack Architecture](./docs/OpenWebUI%20Stack.png)

## Basic Components
These are the core components of the stack
- **Ollama**: A tool for managing and deploying machine learning models locally.
- **Open-WebUI**: A web-based user interface for interacting with AI models and services.

## WIP Proven Components
These are components proven deployed in on-prem, bare metal POCs.
- **Jupyter Server**: To create and share documents that contain live code, 
equations, visualizations, and formatted text.
- **ComfyUI**: A user-friendly interface for configuring and deploying AI image models.
- **nginx**: A web server that can be used to serve the web interfaces of the above services.
- **LightRag**: A framework for rapid development and deployment of RAG applications.

## WIP New Components
These are components that are currently in development and are not yet proven.
- **LiteLLM**: A lightweight library for federating self-hosted and commercial large language models.
- **LangFuse**: A tool for monitoring and debugging language models.


## Usage
To deploy the stack, you will need to have the following installed on your machine:
For all machines:
- Docker (Docker Compose / Docker Desktop)

For OSX:
- Ollama - https://ollama.com/download/mac

  Ollama can be run from Docker on OSX, but as the Apple Metal GPU is [not](https://www.docker.com/llm/) [yet](https://www.youtube.com/watch?v=mk_2MIWxLI0&t=1544s) available via Docker Desktop, it is recommended to run it natively for DRASTICALLY improved performance.

For Linux/Windows:
-  Ollama can be run from Docker, or natively if you have a compatible GPU. To run Ollama from Docker with GPU, see the installation instructions at https://github.com/open-webui/open-webui/pkgs/container/open-webui and adjust the docker-compose.yaml accordingly.

## Security Considerations
This stack is designed to be deployed on a local machine for POC and development purposes.
It is not secure for production use. The following security considerations should be taken into account:

<!-- - **No Authentication**: The services in this stack do not have authentication enabled. -->
<!-- - **No HTTPS**: The services in this stack do not use HTTPS. -->
- **No Firewall**: The services in this stack are all exposed, and expected to be available via gateway.
- **No Rate Limiting**: The services in this stack do not have rate limiting enabled. 
This is especially important for Ollama, as multiple users can quickly exhaust the available resources.
- **No Monitoring**: The services in this stack do not have monitoring enabled outside of logs.
- **No OpSec**: The services in this stack are not protected against common 
    operational security threats.

Mitigation strategies for these security considerations would involve 
consultation from the Security Team and DevOps Team.

## Cost Considerations
This stack is designed to be deployed on a single machine, and is not designed 
for high availability or scalability. In the event of deploying this stack to a 
cloud provider, the following cost considerations should be taken into account:

- **Compute Needs**: The services in this stack require a machine with at least 16GB of RAM and a GPU with at least 22GB of VRAM.

- **Compute Costs**: For a large enterprise with continuous usage, long-term Reserved Instances can dramatically lower costs (up to ~70% off on-demand pricing). The table below shows per-hour rates and quarterly (3 months 24/7) costs for the specified instances in us-east-1 (N. Virginia) under a 3-year Standard Reserved Instance (No Upfront) plan. We assume ~2190 hours for 3 months of 24/7 usage.

    | Instance Type | Hourly Price (USD) | 3-Month 24/7 Cost (USD) | Pricing Model |
    |---------------|--------------------|-------------------------|---------------|
    | p3.2xlarge    | $1.132/hour        | ~$2,479 for 3 months    | 3-year Reserved (No Upfront) |
    | p3.xlarge*    | ~$0.566/hour (est.)| ~$1,240 for 3 months    | 3-year Reserved (No Upfront) |
    | g4dn.xlarge   | $0.227/hour        | ~$497 for 3 months      | 3-year Reserved (No Upfront) |

    *Note: p3.xlarge is not a standard EC2 offering (P3 instances start at p3.2xlarge). For comparison, we’ve estimated p3.xlarge costs as roughly half of p3.2xlarge’s, given it would have half the vCPU/RAM with the same GPU.

    Each price above is for a Linux OS instance in N. Virginia. All prices use a Reserved Instances 3-year term (no upfront payment) – a common choice for enterprises to get discounted hourly rates. This yields the per-hour costs shown, and the total 3-month cost is calculated by multiplying the hourly rate by 24 hours/day * 90 days (~2190 hours). The reserved pricing model used (3-Year Standard RI, No Upfront) is explicitly noted for clarity.

- **Storage, Network, and Monitoring Costs**: As we already have an AWS account 
    and are paying for S3 and have existing provisioned resources and infra on 
    AWS, we can assume the storage, network, and monitoring costs to be negligible 
    and absorbed by the overall organization AWS usage.

**Total Estimated Cost for 3 Months: $2,479 / $1,240 / $497**

### Previous Cost Analysis
Here are some example costs for running the services in this 
stack on AWS. These costs are based on the current pricing as of September 2021 and are subject to change.

All costs are assuming 24/7 operation for 3 months.
- **`p3.2xlarge`**: 16 vCPUs, 61GB RAM, NVIDIA V100 GPU (32GB VRAM)
    - ~$3.06 / hr
    - $6,642.72 / Q

- **`p3.xlarge`**: 4 vCPUs, 30.5GB RAM, NVIDIA V100 GPU (32GB VRAM)
    - ~$1.33 / hr
    - $2,877.60 / Q

- **`g4dn.xlarge`**: 4-8 vCPUs, 16GB RAM, NVIDIA T4 GPU (16GB VRAM)
    - ~$0.526 / hr
    - $1,142.88 / Q

**Total Estimated Cost for 3 Months: $6,642.72 / $2,877.60 / $1,142.88**