# Local LLM Stack — Apple Silicon

**Platform:** macOS, Apple M-series, 48 GB unified memory  
**Inference server:** oMLX 0.4.1  
**Primary model:** `gemma-4-26b-a4b-it-4bit` (vision + tool calling + thinking)  
**Status:** ✅ Working — vision, tool calling, and thinking all confirmed  
**Last validated:** 2026-06-04

---

## What This Is

A fully local, offline-capable LLM inference stack running on Apple Silicon. Handles:

- **Tool calling** — structured JSON function calls, multi-turn agent loops
- **Vision (VLM)** — base64 image input, screenshot analysis, UI review
- **Thinking** — extended reasoning chains via Gemma4's built-in reasoning
- **Gemma4-first serving** — one primary VLM with 256K context and TurboQuant KV 4-bit
- **Agent compatibility** — OpenAI-compatible API consumed by opencode, and any OpenAI-SDK client

---

## Architecture

```
Agent harness (opencode / any OpenAI client)
        │  OpenAI-compat streaming  (:8090)
        ▼
  omlx-proxy.py                     ← thin aiohttp shim
        │  routes reasoning_content → content in stream deltas
        │  forwards to oMLX         (:8080)
        ▼
  oMLX 0.4.1 (multi-model server, --memory-guard-gb 30)
    └── gemma-4-26b-a4b-it-4bit     ← VLMBatchedEngine, TurboQuant KV 4-bit
                                       vision + tools + thinking, 256K context
```

The proxy sits between the agent and oMLX to handle one compatibility fix:
- Routes `reasoning_content → content` in streaming deltas
- Forwards all other traffic to oMLX unchanged
- Provides a stable endpoint even when oMLX restarts

---

## Quick Start

```bash
# Install oMLX — pin to 0.4.1 (VLM patches are version-specific)
brew tap jundot/omlx && brew install jundot/omlx/omlx && brew pin omlx

# Install proxy deps
pip3 install aiohttp

# CLI ask wrapper (pipe-friendly, no deps)
# Usage: ./scripts/llm-ask.sh "prompt" or echo "prompt" | ./scripts/llm-ask.sh
# Web UI: http://localhost:8080/admin/chat

# Download model
mkdir -p ~/.llm/models && cd ~/.llm/models
huggingface-cli download mlx-community/gemma-4-26b-a4b-it-4bit \
  --local-dir gemma-4-26b-a4b-it-4bit

# Apply oMLX patches (see patches/ dir)
# Apply Gemma4 config fix (see docs/gemma4-vlm-investigation.md)
# Generate local config and helper files
bash scripts/setup.sh

# Start the stack
bash scripts/llm-start.sh
```

---

## Directory Layout

```
local-llm/
├── README.md                        ← this file
├── docs/
│   ├── setup.md                     ← full setup guide, bare-metal recreation
│   ├── gemma4-vlm-investigation.md  ← root-cause analysis and all fixes applied
│   └── advanced-omlx-settings.md    ← source-audited config reference + benchmark results
├── scripts/
│   ├── llm-start.sh                 ← start oMLX + proxy
│   ├── llm-stop.sh                  ← graceful shutdown
│   ├── llm-restart.sh               ← stop + start
│   ├── llm-status.sh                ← health check, memory, active models
│   ├── llm-keepwarm.sh              ← ping loop to prevent model eviction
│   ├── llm-sync.sh                  ← drift-check and sync scripts between working copy and collab
│   ├── llm-rotate-model.sh          ← hot-swap primary model
│   ├── llm-cache-watchdog.sh        ← monitor and prune SSD KV cache
│   ├── browser-preflight.sh         ← preflight check before browser agent tasks
│   ├── setup.sh                     ← one-shot install: patches, model config, oMLX settings, opencode.json
│   ├── omlx-proxy.py                ← compatibility proxy (aiohttp)
│   └── llm.conf                     ← shared config (ports, paths, model names)
└── patches/
    ├── omlx-model-discovery.patch   ← Gemma4 VLM detection fix
    └── gemma4-config.json           ← corrected vision_config for local quantized copy
```

---

## Model Selection

| Model | Role | Disk | Notes |
|-------|------|------|-------|
| `gemma-4-26b-a4b-it-4bit` | Primary — VLM | ~15 GB | vision ✅ tools ✅ thinking ✅ 256K ctx |

## Performance

Benchmarked on M4 Pro 48 GB, oMLX 0.4.1, TurboQuant KV 4-bit, **powermode=0** (macOS normal power). All tests use correct streaming measurement: TTFT separated from generation speed.

> ⚠️ macOS Low Power Mode cuts throughput ~50% across all context sizes. Always verify `pmset -g | grep powermode` shows `0` before benchmarking or running agentic workloads.

| Test | Context | TTFT | Gen tok/s |
|------|---------|------|-----------|
| Short text | ~28 tok | 0.4 s | **65.5** |
| Medium text | ~3K tok | 2.4 s | **51.8** |
| Medium text | ~11K tok | 8.6 s | **55.8** |
| Tool calling | ~200 tok | 1.0 s | ✅ `finish=tool_calls` |
| Tool response | ~400 tok | 0.2 s | **58.8** |
| Vision (PNG) | ~300 tok | 1.2 s | **52.3** |
| 32K context | 32K tok | 30.6 s | **47.4** |
| 64K context | 64K tok | 85.2 s | **41.2** |
| 96K context | 96K tok | 161 s | **35.2** |
| 128K context | 128K tok | 276 s | **31.2** |

All 10/10 tests pass the ≥20 tok/s generation floor. Model footprint: 15.3 GB weights (pinned), 30 GB hard ceiling including KV cache.

See `docs/advanced-omlx-settings.md` for full benchmark methodology and Low Power Mode comparison table.

---

## Testing

```bash
# Vision
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma-4-26b-a4b-it-4bit","messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}},{"type":"text","text":"What color?"}]}]}' \
  | jq '.choices[0].message.content'

# Tool calling
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma-4-26b-a4b-it-4bit","tools":[{"type":"function","function":{"name":"get_weather","description":"Get weather","parameters":{"type":"object","properties":{"location":{"type":"string"}},"required":["location"]}}}],"messages":[{"role":"user","content":"Weather in Tokyo?"}]}' \
  | jq '.choices[0]'

# Stack health
bash scripts/llm-status.sh
```

---

## Known Issues / Limits

- **oMLX version must stay at 0.4.1** — all three VLM patches (`model_discovery.py`, `config.json` `vision_config`, `audio_config: null`) are specific to this version. `brew upgrade omlx` silently breaks VLM loading. Pin is set by `setup.sh` (`brew pin omlx`); `llm-start.sh` checks the installed version against `OMLX_VERSION` in `llm.conf` and refuses to start on mismatch.
- Vision requests are slower on first call (model lazy-loads vision encoder on first image).
- `standardize=true` in vision_config is required — without it vision outputs are wrong.
- The proxy (`omlx-proxy.py`) must be running for OpenAI-compatible streaming clients on :8090; direct oMLX calls on :8080 work but skip `reasoning_content → content` routing.
- macOS Low Power Mode halves throughput — disable it for consistent performance.
- 128K context cold prefill takes ~276 s TTFT; in opencode sessions the prefix cache covers repeated system prompts so effective TTFT stays low.