# Local LLM Stack — Setup Guide

*Platform: macOS, Apple Silicon M-series, 48 GB unified memory*  
*Stack: oMLX 0.4.1, mlx-vlm 0.5.0, Python 3.11*  
*Last validated: 2026-06-04*

---

## Prerequisites

```bash
# Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# oMLX inference server — MUST be pinned to 0.4.1
# All VLM patches are specific to this version. brew upgrade omlx WILL break VLM loading.
brew tap jundot/omlx
brew install jundot/omlx/omlx   # installs to /opt/homebrew/Cellar/omlx/0.4.1/
brew pin omlx                    # blocks brew upgrade from touching it

# Proxy dependencies
pip3 install aiohttp
```

---

## Model Downloads

```bash
mkdir -p ~/.llm/models ~/.llm/models-disabled
cd ~/.llm/models

# Primary — Gemma4 26B-A4B (VLM: vision + tool calling + thinking) ~15 GB
huggingface-cli download mlx-community/gemma-4-26b-a4b-it-4bit \
  --local-dir gemma-4-26b-a4b-it-4bit
```

---

## oMLX Patch — VLM Detection Fix

The Gemma4 model uses `vision_soft_tokens_per_image` as its vision indicator rather than `vision_config` in the config. oMLX 0.4.1's model discovery doesn't recognize this key, causing it to load Gemma4 as a text-only model.

**File:** `/opt/homebrew/Cellar/omlx/0.4.1/libexec/lib/python3.11/site-packages/omlx/model_discovery.py`

Find `_has_vision_subconfig` (~line 405) and add one line:

```python
# BEFORE
return (
    "vision_config" in config
    or "vit_config" in config
    or bool(config.get("mm_vision_tower"))
)

# AFTER
return (
    "vision_config" in config
    or "vit_config" in config
    or bool(config.get("mm_vision_tower"))
    or "vision_soft_tokens_per_image" in config  # Gemma4 VLM convention
)
```

See `patches/omlx-model-discovery.patch` for the exact diff.

---

## Gemma4 Config Fix

The locally downloaded `gemma-4-26b-a4b-it-4bit` may be missing `vision_config` in its `config.json`. Without it, mlx-vlm instantiates a 16-layer vision tower against a 27-layer model and falls back to text-only.

Run this once after downloading:

```bash
python3 - <<'PYEOF'
import json, os

path = os.path.expanduser("~/.llm/models/gemma-4-26b-a4b-it-4bit/config.json")
with open(path) as f:
    config = json.load(f)

if "vision_config" not in config or not config.get("vision_config"):
    config["vision_config"] = {
        "model_type": "gemma4_vision",
        "hidden_size": 1152,
        "intermediate_size": 4304,
        "num_hidden_layers": 27,
        "num_attention_heads": 16,
        "num_key_value_heads": 16,
        "head_dim": 72,
        "global_head_dim": 72,
        "hidden_activation": "gelu_pytorch_tanh",
        "patch_size": 16,
        "pooling_kernel_size": 3,
        "position_embedding_size": 10240,
        "default_output_length": 280,
        "max_position_embeddings": 131072,
        "rms_norm_eps": 1e-06,
        "rope_parameters": {"rope_theta": 100.0, "rope_type": "default"},
        "standardize": True,
        "use_clipped_linears": False,
        "attention_bias": False,
        "attention_dropout": 0.0
    }
    print("Added vision_config")
else:
    print("vision_config already present")

# Suppress audio_config injection (Gemma4 26B has no audio tower)
config["audio_config"] = None
with open(path, "w") as f:
    json.dump(config, f, indent=2)
print("Done. config.json updated.")
PYEOF
```

---

## oMLX Model Settings

Create `~/.omlx/model_settings.json` (minimum Gemma4 override):

```json
{
  "version": 1,
  "models": {
    "gemma-4-26b-a4b-it-4bit": {
      "model_type_override": "vlm",
      "model_alias": "gemma4",
      "max_context_window": 262144,
      "max_tokens": 65536,
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 64,
      "turboquant_kv_enabled": true,
      "turboquant_kv_bits": 4.0,
      "turboquant_skip_last": true,
      "specprefill_enabled": true,
      "is_pinned": true,
      "is_default": true,
      "thinking_budget_enabled": false,
      "vlm_mtp_enabled": false
    }
  }
}
```

This forces oMLX to use `VLMBatchedEngine` even if model discovery still classifies it as text-only. Key fields:
- `model_alias: "gemma4"` — shorter alias for API requests
- `max_context_window: 262144` — override global 32K default; Gemma4 native 256K
- `specprefill_enabled: true` — sparse MoE prefill, activates at 8K+ token prompts; measurably reduces TTFT at large context
- `turboquant_kv_enabled` — 4-bit KV cache via mlx-vlm TurboQuant; ~4× memory reduction vs fp16 KV; critical for 256K context without blowout
- `is_pinned: true` — keep model weights resident in unified memory; prevents eviction between requests

---

## Directory Setup

```bash
mkdir -p ~/.llm/{models,models-disabled,logs,run}
mkdir -p ~/.omlx/cache
```

Use the stack scripts directly from this checkout:

```bash
cd ~/src/collab/lvona/local-llm
chmod +x scripts/*.sh
# If you copy the scripts elsewhere, update llm.conf paths accordingly
```

---

## Starting the Stack

```bash
bash scripts/llm-start.sh
```

Output should show:
```
omlx:    starting multi-model server on :8080 (models: /Users/YOU/.llm/models)
  PID XXXXX — log: /Users/YOU/.llm/logs/primary.log
  Waiting for omlx on :8080 .... ready
proxy:   starting on :8090
  PID XXXXX — log: /Users/YOU/.llm/logs/omlx-proxy.log
  Waiting for proxy on :8090 . ready

╔══════════════════════════════════════════════╗
║  Stack UP                                    ║
╠══════════════════════════════════════════════╣
║  omlx      http://localhost:8080 /v1        ║
║  proxy     http://localhost:8090 /v1        ║
╚══════════════════════════════════════════════╝
```

---

## Verifying the Stack

```bash
# Model discovery check
curl -s http://localhost:8080/v1/models | python3 -m json.tool | grep '"id"'

# Vision test (paste a valid base64 PNG)
python3 - <<'EOF'
import urllib.request, json, struct, zlib, base64

def make_png(w, h, rgb):
    def chunk(n, d):
        c = n+d; return struct.pack('>I',len(d))+c+struct.pack('>I',zlib.crc32(c)&0xffffffff)
    raw = b''.join(b'\x00'+bytes(rgb)*w for _ in range(h))
    return b'\x89PNG\r\n\x1a\n'+chunk(b'IHDR',struct.pack('>IIBBBBB',w,h,8,2,0,0,0))+chunk(b'IDAT',zlib.compress(raw))+chunk(b'IEND',b'')

img = base64.b64encode(make_png(4,4,[255,0,0])).decode()
payload = {"model":"gemma-4-26b-a4b-it-4bit","stream":False,"max_tokens":20,"messages":[{
    "role":"user","content":[
        {"type":"image_url","image_url":{"url":f"data:image/png;base64,{img}"}},
        {"type":"text","text":"What color? One word."}
    ]
}]}
req = urllib.request.Request("http://localhost:8080/v1/chat/completions",
    data=json.dumps(payload).encode(), headers={"Content-Type":"application/json"})
with urllib.request.urlopen(req, timeout=60) as r:
    resp = json.loads(r.read())
print("Vision:", resp['choices'][0]['message']['content'])  # Should print "Red"
EOF

# Tool calling test
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model":"gemma-4-26b-a4b-it-4bit",
    "messages":[{"role":"user","content":"Weather in Tokyo?"}],
    "tools":[{"type":"function","function":{"name":"get_weather","description":"Get weather","parameters":{"type":"object","properties":{"location":{"type":"string"}},"required":["location"]}}}]
  }' | python3 -c "import sys,json; r=json.load(sys.stdin); c=r['choices'][0]; print('Tools:', c['finish_reason'], c['message'].get('tool_calls',[{}])[0].get('function',{}).get('name',''))"
```

---

## opencode Integration

opencode uses `~/.config/opencode/opencode.json` (not `config.json`). It requires the
`@ai-sdk/openai-compatible` npm package to talk to local OpenAI-compatible servers.

### 1. Install the AI SDK adapter

```bash
cd ~/.config/opencode
npm install @ai-sdk/openai-compatible
```

### 2. Write `~/.config/opencode/opencode.json`

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "model": "omlx/gemma-4-26b-a4b-it-4bit",
  "provider": {
    "omlx": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "oMLX (local)",
      "options": {
        "baseURL": "http://localhost:8090/v1",
        "apiKey": "omlx-local",
        "headerTimeout": false,    // CRITICAL: default 10 s kills cold prefill (Gemma4 @32K = ~30 s TTFT)
        "timeout": 600000,         // 10 min total — for 128K context runs
        "chunkTimeout": 600000     // 10 min — must cover worst-case TTFT (128K = 276 s; under memory pressure can be longer). Default 120 s caused retry loops on large-context prefills
      },
      "models": {
        "gemma-4-26b-a4b-it-4bit": {
          "name": "Gemma 4 26B-A4B (local)",
          "tool_call": true,
          "modalities": {
            "input": ["text", "image"],  // REQUIRED for vision — omitting silently discards images
            "output": ["text"]
          },
          "limit": {
            "context": 262144,     // tell opencode the real context window
            "output": 16384        // cap max_tokens at 16K (default sends 32000 every request)
          }
        }
      }
    }
  },
  "compaction": {
    "auto": true                   // auto-compact when context fills
  },
  "agent": {
    "build": {
      "temperature": 0.3,          // lower temp = more deterministic tool-call JSON
      "steps": 30                  // cap agentic loop iterations; prevents runaway retries
    },
    "plan": {
      "temperature": 0.1           // planning phase: most deterministic
    },
    "title": {
      "temperature": 0.1           // session title generation
    }
  },
  "experimental": {
    "continue_loop_on_deny": true  // don't abort agent loop when a tool is denied
  }
}
```

> **`baseURL` points to the proxy on `:8090`**, not oMLX directly on `:8080`. The proxy
> handles `reasoning_content → content` routing for streaming responses. Direct `:8080`
> works but skips that compatibility shim.
>
> **Do not use the old `capabilities: { tool_call, image }` schema** — that key has no
> effect in opencode. Vision is controlled by `modalities.input`, tools by `tool_call`.
>
> For the exact `~/.config/opencode/AGENTS.md` content and tool-call discipline used with
> Gemma 4, see [`docs/opencode-agents.md`](./opencode-agents.md).
>
> Add additional models to the `models` block as needed — any model served by oMLX
> can be referenced as `omlx/<model-name>`.

### 2b. Configure `~/.config/opencode/AGENTS.md` (global tool-call discipline)

Gemma 4 has confirmed bugs in opencode that cause tool-call loops, wrong tool names, and heredoc syntax errors. Copy the contents of `docs/opencode-agents.md` into `~/.config/opencode/AGENTS.md` to apply mitigations for all sessions. Key rules injected:

- Exact tool name list (prevents hallucinated `read_file`, `write_file` etc.)
- No heredoc/herestring in bash (prevents the `<<<` loop bug, issue #22481)
- `edit` oldString rules + re-read-before-retry (prevents infinite loop, issue #21850)
- Error recovery protocol (never repeat identical failed calls)

See `docs/opencode-agents.md` for the full content and issue citations.

### 3. Configure the `ui-designer` subagent

Create `~/.config/opencode/agents/ui-designer.md`:

```bash
mkdir -p ~/.config/opencode/agents
cat > ~/.config/opencode/agents/ui-designer.md << 'EOF'
---
name: ui-designer
description: Expert in reviewing UI screenshots and asset layouts.
model: omlx/gemma-4-26b-a4b-it-4bit
---

You are a UI/UX design reviewer. When given screenshots or images, describe
layout issues, accessibility concerns, and visual hierarchy problems concisely.
EOF
```

### 4. Verify

```bash
# Check opencode sees the provider
opencode models 2>/dev/null | grep omlx || echo "start opencode and check /models in the TUI"

# Quick sanity check via the proxy
curl -s http://localhost:8090/v1/models | python3 -c \
  "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data']]"
```

---

## oMLX Advanced Configuration

### model_settings.json — Gemma4 per-model settings

`~/.omlx/model_settings.json` controls per-model behavior. Full working config for Gemma4:

```json
{
  "version": 1,
  "models": {
    "gemma-4-26b-a4b-it-4bit": {
      "model_type_override": "vlm",
      "max_context_window": 262144,
      "max_tokens": 32768,
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 64,
      "turboquant_kv_enabled": true,
      "turboquant_kv_bits": 4.0,
      "turboquant_skip_last": true,
      "is_pinned": true,
      "is_default": true,
      "thinking_budget_enabled": false,
      "ttl_seconds": null
    }
  }
}
```

Key fields (most undocumented in official docs):
- `model_type_override: "vlm"` — force VLMBatchedEngine regardless of auto-detection
- `max_context_window: 262144` — override global 32K default; Gemma4 native 256K
- `turboquant_kv_enabled` — 4-bit KV cache via mlx-vlm TurboQuant; ~4× memory reduction vs fp16 KV; critical for 256K context without blowout
- `turboquant_kv_bits: 4.0` — supported: 2, 2.5, 3, 3.5, 4, 6, 8
- `turboquant_skip_last: true` — skip last KV layer (prevents known corruption on sensitive models)
- `is_pinned: true` — keep model weights resident in unified memory; prevents eviction between requests
- `is_default: true` — serve this model when no `model` field specified in API request

### settings.json — server-wide defaults

`~/.omlx/settings.json` relevant sections:

```json
{
  "sampling": {
    "max_context_window": 262144,
    "max_tokens": 65536
  },
  "memory": {
    "memory_guard_tier": "custom",
    "memory_guard_custom_ceiling_gb": 34.0,
    "soft_threshold": 0.90,
    "hard_threshold": 0.95,
    "prefill_safe_zone_ratio": 0.90,
    "prefill_min_chunk_tokens": 512
  },
  "scheduler": {
    "max_concurrent_requests": 2,
    "chunked_prefill": true
  },
  "cache": {
    "hot_cache_max_size": "6GB",
    "hot_cache_only": false,
    "initial_cache_blocks": 1024,
    "ssd_cache_dir": "~/.llm/prefix-cache",
    "ssd_cache_max_size": "40GB",
  },
  "huggingface": {
    "hf_cache_enabled": false
  }
}
```

- `memory_guard_custom_ceiling_gb: 34.0` — ceiling raised from 30 → 34 GB to eliminate `adaptive_prefill_throttle` on two concurrent 16K-token sessions (observed peak 30.3 GB). Leaves 14 GB headroom on 48 GB system; below Metal cap (37.4 GB).
- `soft_threshold: 0.90` — LRU eviction/throttle fires at 30.6 GB (34×0.90); above concurrent session peak
- `prefill_safe_zone_ratio: 0.90` — secondary prefill guard raised to match soft_threshold
- `prefill_min_chunk_tokens: 512` — 16× fewer memory checks per large chunked prefill (default 32)
- `scheduler.chunked_prefill: true` — spreads long prefills across scheduler steps; prevents starvation of running requests
- `cache.hot_cache_max_size: "6GB"` — resident KV budget; reduced from 8 GB to free 2 GB headroom for concurrent large-context sessions
- `cache.initial_cache_blocks: 1024` — pre-allocates cache at startup (default 256); reduces latency spikes on first long request
- `cache.ssd_cache_dir` — prefix cache: reuses computed prefill states; opencode sends same ~3K system prompt every turn — skips that prefill entirely
- `cache.ssd_cache_max_size: "40GB"` — disk cap for the prefix cache. **Theoretical useful maximum** for this config: 2 concurrent sessions × 256K context × 4-bit KV ≈ **16 GB live**. Everything beyond that is LRU historical snapshots from prior sessions. oMLX auto-evicts the oldest entries when the cap is reached — no manual cleanup needed. Set to 40 GB to absorb eviction bursts without the `SSD write queue full` warning that occurs when the cache is at capacity and blocks are dropped instead of written. Going beyond 40 GB yields no performance benefit for single-user 2-session workloads.

### CLI — llm-ask.sh

`scripts/llm-ask.sh` is a pipe-friendly curl wrapper — no extra dependencies.

```bash
# Direct prompt
./scripts/llm-ask.sh "What is the capital of France?"

# Pipe stdin
echo "Explain this" | ./scripts/llm-ask.sh
cat file.py | ./scripts/llm-ask.sh "What does this do?"

# Options
./scripts/llm-ask.sh --max-tokens 512 --no-stream "Summarise this" < doc.txt
./scripts/llm-ask.sh --port 8080 "Direct to oMLX, skip proxy"
```

### Web UI

Open **http://localhost:8080/admin/chat** — oMLX built-in chat with markdown, code highlighting, and tool call display. No extra process required.

---

## Claude Code Integration

oMLX exposes a native Anthropic Messages API at `/v1/messages` — no proxy or adapter
needed. Claude Code can route directly to the local stack by setting two environment variables.

### 1. Set environment variables

```bash
export ANTHROPIC_BASE_URL="http://localhost:8080"
export ANTHROPIC_API_KEY="omlx-local"
```

Add to `~/.zshrc` to make permanent. `setup.sh` writes a sourceable env file to
`~/.config/local-llm/claude-code-env.sh` that does the same thing:

```bash
source ~/.config/local-llm/claude-code-env.sh
# or permanently:
echo 'source ~/.config/local-llm/claude-code-env.sh' >> ~/.zshrc
```

### 2. Set the model

Claude Code defaults to whichever `claude-*` model your subscription provides. With the
local endpoint active you pass the local model name directly:

```bash
claude --model gemma-4-26b-a4b-it-4bit "Explain this file" src/main.py
```

Or configure `~/.claude/settings.json` to set a default model:

```json
{
  "model": "gemma-4-26b-a4b-it-4bit"
}
```

### 3. Verify

```bash
# Confirm Anthropic endpoint is live
curl -s http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: omlx-local" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model":"gemma-4-26b-a4b-it-4bit","max_tokens":8,"messages":[{"role":"user","content":"Say OK"}]}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['content'][0]['text'])"
```

Expected output: `OK`

### Notes

- `ANTHROPIC_BASE_URL` must point to **:8080** (oMLX native), not :8090 (proxy). The proxy
  is OpenAI-format only; the Anthropic endpoint is served directly by oMLX.
- If Claude Code requests a `claude-*` model name, configure `model_alias` in
  `~/.omlx/model_settings.json` — `setup.sh` sets `model_alias: "gemma4"` automatically.
- `context_scaling_enabled` and `target_context_size` can be tuned in the `claude_code`
  block of `~/.omlx/settings.json` (see `docs/advanced-omlx-settings.md`).

---

## Operational Scripts

| Script | Purpose |
|--------|---------|
| `llm-start.sh` | Start oMLX + proxy, wait for health |
| `llm-stop.sh` | SIGTERM both processes, clean PID files |
| `llm-restart.sh` | Stop + start in sequence |
| `llm-status.sh` | Show PID, port health, active models, memory |
| `llm-keepwarm.sh` | Ping loop to prevent model eviction from LRU cache |
| `llm-sync.sh` | Drift-check and one-command sync working copy → collab |
| `llm-rotate-model.sh` | Hot-swap the primary model without full restart |
| `llm-cache-watchdog.sh` | Monitor SSD KV cache size, prune if over threshold |
| `browser-preflight.sh` | Verify stack is up before launching browser agent |
| `setup.sh` | One-shot install: patches, model config, oMLX settings, opencode.json |
| `omlx-proxy.py` | Aiohttp compatibility proxy (`reasoning_content → content` routing) |
| `llm.conf` | Shared env vars: ports, model dir, log dir |

---

## Troubleshooting

### Model loads as LLM instead of VLM

```
omlx.engine_pool - INFO - Successfully loaded gemma-4-26b-a4b-it-4bit as LLM (fallback from VLM)
```

Check primary.log for the warning before this line:
- `Received N parameters not in model` → `vision_config` is wrong (wrong `num_hidden_layers` or `head_dim`). Re-run the config fix script.
- `Missing N parameters: audio_tower.*` → `audio_config` is not null. Add `"audio_config": null` to config.json.
- `VLM loading failed: ...` → Some other mlx-vlm exception. Check full stacktrace in primary.log.

### Image not seen by model

```
omlx.utils.image - WARNING - Failed to load image: broken PNG file
```

The base64 image data is malformed. Use the `make_png` helper above to generate valid test images.

### Stream bug on third-party VLMs

```
There is no Stream(gpu, 1) in current thread
```

This error indicates an oMLX stream/executor ordering bug (GitHub issue #1636). It affected earlier candidate models during the Gemma4 evaluation; Gemma4 itself was not affected. See `docs/gemma4-vlm-investigation.md §A` for full historical context.

### Throughput drops unexpectedly / extra models appear in /v1/models

oMLX has `hf_cache_enabled: true` by default. When enabled it automatically scans
`~/.cache/huggingface/hub` as a **second** model directory alongside `model_dirs`, surfacing
every MLX-compatible model in your HF cache. On a machine with many downloaded models this
inflates the LRU pool and causes cache fragmentation that measurably reduces tok/s.

Symptoms: `/v1/models` returns models you didn't configure; `avg_generation_tps` in
`/api/status` is lower than expected.

Fix — add to `~/.omlx/settings.json`:
```json
{
  "huggingface": {
    "hf_cache_enabled": false
  }
}
```

Requires a **full restart** (hot-reload does not re-run directory scanning).

### Proxy not routing thinking content

If `reasoning_content` is empty but the model should be thinking, verify `omlx-proxy.py` is running on :8090 (not connecting directly to :8080). The proxy routes thinking tokens from `reasoning_content` into `content` for streaming responses and otherwise forwards requests unchanged.

### Tokens arriving all-at-once instead of streaming (opencode shows no progress bar)

**Symptom:** opencode TUI shows a spinner with no intermediate tokens; response appears in one shot.

**Root cause:** The proxy was using `iter_any()` to read upstream chunks, which returns large TCP segments containing multiple SSE events batched together. Additionally, `write()` without `drain()` allows aiohttp to buffer writes internally before flushing to the client.

**Fix (already applied in current `omlx-proxy.py`):**
- Read line-by-line: `async for raw_line in upstream_resp.content` — each SSE event is one line
- Call `await response.drain()` after every `write()` to flush immediately to TCP
- Result: tokens arrive ~16 ms apart (measured at 65 tok/s)

If you see this after an upgrade or manual edit, verify the proxy contains `drain()`:

```bash
grep -n "drain\|iter_any" scripts/omlx-proxy.py
# Should show drain() and NOT iter_any()
```

### Tool calls not executing in opencode (model outputs tool JSON as text)

**Symptom:** opencode shows the model writing raw JSON like `{"name":"write","arguments":{...}}` in the chat instead of executing the tool.

**Root cause:** This is a model capability issue, not a proxy or config issue. Models that don't natively implement OpenAI function calling output the tool invocation as literal text. opencode sees a `finish_reason: stop` text response instead of `finish_reason: tool_calls`.

**Gemma4 26B-A4B status:** Supports native function calling — `finish_reason: tool_calls` confirmed via direct API test. If this happens with Gemma4, it is a prompt/context issue (model chose not to use a tool), not a structural failure.

**oMLX tool call streaming behavior** (from source, `omlx/api/tool_calling.py`): oMLX's `ToolCallStreamFilter` **suppresses tool call tokens from content deltas in real-time** (Gemma4 uses `<|tool_call>...<tool_call|>` markers). After generation completes, it parses the buffered tool call text and emits one structured `tool_calls` chunk. The proxy must not modify this chunk — it is forwarded intact.

**Debug steps:**
```bash
# 1. Confirm proxy delivers tool_calls correctly
python3 - <<'EOF'
import urllib.request, json
tools = [{"type":"function","function":{"name":"bash","description":"Run shell command",
  "parameters":{"type":"object","properties":{"command":{"type":"string"}},"required":["command"]}}}]
body = json.dumps({"model":"gemma4","messages":[{"role":"user","content":"List files in /tmp using bash."}],
  "tools":tools,"tool_choice":"auto","max_tokens":200,"stream":True}).encode()
req = urllib.request.Request("http://127.0.0.1:8090/v1/chat/completions",data=body,
  headers={"Content-Type":"application/json"},method="POST")
finish=None; tool_seen=False
with urllib.request.urlopen(req,timeout=60) as r:
    for raw in r:
        line=raw.decode().strip()
        if line.startswith("data:") and "[DONE]" not in line:
            try:
                d=json.loads(line[5:])["choices"][0]
                if d.get("delta",{}).get("tool_calls"): tool_seen=True
                if d.get("finish_reason"): finish=d["finish_reason"]
            except: pass
print(f"finish_reason={finish}  tool_calls_seen={tool_seen}")
# Expected: finish_reason=tool_calls  tool_calls_seen=True
EOF

# 2. Check opencode config has tool_call: true (default is true, omitting is fine)
cat ~/.config/opencode/opencode.json | python3 -m json.tool | grep tool_call
```

**Known opencode issues** (open as of 2026-06-05):
- [#29996](https://github.com/sst/opencode/issues/29996): gemma4 via Ollama shows code in chat, doesn't generate files — model-side tool call failure
- [#21181](https://github.com/sst/opencode/issues/21181): Local subagents return tool call payload as text — confirmed model capability issue

### ClientConnectionResetError in proxy log

```
aiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport
```

**This is benign.** It occurs when the client (opencode or a benchmark script) closes the TCP connection while the proxy is still writing. The current proxy catches `OSError` (the parent class) and breaks cleanly. If you see this in an older proxy version, upgrade to the current `omlx-proxy.py`.

### opencode retry loop — queue floods oMLX (6+ active requests, 12 tok/s)

**Symptom:** `active_requests` in `/api/status` exceeds `max_concurrent_requests`; `avg_generation_tps` drops to 10–15; opencode TUI shows spinner indefinitely; proxy log shows `TransferEncodingError: 400 Not enough data to satisfy transfer length header`.

**Root cause:** `chunkTimeout` in `opencode.json` fires when no SSE chunk arrives for N ms. During large-context prefill (e.g. 38K tokens under memory pressure: ~148 s) oMLX produces no output chunks. opencode treats the timeout as a failure and immediately retries — each retry queues another request. The queue grows faster than requests drain.

**Fix (already applied):** `chunkTimeout: 600000` (10 min) — matches `timeout` and covers worst-case TTFT (128K = 276 s). Default `120000` was the trigger.

**If you see this happening:**
1. Drop the stuck opencode session (Ctrl-C or `/exit`)
2. Restart the stack to flush zombies: `bash scripts/llm-stop.sh && bash scripts/llm-start.sh`
3. Start a new session — compaction will keep context manageable going forward