# Runbook

Troubleshooting guide for common issues with ows-coda in development and production.

```mermaid
flowchart TD
    symptom["Issue observed"] --> bedrock{"500 / stream dies?"}
    symptom --> redis{"Redis errors?"}
    symptom --> snowflake{"Snowflake tools missing?"}
    symptom --> graphql{"GraphQL tools missing?"}
    symptom --> auth{"401 responses?"}
    symptom --> db{"DB / Prisma errors?"}
    symptom --> downstream{"Tool call errors?"}

    bedrock --> b1["Check Bedrock quotas<br/>Reduce thinking budget<br/>Increase REQUEST_TIMEOUT"]
    redis --> r1["Check REDIS_URL<br/>Service degrades gracefully"]
    snowflake --> s1["Check SNOWFLAKE_READER_* vars<br/>Verify key format + role grants"]
    graphql --> g1["Check GRAPHQL_GATEWAY_URL<br/>Verify gateway reachable"]
    auth --> a1["Check AUTH0_DOMAIN/AUDIENCE<br/>Verify ows-grass headers"]
    db --> d1["Check connection pool<br/>Run prisma migrate deploy<br/>Regenerate prisma client"]
    downstream --> ds1["Check VPN connection<br/>Inspect tool error in SSE stream<br/>Check Datadog traces"]
```

## Bedrock throttling / timeout

**Symptoms:** 500 errors, SSE stream dies mid-response, `ThrottlingException` in logs

**Cause:** AWS Bedrock rate limits or model capacity

**Fix:**

- Check Bedrock quotas in the AWS console (Service Quotas → Bedrock)
- If using extended thinking, try reducing `BEDROCK_THINKING_BUDGET` (default 2048)
- The `REQUEST_TIMEOUT` env var controls streaming timeout (default 30s). Increase if Claude's responses are being cut off on complex multi-tool queries
- Check Datadog for `bedrock:InvokeModel` latency spikes

## Redis connection failures

**Symptoms:** `[redis-client] connection error` in logs, conversations not persisting

**Cause:** Redis is unreachable or credentials are wrong

**Fix:**

- **Local dev:** Ensure Redis is running (`docker run -p 6379:6379 redis:7`) and `REDIS_URL=redis://localhost:6379` is set in `.env`. Or unset `REDIS_URL` — the service runs stateless without it
- **QA/prod:** Check the Elasticache endpoint is accessible from the Fargate task's VPC. The Redis client auto-retries up to 5 times with exponential backoff, then gives up
- The service degrades gracefully — if Redis is down, conversations reset per-request and rate limiting falls back to in-memory
- Conversation TTL is **7 days** — if conversations seem to disappear, check if the TTL expired

## Snowflake connection failures

**Symptoms:** Snowflake tools return errors or are missing from the tool list, `SnowflakeError` in logs

**Cause:** Snowflake pool not configured, key-pair auth failure, or network issues

**Fix:**

- Check that all `SNOWFLAKE_READER_*` env vars are set (account, user, role, warehouse, private key)
- Snowflake tools are **conditional** — they only appear when the reader pool is configured and healthy. Check the `enabled()` status in logs
- Verify the private key format (PEM) and passphrase if using encrypted keys
- Check that the service user (`SNOWFLAKE_READER_USER`) has the expected role grants
- Statement timeout defaults to 30s — increase `SNOWFLAKE_READER_STATEMENT_TIMEOUT` for complex queries

## GraphQL gateway errors

**Symptoms:** GraphQL tools return errors or are missing from the tool list

**Cause:** Gateway URL not configured or gateway unreachable

**Fix:**

- Check that `GRAPHQL_GATEWAY_URL` is set in `.env`
- GraphQL tools are **conditional** — they only appear when the gateway URL is configured
- Verify the gateway is reachable from the server (check VPN in local dev)
- Introspection queries are blocked by the tools — this is by design

## Auth0 / JWT errors

**Symptoms:** 401 responses from chat endpoints

**Cause:** Token expired, wrong audience, or ows-grass not injecting headers

**Fix:**

- Verify `AUTH0_DOMAIN` and `AUTH0_AUDIENCE` in your `.env` match the Auth0 tenant
- In production, ows-grass sits upstream and validates JWTs. Coda also validates independently — both must agree on issuer/audience
- Check that ows-grass is forwarding identity headers (`X-Orchard-*`)

## Missing env vars at startup

**Symptoms:** `Required env var X is not set` crash immediately on startup

**Cause:** Required service URLs or keys are not set

**Fix:** Ensure these are all set in `apps/server/.env`:

```
OWS_ABACUS_ACCOUNT_URL=...
OWS_ROYALTIES_URL=...
OWS_MONEYHUB_URL=...
OWS_PRODUCT_URL=...
OWS_LEDGER_URL=...
REDIS_URL=...
SENTRY_DSN=...
```

Copy from `apps/server/.env.shadow` if starting fresh.

## Downstream service errors

**Symptoms:** Tool calls return errors, Claude reports "I couldn't retrieve that data"

**Cause:** One of the backend services is down or returning errors

**Fix:**

- Check which tool failed in the SSE stream (`error` events) or Datadog traces
- Verify VPN is connected (required for QA services in local dev)
- Auth errors (401/403) from downstream services are sanitized — check the raw logs for the actual HTTP status
- Tool executor retries are not implemented — if a downstream call fails, it fails immediately

## Docker build fails

**Symptoms:** `GITHUB_NPM_TOKEN` errors during `pnpm install` in Docker

**Cause:** The build requires a GitHub personal access token for private npm packages

**Fix:** Ensure `GITHUB_NPM_TOKEN` is set in your environment before running `pnpm docker:up`.

## Observability

- **Datadog** — dd-trace initializes before all imports in `apps/server/src/index.ts`. Traces cover Express routes and outbound HTTP calls. Disable locally with `DD_TRACE_ENABLED=false`
- **Sentry** — error tracking via `apps/server/src/instrument.ts`. Configure via `SENTRY_DSN` env var. Blank to disable
- **Slack alerts** — Jenkins posts build regressions/fixes to `#abacus-devs`
- **Jira board** — [COD Board](https://theorchard.atlassian.net/jira/software/c/projects/COD/boards/1318) for tracking incidents and work items

## Database (Aurora MySQL / Prisma) issues

**Symptoms:** 500 errors on chat operations, `PrismaClientKnownRequestError` in logs, messages not persisting

**Common causes and fixes:**

- **Connection pool exhausted** — check Prisma connection pool size (`DATABASE_POOL_SIZE`, default 20 dev / 50 prod). Under heavy load, increase the pool or check for connection leaks. Look for `Timed out fetching a new connection from the connection pool` in logs.
- **Migration not applied** — if a new column or table is missing, run `pnpm --filter @coda/db prisma migrate deploy`. In Docker, the `migrate` service runs this automatically on startup.
- **Stale Prisma client** — after schema changes, run `pnpm --filter @coda/db prisma generate` to regenerate the client. The build step handles this, but local dev may need a manual regenerate.
- **FK constraint failures** — usually caused by orphaned references. Check the cascade rules in `prisma/schema.prisma`. Chat deletion cascades to messages, which cascade to tool calls, thoughts, sources, and feedback.

## Model update issues

**Symptoms:** Degraded response quality, thinking/reasoning missing from responses, new model ID not recognized

**Cause:** New Claude model family or version deployed without updating the model configuration

**Fix:**

- Check `supportsThinking()` in `apps/server/src/ai/providers/bedrock/provider.ts` — the regex must match the new model family name. This is a known fragile pattern (P0 issue tracked in [Codebase Review Remediation TRD](../decisions/trds/codebase-review-remediation.md)).
- Verify the model ID is in the `ALLOWED_MODELS` list or the `models` DB table.
- After adding a new model, update the prompt cache warm-up to use the correct `ConverseStreamCommand` (not `ConverseCommand` — another tracked issue).
- Monitor for silent thinking degradation: if `supportsThinking()` returns false for a model that supports it, the agent works but produces lower-quality responses with no error signal.

## Debugging multi-round conversations

When a conversation produces unexpected results across multiple tool-use rounds:

1. **Check Datadog traces** — each request has a trace ID. Filter by `orchard-identity-id` header to find the user's traces.
2. **Check structured logs** — with the logging improvements, filter by `requestId`, `conversationId`, or `identityHash` to see the full request lifecycle.
3. **Inspect Redis conversation state** — use `redis-cli HGETALL conversation:<id>` to see the full message history sent to Bedrock. Look for truncated messages, missing tool results, or malformed content.
4. **Check tool execution logs** — each tool call is logged with input, output, duration, and status. Look for timeouts (30s default), 4xx/5xx from downstream services, or empty result sets.
5. **Verify identity context** — Snowflake queries use session variables for RBAC. Check that `orchard-identity-id` is flowing through to the Snowflake session.

## Health checks

- `GET /health` — liveness probe. Returns `{ "status": "ok" }`. No auth required. Use for load balancer health probes.
- `GET /health/ready` — readiness probe. Returns `503` until Bedrock prompt cache is seeded, then `{ "status": "ready" }`. Use for traffic routing.