# Observability & Tracing

How to monitor, trace, and debug ows-coda in development and production.

## Stack

| Tool           | Purpose                                | Access                                |
| -------------- | -------------------------------------- | ------------------------------------- |
| **Datadog**    | APM traces, metrics, log aggregation   | Org Datadog instance                  |
| **Sentry**     | Error tracking, stack traces           | Project DSN in `.env` (`SENTRY_DSN`)  |
| **CloudWatch** | Fargate task logs, ECS service events  | AWS Console (`us-east-1`)             |
| **Langfuse**   | Prompt versioning, trace correlation   | Self-hosted (see infrastructure docs) |
| **pino**       | Structured JSON logging (all services) | stdout → Datadog log pipeline         |

## Request-scoped context

Every log line and trace span carries structured fields set via `AsyncLocalStorage` in `request-context.ts`:

| Field            | Source                              | Purpose                                          |
| ---------------- | ----------------------------------- | ------------------------------------------------ |
| `requestId`      | UUID generated per request          | Correlate all logs within a single HTTP request  |
| `identityHash`   | SHA-256 of user identity (12 chars) | Track per-user activity without exposing raw IDs |
| `conversationId` | Chat UUID from route params         | Group logs by conversation                       |

These fields are automatically included in every pino log entry via the mixin pattern. Datadog indexes them as facets for filtering. Additionally, `orchard-identity-id` and other identity headers from the Grass JWT are forwarded to downstream services on every tool call for end-to-end tracing (HTTP headers, not AsyncLocalStorage fields).

## Tracing a request end-to-end

### 1. Find the request

In Datadog APM, filter by:

- **Service:** `ows-coda` (server), `ows-coda-search` (search service)
- **Resource:** `POST /api/v1/chats/:id/stream` (streaming), `GET /api/v1/chats` (listing)
- **Tag:** `orchard.identity.id:<hash>` for per-user traces

Or in logs, search by `requestId` to find all log entries for a single request.

### 2. Follow the trace

A typical streaming request produces this trace:

```
POST /api/v1/chats/:id/stream
  ├── requireAuth (JWT validation)
  ├── enrichRequestContext (identity resolution)
  ├── loadMemory (agent memory consolidation, 500ms timeout)
  ├── getSystemPrompt (Langfuse fetch with local fallback)
  ├── converseWithTools (agent loop)
  │     ├── bedrock.ConverseStream (Claude API call)
  │     ├── executeTools (parallel tool dispatch)
  │     │     ├── search_accounts → ows-abacus-account HTTP
  │     │     └── get_revenue_summary → ows-moneyhub HTTP
  │     ├── bedrock.ConverseStream (follow-up with tool results)
  │     └── suggestions (non-streaming Haiku call)
  └── StreamPersister.complete (Redis + Aurora write)
```

### 3. Downstream tool calls

Each tool call via `safeGet`/`safePost` forwards auth headers and logs:

- Tool name, input parameters, duration, success/failure
- HTTP status from downstream service
- Sanitized error (internal URLs and tokens stripped)

Slow requests (>3s) are logged at `warn` level.

### 4. Search service traces

The search service (`apps/search`) has its own trace context. Key operations:

- `SearchPipeline.run` — BM25 + HNSW + glossary fusion
- `RerankProvider.rerank` — cross-encoder reranking
- `SearchEngine.refresh` — polling and incremental index updates
- `SnapshotManager.save` — S3 snapshot persistence

The admin `TraceQuery` RPC returns per-stage span details for search debugging.

## Datadog APM setup

`dd-trace` is loaded at process startup via the Node.js `--import` flag in the Dockerfile: `node --import dd-trace/initialize.mjs dist/index.mjs`. This ensures dd-trace instruments all modules before they load. The server compiles to ESM (`.mjs` output).

To disable locally: `DD_TRACE_ENABLED=false` in `.env`.

## Sentry setup

Sentry is initialized in `apps/server/src/instrument.ts` (imported as a side-effect in `index.ts` before all other imports) with:

- 10% trace sample rate
- PII sending disabled
- 401/403/429 errors filtered out (operational noise)
- Health probe transactions dropped

The Sentry Express error handler is registered before the custom error handler so exceptions are captured before the generic 500 response.

## Structured logging

All services use pino with JSON output. Key log patterns:

```
// Request lifecycle
{ level: "info", requestId: "abc-123", msg: "request started", method: "POST", path: "/api/v1/chats/..." }
{ level: "info", requestId: "abc-123", msg: "request completed", statusCode: 200, durationMs: 1423 }

// Tool execution
{ level: "info", requestId: "abc-123", tool: "search_accounts", durationMs: 142, success: true }
{ level: "warn", requestId: "abc-123", tool: "query_snowflake", durationMs: 3200, msg: "slow tool call" }

// Search service
{ level: "info", engine: "snowflake", event: "refresh.complete", added: 3, removed: 0, durationMs: 850 }
```

## Key metrics to watch

| Metric                       | Alert threshold | Impact                           |
| ---------------------------- | --------------- | -------------------------------- |
| Bedrock latency p99          | > 15s           | Streaming responses feel slow    |
| Tool call error rate         | > 5% over 5 min | Downstream service degradation   |
| Redis connection errors      | Any sustained   | Conversations reset per-request  |
| Search circuit breaker state | `open`          | Keyword-only search (no vectors) |
| Fargate task restarts        | > 2 in 10 min   | Possible OOM or crash loop       |
| Prompt cache miss rate       | > 20%           | Higher Bedrock costs             |

## Local debugging

For local development without Datadog:

1. Logs go to stdout in JSON — pipe through `pnpm exec pino-pretty` for human-readable output
2. Set `DD_TRACE_ENABLED=false` to skip dd-trace initialization
3. Set `SENTRY_DSN=` (empty) to disable Sentry
4. Use `DEBUG=*` for verbose module-level logging (not recommended — very noisy)

## See also

- [Runbook](../operations/runbook.md) — troubleshooting common issues
- [Deployment](../operations/deployment.md) — CI/CD pipeline and rollback
- [Server Architecture](../architecture/server.md#observability) — dd-trace and Sentry setup details