# Search Service — TRD

## Status

Implemented — 2026-03-29

## Overview

### The problem

The Coda AI agent needs to discover which GraphQL operations or Snowflake tables are relevant to a natural-language query. The original hybrid search implementation (see [Semantic Schema Search TRD](./semantic-schema-search.md)) ran an in-process ONNX embedding model inside the Fargate app server. This worked but created a CPU bottleneck: the HuggingFace transformers pipeline exhausts all available CPU during embedding and reranking, blocking request handling and inflating latency. Fargate does not support GPU instances, so scaling vertically is not an option.

### The solution

A standalone ConnectRPC microservice (`apps/search`) that owns all search infrastructure: hybrid BM25 + HNSW vector + glossary search, S3 snapshot persistence, circuit-breaker-wrapped embedding, and independent schema polling. The app server calls it over internal VPC via the `@coda/search-api` client package.

This separation provides:

- **GPU acceleration** — model inference moves to a GPU-backed EC2 instance (g4dn.xlarge with NVIDIA T4)
- **Independent scaling** — search scales by model size/concurrency, app server by request volume
- **Decoupled iteration** — model upgrades, algorithm changes, and new pipeline stages ship without touching the app server
- **Graceful degradation** — app server falls back to keyword-only search when the service is unreachable (500ms timeout)

---

## Architecture

### System context

```
                 ConnectRPC (internal VPC, port 8081)

 ows-coda server ──────────────┐
 (Fargate)                     |
                               v
                   [Search Service (EC2 g4dn.xlarge)]
                     |       |            |
                     v       v            v
                   GraphQL  Snowflake    S3
                   Gateway  (ACCOUNT_    (snapshots)
                            USAGE)
```

The app server uses `@coda/search-api` to call the search service. If the service is unreachable, the server falls back to keyword-only search.

### Service internals

```
apps/search/src/
  index.ts              Entrypoint: init providers, create indexes, start server
  server.ts             Express + ConnectRPC middleware + health endpoints
  config/
    load-config.ts      Zod-validated env vars -> typed SearchConfig
  embedding/
    huggingface.ts      HuggingFace ONNX embedding provider (CPU/CUDA)
    reranker.ts         HuggingFace cross-encoder reranker
  handlers/
    search-graphql.ts   SearchGraphQL RPC handler
    search-snowflake.ts SearchSnowflake RPC handler
    get-graphql-schema.ts  GetGraphQLSchema RPC handler
    get-snowflake-schema.ts GetSnowflakeSchema RPC handler
    validate.ts         Input validation (query length, limit)
    cursor.ts           Cursor-based pagination
  pipeline/
    pipeline.ts         SearchPipeline<T>: hybrid search -> rerank -> graph augment
  graphql/
    schema-index.ts     GraphQLSchemaIndex: lifecycle, polling, snapshot
    introspect.ts       Gateway introspection + hash-based change detection
    search-adapter.ts   GraphQLSearchAdapter + type graph builder
  snowflake/
    schema-index.ts     SnowflakeSchemaIndex: lifecycle, polling, snapshot
    schema-loader.ts    ACCOUNT_USAGE catalog loader + FK graph inference
    search-adapter.ts   SnowflakeSearchAdapter
  snapshot/
    snapshot.ts         Serialize/deserialize (gzip JSON, base64 vectors)
    snapshot-store.ts   SnapshotStore interface, S3SnapshotStore, NullSnapshotStore
```

### Request flow

```
ConnectRPC request (e.g. SearchGraphQL)
  |
  v
[Handler] -- validates input (query length, limit)
  |
  v
[SearchPipeline<T>]
  |
  +-- Stage 1: HybridSearch (BM25 + HNSW + glossary via RRF)
  |     over-fetches limit * overFetch candidates
  |
  +-- Stage 2: Reranker (optional cross-encoder)
  |     re-scores candidates, returns top-limit
  |     on failure: returns Stage 1 results unmodified
  |
  +-- Stage 3: Graph augmentation
  |     appends 1-hop neighbors from type graph (GraphQL) or FK graph (Snowflake)
  |     on failure: returns empty related list
  |
  v
Response: { queryId, results, scores, related }
```

### Index lifecycle

Each schema index (GraphQL and Snowflake) follows the same lifecycle:

1. **Warm start** — load gzip-compressed snapshot from S3, validate `modelId` matches current config, diff against live schema, re-embed only changed documents, merge into live index.
2. **Cold start** (no snapshot or model mismatch) — fetch full catalog, batch-embed all documents, build HNSW + inverted index, save snapshot to S3.
3. **Polling** — configurable interval (default 5 min) with random jitter to avoid thundering herd. Hash-based change detection; incremental add/update/remove operations on the live index.

The `GraphQLSchemaIndex` introspects the federated gateway. The `SnowflakeSchemaIndex` queries `ACCOUNT_USAGE` via a dedicated system connection pool. Both run their poll timers independently.

---

## API

Source of truth: `packages/search-api/proto/coda/search/v1/search.proto`

### RPCs

| RPC                                               | Purpose                                                                                                              |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `SearchGraphQL(query, limit, cursor)`             | Find GraphQL query fields and types. Returns ranked `QueryFieldResult[]`, `TypeResult[]`, and `GraphNeighborhood[]`. |
| `SearchSnowflake(query, database, limit, cursor)` | Find Snowflake tables and columns. Returns ranked `TableResult[]`, `ColumnResult[]`, and related tables.             |
| `GetGraphQLSchema(known_hash)`                    | Conditional-GET for the full introspection JSON. Returns `introspection_json` only if `hash != known_hash`.          |
| `GetSnowflakeSchema(known_hash)`                  | Conditional-GET for the Snowflake table catalog. Returns `SnowflakeSchemaTable[]` only if `hash != known_hash`.      |
| `ReportUsage(query_id, selected_ids)`             | Feedback signal — records which results the agent actually used.                                                     |

### Transport

ConnectRPC serves both protocols on port 8081:

- **Connect JSON** (HTTP/1.1) — REST-compatible, used by the app server client
- **Native gRPC** (HTTP/2, binary protobuf) — available for high-throughput callers

Compression: gzip and brotli accepted, applied to responses over 1024 bytes.

### Health checks

- `GET /health` — always 200
- `GET /health/ready` — 200 when all configured indexes are ready, 503 otherwise (reports per-index status)

---

## Packages

### `@coda/search` — search primitives

The `packages/search/src/` module provides the generic search building blocks:

| Module                 | Purpose                                                                |
| ---------------------- | ---------------------------------------------------------------------- |
| `hybridSearch.ts`      | `HybridSearch<T>` — BM25 + HNSW + glossary fusion via RRF              |
| `hnswIndex.ts`         | HNSW approximate nearest neighbor graph (uint8 quantized)              |
| `invertedIndex.ts`     | Token-based inverted index for BM25 scoring                            |
| `tokenize.ts`          | camelCase split, lowercase, stop words, Porter stemming                |
| `glossary.ts`          | Glossary loader, fuzzy matcher (maxEditDistance=2), query expansion    |
| `quantize.ts`          | Float32 to uint8 vector quantization                                   |
| `embeddingProvider.ts` | `EmbeddingProvider` interface: `embed()`, `embedQuery()`, `dimensions` |
| `rerankProvider.ts`    | `RerankProvider` interface: `rerank(query, documents)`                 |

These are domain-agnostic. The search service and any future consumer use the same primitives.

### `@coda/extensions` — glossary data

Glossary JSON files live in `packages/extensions/`. The search service imports `snowflakeGlossaryFile` from `@coda/extensions` and resolves `{{env}}` placeholders in its entries via `resolveGlossaryVars()` from `@coda/search` at startup.

### `@coda/search-api` — proto contract and client

| File                                | Purpose                                                                                               |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `proto/coda/search/v1/search.proto` | Protobuf service and message definitions                                                              |
| `gen/coda/search/v1/search_pb.ts`   | buf-generated TypeScript types                                                                        |
| `src/client.ts`                     | `buildClient()`, lazy singleton `getSearchClient()`, typed RPC wrappers with timeout + error handling |

The app server imports `searchGraphQL()`, `searchSnowflake()`, etc. from this package. The client uses `SEARCH_URL` env var and defaults to a 500ms timeout (10s for schema downloads).

---

## Search pipeline details

### Hybrid search (Stage 1)

Three signals fused via Reciprocal Rank Fusion:

```
score(doc) = sum_i  1 / (k + rank_i(doc))
```

| Signal         | Method                                       | Notes                                                          |
| -------------- | -------------------------------------------- | -------------------------------------------------------------- |
| BM25 keyword   | Inverted index, k1=1.2, b=0.75               | Tokenized: camelCase split, lowercase, stop words, Porter stem |
| HNSW vector    | Uint8-quantized cosine similarity            | O(log n), m=16, efConstruction=200, efSearch=50                |
| Glossary boost | Word-boundary match, fuzzy (edit distance 2) | Primary targets: weight 1.0, related: 0.5                      |

Default RRF k=25 (configurable via `RRF_K`).

### Reranking (Stage 2)

When `SEARCH_RERANKER_MODEL` is set, a cross-encoder rescores the top `limit * overFetch` (default 4x) candidates. If the reranker fails, Stage 1 results pass through unmodified.

### Graph augmentation (Stage 3)

- **GraphQL**: type graph — query fields to return types (`returns`), types to field types (`hasField`). 1-hop neighbors of matched query fields are appended.
- **Snowflake**: FK graph — inferred from `<X>_ID` column naming conventions (resolves to `X`, `DIM_X`, or `FACT_X`). Bidirectional neighbors included.

### Circuit breaker

The embedding provider is wrapped with a configurable circuit breaker:

- **Closed**: requests pass through; failures counted in a sliding window
- **Open**: after `CB_FAILURE_THRESHOLD` failures in `CB_WINDOW_MS`, requests fail fast
- **Half-open**: after `CB_COOLDOWN_MS`, a probe is allowed; `CB_SUCCESS_THRESHOLD` successes close the circuit

Strategies: `rolling` (default), `fixed`, `consecutive`.

### Snapshot persistence

Snapshots are gzip-compressed JSON stored in S3 at `<prefix><domain>/latest.json.gz`. Float32 vectors are base64-encoded for bit-exact round-tripping. A snapshot includes a `modelId` field; snapshots with a mismatched model are discarded (triggering cold start).

---

## Configuration

All configuration is via environment variables, validated by Zod at startup. Full reference in `apps/search/README.md`.

Key variables:

| Variable                     | Default                                                                  | Purpose                                                            |
| ---------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------ |
| `PORT`                       | 8081                                                                     | HTTP listen port                                                   |
| `SEARCH_EMBEDDING_MODEL`     | `mixedbread-ai/mxbai-embed-large-v1`                                     | HuggingFace embedding model ID                                     |
| `SEARCH_RERANKER_MODEL`      | _(none)_                                                                 | Cross-encoder model; omit to disable reranking                     |
| `GRAPHQL_GATEWAY_URL`        | _(none)_                                                                 | Gateway URL; omit to disable GraphQL indexing                      |
| `SNOWFLAKE_ACCOUNT`          | _(none)_                                                                 | Snowflake account; omit to disable Snowflake indexing              |
| `S3_SNAPSHOT_BUCKET`         | _(none)_                                                                 | S3 bucket for snapshots; omit for no persistence                   |
| `RRF_K`                      | 25                                                                       | RRF fusion constant (1-100)                                        |
| `GRAPHQL_POLL_INTERVAL_MS`   | 3600000                                                                  | GraphQL poll interval (1 hr)                                       |
| `SNOWFLAKE_POLL_INTERVAL_MS` | 3600000                                                                  | Snowflake poll interval (1 hr)                                     |
| `CB_FAILURE_THRESHOLD`       | 5                                                                        | Circuit breaker failure threshold                                  |
| `CB_COOLDOWN_MS`             | 30000                                                                    | Circuit breaker cooldown                                           |
| `SNOWFLAKE_ALLOWLIST`        | `FACTS.{{env}},...`                                                      | FQN prefixes to include in indexing; closed-by-default             |
| `SNOWFLAKE_BLOCKLIST`        | regex patterns (TEMP, TMP, TEST, STAGING, ROLLBACK, short-prefix+digits) | FQN prefixes/regexes to exclude from indexing; overrides allowlist |

---

## Deployment

### Compute

| Resource    | Value                                           |
| ----------- | ----------------------------------------------- |
| Instance    | `g4dn.xlarge` (1x NVIDIA T4, 4 vCPU, 16 GB RAM) |
| Tasks       | 2 (HA)                                          |
| Launch type | ECS EC2 (Fargate does not support GPU)          |
| Port        | 8081 (internal VPC only)                        |

### ECR

Separate ECR repo: `terraform-infra/shared/prod/ecr/repos/ows-coda-search/`.

### Docker

Multi-stage build with an isolated `model-cache` layer:

```
prod-deps --> model-cache (download script, stable layer, ~400 MB)
           |
           +-> dev-deps --> build-artifacts --> deploy-search
                                                (COPY from model-cache)
```

Source changes do not invalidate the model download layer.

### Networking

- Private subnet, no public IP
- Security group: inbound TCP 8081 from app server SG
- Poll jitter (default up to 60s) staggers requests across replicas

---

## Graceful degradation

| Failure                                     | Behavior                                                |
| ------------------------------------------- | ------------------------------------------------------- |
| Embedding provider fails during startup     | Service exits (cannot build initial index)              |
| Embedding provider fails during poll        | Circuit breaker opens; existing index continues serving |
| Reranker unavailable                        | Stage 2 skipped; RRF scores from Stage 1 used           |
| Snapshot absent or stale (modelId mismatch) | Cold start — full re-embed                              |
| Graph augmentation throws                   | `related` returned as empty array                       |
| Poll fails                                  | Warning logged; next poll retries at the next interval  |
| Search service unreachable from app server  | App server keyword fallback (500ms timeout)             |

---

## Testing

### Unit tests

Located in `apps/search/src/__tests__/` and domain-specific `__tests__/` directories:

| Area              | Test files                                                                                                                                                               |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Server + handlers | `server.test.ts`, `search-graphql-handler.test.ts`, `search-snowflake-handler.test.ts`, `get-graphql-schema.test.ts`, `get-snowflake-schema.test.ts`, `validate.test.ts` |
| Pipeline          | `pipeline.test.ts`, `reranker.test.ts`                                                                                                                                   |
| GraphQL index     | `graphql/schema-index.test.ts`, `graphql/introspect.test.ts`, `graphql-search-adapter.test.ts`, `graphql-augment.test.ts`, `graphql-search-integration.test.ts`          |
| Snowflake index   | `snowflake/schema-index.test.ts`, `snowflake/schema-loader.test.ts`, `snowflake-search-adapter.test.ts`, `snowflake-augment.test.ts`, `build-fk-graph.test.ts`           |
| Snapshot          | `snapshot.test.ts`, `snapshot-store.test.ts`                                                                                                                             |
| Config            | `load-config.test.ts`                                                                                                                                                    |
| Embedding         | `embedding/huggingface.test.ts`                                                                                                                                          |
| Cursor/misc       | `cursor.test.ts`, `exec-sql.test.ts`, `sanitize-identifier.test.ts`, `schema-loader-filters.test.ts`, `sql-builders.test.ts`                                             |

### Quality benchmark

`@coda/common` includes an offline quality benchmark validating NDCG@10, MRR, and Recall@10 against regression thresholds. Runs in keyword + glossary mode (no embedding model), executes in < 0.5s, and is part of the standard test suite.

### Client tests

`packages/search-api/src/__tests__/client.test.ts` — covers the ConnectRPC client factory, timeout behavior, error handling, and lazy singleton lifecycle.

### Running

```bash
cd apps/search && pnpm test:unit         # unit tests
cd packages/common && npx jest --testPathPattern=benchmark  # quality benchmark
```

---

## Decision log

| Decision                           | Rationale                                                                                            |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Standalone service over in-process | CPU bottleneck from embedding model blocks app server request handling; GPU not available on Fargate |
| ConnectRPC over REST/gRPC          | Same port serves both protocols; strongly typed; streaming-ready                                     |
| S3 snapshots over database         | Append-only, versioned, cheap, no schema migration                                                   |
| Independent polling over push      | No coupling to app server lifecycle; configurable per domain; jitter prevents thundering herd        |
| HNSW with uint8 quantization       | O(log n) search; 4x memory reduction vs float32; < 5% mean error at 1024 dimensions                  |
| RRF fusion over weighted-sum       | Rank-based fusion is robust to score scale differences between signals                               |
| Glossary as JSON files             | Version-controlled, PR-reviewable, deterministic                                                     |
| Circuit breaker on embedding       | Prevents cascading failures from model inference; index continues serving during outage              |
| NullSnapshotStore pattern          | Clean testing without S3; no conditional null checks in index lifecycle code                         |

---

## Related documents

- [Semantic Schema Search TRD](./semantic-schema-search.md) — original hybrid search design (in-process)
- [Snowflake Schema Index TRD](./snowflake-schema-index.md) — Snowflake indexing design
- [Search Architecture](../../architecture/search.md) — full algorithms and ranking reference