# Cookbook

Common recipes for extending `@coda/search`. Each recipe is self-contained — copy the relevant section and adapt to your domain.

## Add a new data source

Implement `SchemaFetcher` and `DocumentTransformer`, then wire into `SearchEngine`.

```ts
import type { SchemaFetcher, FetchResult, SchemaDiff } from "@coda/search";

interface ApiEndpoint {
  path: string;
  method: string;
  description: string;
  parameters: string[];
}

class OpenApiFetcher implements SchemaFetcher<ApiEndpoint> {
  readonly name = "openapi";

  async fetch(signal: AbortSignal): Promise<FetchResult<ApiEndpoint>> {
    const spec = await fetchOpenApiSpec(this.specUrl, signal);
    const endpoints = parseEndpoints(spec);
    return {
      items: endpoints,
      hash: computeSpecHash(spec),
    };
  }

  async diff(signal: AbortSignal): Promise<SchemaDiff<ApiEndpoint> | null> {
    const current = await this.fetch(signal);
    // Compare with last-known state and return changes
    return computeDiff(this.lastItems, current.items);
  }

  async teardown(): Promise<void> {}
}
```

The transformer maps your domain objects to searchable documents:

```ts
import type { DocumentTransformer, KeywordField } from "@coda/search";
import { tokenize } from "@coda/search";

class EndpointTransformer implements DocumentTransformer<
  ApiEndpoint,
  ApiEndpoint
> {
  adapt(raw: ApiEndpoint): ApiEndpoint {
    return raw;
  }

  getId(doc: ApiEndpoint): string {
    return `${doc.method}:${doc.path}`;
  }

  buildDocument(doc: ApiEndpoint): string {
    return `${doc.method} ${doc.path} ${doc.description}`;
  }

  getKeywords(doc: ApiEndpoint): KeywordField[] {
    return [
      { tokens: tokenize(doc.path), boost: 3 },
      { tokens: tokenize(doc.description) },
      { tokens: doc.parameters.flatMap((p) => tokenize(p)) },
    ];
  }
}
```

Wire them together:

```ts
const engine = new SearchEngine<ApiEndpoint, ApiEndpoint>({
  fetcher: new OpenApiFetcher(),
  transformer: new EndpointTransformer(),
  snapshotStore: mySnapshotStore,
  modelId: "my-model",
  factory: {
    embeddingProvider: myEmbedder,
    engineName: "openapi",
  },
});
```

---

## Add a custom ranking signal

Implement `StaticSignal` (computed once, cached) or `QuerySignal` (computed per query).

### Static signal: popularity boost

```ts
import type { StaticSignal, NamedSignal, RankedEntry } from "@coda/search";

class PopularitySignal implements StaticSignal {
  readonly name = "popularity";

  constructor(private readonly getPopularity: () => Map<string, number>) {}

  compute(): NamedSignal | null {
    const popularity = this.getPopularity();
    if (popularity.size === 0) return null;

    const max = Math.max(...popularity.values());
    if (max === 0) return null;

    const entries: RankedEntry[] = [];
    for (const [id, count] of popularity) {
      entries.push({ id, score: count / max });
    }

    entries.sort((a, b) => b.score - a.score);
    return { name: this.name, entries, desc: true };
  }
}
```

Pass to the factory config:

```ts
factory: {
  // ... other config
  staticSignals: [new PopularitySignal(() => queryCountsByTable)],
}
```

Static signals are automatically cached between queries and recomputed when `invalidateStaticSignals()` is called (happens on every refresh).

### Query signal: contextual boost

Query signals receive the current query's top candidates and compute scores relative to them.

```ts
import type { QuerySignal, NamedSignal, RankedEntry } from "@coda/search";

class ContextSignal implements QuerySignal {
  readonly name = "context";

  compute(candidateIds: string[], query: string): NamedSignal | null {
    // Boost candidates that match the user's current context
    const entries: RankedEntry[] = candidateIds
      .filter((id) => isInCurrentContext(id))
      .map((id, i) => ({ id, score: 1 - i * 0.1 }));

    return entries.length > 0 ? { name: this.name, entries, desc: true } : null;
  }
}
```

---

## Add a domain glossary

Glossary entries map business terms (what users say) to document IDs (what the index stores). The glossary has two effects:

1. **Query expansion** — "royalties" expands to include tokens from matched glossary terms
2. **Direct boosting** — matched targets get a score boost via `GlossaryMatchStage`

```ts
import type { GlossaryProvider, GlossaryEntry } from "@coda/search";

class MyGlossary implements GlossaryProvider {
  getGlossaryEntries(): GlossaryEntry[] {
    return [
      {
        // "revenue", "earnings", "income" all map to these tables
        terms: ["revenue", "earnings", "income"],
        targets: ["REVENUE_SUMMARY", "REVENUE_DETAIL"],
        // related tables get a weaker boost
        related: ["ACCOUNT", "CONTRACT"],
        // context is concatenated into the embedding for richer vectors
        context: "Revenue flows through contracts to account balances",
        // priority multiplies the boost (default 1)
        priority: 2,
      },
      {
        terms: ["DSP", "digital service provider", "streaming platform"],
        targets: ["STORE", "STORE_REVENUE"],
        context: "DSPs like Spotify, Apple Music report streaming revenue",
      },
    ];
  }
}
```

The glossary is hot-reloadable — `getGlossaryEntries()` is called on every refresh, so you can update entries at runtime without restarting the engine.

---

## Build a relationship graph

Implement `GraphBuilder` to create edges between documents. The graph powers several ranking signals:

- **DegreeSignal** (static): highly-connected nodes rank higher
- **AdamicAdarSignal** (static): nodes with rare, specific neighbors rank higher
- **BetweennessSignal** (static): "bridge" nodes connecting separate domains rank higher
- **ProximitySignal** (per-query): nodes close to top results get a boost

```ts
import type { GraphBuilder } from "@coda/search";
import { LabeledGraph } from "@coda/search";

class ForeignKeyGraphBuilder implements GraphBuilder<MyDoc> {
  buildGraph(docs: MyDoc[]): LabeledGraph<string> {
    const g = new LabeledGraph<string>();

    // Add nodes
    for (const doc of docs) {
      g.addNode({ key: doc.id, kind: doc.type, data: doc.id });
    }

    // Add FK edges — infer from column names ending in _id
    for (const doc of docs) {
      for (const col of doc.columns) {
        if (!col.endsWith("_id")) continue;
        const targetName = col.replace(/_id$/, "").toUpperCase();
        const target = docs.find((d) => d.name === targetName);
        if (target) {
          g.addEdge({ from: doc.id, to: target.id, relation: col });
        }
      }
    }

    return g;
  }

  // Optional: incremental updates instead of full rebuild
  updateGraph(
    graph: LabeledGraph<string>,
    diff: { added: MyDoc[]; changed: MyDoc[]; removed: string[] },
  ): boolean {
    // Remove deleted nodes
    for (const id of diff.removed) {
      graph.removeNode(id);
    }

    // Add new nodes and edges
    for (const doc of diff.added) {
      graph.addNode({ key: doc.id, kind: doc.type, data: doc.id });
      // Re-infer edges for new node...
    }

    return true; // return false to trigger full rebuild instead
  }
}
```

The graph also powers **join path discovery** and **graph augmentation** — search results include `related` items and `joinPaths` automatically.

---

## Filter results at query time

`SearchFilter` prunes results after fusion, before reranking. Useful for access control or scope limiting.

```ts
import { type SearchFilter } from "@coda/search";

// Prefix-based: only tables in a specific database
const databaseFilter: SearchFilter = {
  include: ["ANALYTICS.*"],
};

// Exclude: hide deprecated tables
const noLegacy: SearchFilter = {
  exclude: ["LEGACY_*"],
};

// Predicate: arbitrary logic
const onlyLargeTables: SearchFilter = {
  predicate: (doc) => (doc as MyDoc).rowCount > 1000,
};

// Combine all three
const combined: SearchFilter = {
  include: ["ANALYTICS.*"],
  exclude: ["LEGACY_*"],
  predicate: (doc) => (doc as MyDoc).rowCount > 1000,
};

// Pass to search()
const results = await engine.search("revenue", { limit: 10, filter: combined });
```

---

## Monitor with events

Subscribe to structured events for logging, metrics, or debugging.

```ts
import { InMemoryEventBus, SearchEventType } from "@coda/search";

const bus = new InMemoryEventBus();

// Glob patterns: "poll.*" matches poll.started, poll.completed, poll.failed
bus.on("poll.*", (e) => {
  metrics.increment(`search.${e.type}`, { source: e.source });
});

// Type-safe payload access
bus.on(SearchEventType.EMBED_COMPLETED, (e) => {
  metrics.histogram("search.embed.duration_ms", e.data.embedDurationMs);
  metrics.gauge("search.vectors", e.data.vectorCount);
});

// Error monitoring
bus.on("*.failed", (e) => {
  logger.error({
    type: e.type,
    error: e.data.error,
    recoverable: e.data.recoverable,
    fallback: e.data.fallback,
  });
});

// Collect events for a trace
bus.on(SearchEventType.QUERY_COMPLETED, (e) => {
  tracer.endSpan(e.traceId, { resultCount: e.data.resultCount });
});
```

All 29 event types are listed in `SearchEventType`. The envelope includes OTel-compatible fields (`traceId`, `spanId`, `parentSpanId`) for future bridging.

---

## Filter documents at fetch time

Use a `Filter` to exclude documents before they enter the index. The Snowflake
engine uses `PrefixFqnFilter` (trie-backed allowlist + blocklist) to control
which `DATABASE.SCHEMA.TABLE` FQNs are fetched. Excluded documents never reach
the keyword or vector indexes.

```ts
import { PrefixFqnFilter, type Filter } from "@coda/search";

const filter: Filter = new PrefixFqnFilter(
  ["FACTS.PROD", "ROYALTY_ACCOUNTING.PROD"],
  ["FACTS.PROD.TEMP_LOAD"], // blocklist overrides allowlist
);

// Pass to SchemaFetcher — only matching FQNs are fetched and indexed
```

For regex-based exclusion patterns, use `parseFqnFilter` which composes
`PrefixFqnFilter` with `PatternFqnFilter` automatically.

---

## Add fuzzy typo tolerance

`FuzzyStage` catches typos and abbreviations that keyword search misses. It uses edit-distance matching over a trie — no LLM needed.

```ts
import { HybridSearch, tokenize } from "@coda/search";

const search = HybridSearch.create<MyDoc>({
  getId: (doc) => doc.id,
  buildDocument: (doc) => doc.text,
  getKeywords: (doc) => [{ tokens: tokenize(doc.text) }],
  embeddingProvider: null,
  // FuzzyStage is built-in — it activates automatically.
  // Configure via these HybridSearch options:
  // minTokenLength: 5,  // tokens shorter than this skip fuzzy (default 5)
  onError: console.error,
});
```

FuzzyStage only fires for query tokens >= 5 characters (configurable via `minTokenLength` on `FuzzyStageConfig`). Shorter tokens produce too many false positives at edit distance 1 (e.g., "ad" matches "id", "at", "an").

Exact matches are excluded from fuzzy results — the keyword stage handles those. Fuzzy only contributes corrections: "conrtact" → "contract" (1 edit) participates in RRF, but "contract" → "contract" (0 edits) does not.

---

## Add a cross-encoder reranker

`RerankProvider` rescores fusion results with a cross-encoder model for higher precision. It runs after fusion and before graph augmentation.

```ts
import type { RerankProvider } from "@coda/search";

class CohereReranker implements RerankProvider {
  async rerank(query: string, documents: string[]): Promise<number[]> {
    const response = await fetch("https://api.cohere.ai/v1/rerank", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.COHERE_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: "rerank-english-v3.0",
        query,
        documents,
        top_n: documents.length,
      }),
    });
    const data = await response.json();
    // Return scores in the same order as the input documents
    const scores = new Array(documents.length).fill(0);
    for (const result of data.results) {
      scores[result.index] = result.relevance_score;
    }
    return scores;
  }
}
```

Wire into the factory:

```ts
factory: {
  embeddingProvider: myEmbedder,
  rerankProvider: new CohereReranker(),
  engineName: "my-engine",
}
```

The pipeline over-fetches by 4x (configurable) to give the reranker a larger candidate pool, then slices to the requested limit after rescoring. If the reranker fails, results fall back to fusion scores — no degradation in availability.

---

## Allocate token budgets

`allocateBudget()` distributes a fixed token budget across ranked results using a linear gradient — top results get more context, lower results get less.

```ts
import { allocateBudget } from "@coda/search";

const results = ["doc1", "doc2", "doc3", "doc4", "doc5"];
const totalBudget = 2000; // max tokens across all results

const allocations = allocateBudget(results.length, totalBudget);
// allocations: [600, 480, 360, 320, 240] (linear gradient, sums to 2000)

for (let i = 0; i < results.length; i++) {
  const context = truncateToTokens(getDocument(results[i]), allocations[i]);
  // ... use context in prompt
}
```

This prevents unbounded context injection when many results match. Top results (most relevant) get more tokens; lower results get less.

---

## Use SearchPipeline directly

`SearchPipeline` is the three-stage pipeline used internally by `SearchEngine`. You can use it directly for custom orchestration.

```ts
import { SearchPipeline } from "@coda/search/pipeline";

const pipeline = new SearchPipeline<MyDoc>({
  // Stage 1: search function (from HybridSearch or any source)
  search: (query, limit) => hybridSearch.search(query, limit),

  // Stage 2: reranking (optional)
  projectToDocument: (doc) => doc.text,
  rerankProvider: myReranker,

  // Stage 3: graph augmentation (optional)
  graphAugment: (results) => {
    const related = findGraphNeighbors(results);
    return { related, joinPaths: [] };
  },

  // Over-fetch multiplier (default 4)
  overFetch: 6,
});

const result = await pipeline.run("revenue by artist", { limit: 10 });
// result.results — reranked top-10
// result.scores — reranker scores (or fusion scores if no reranker)
// result.related — graph neighbors of top results
// result.joinPaths — FK join paths between results
```

Each stage degrades independently: if the reranker throws, fusion scores pass through; if graph augmentation fails, results return without related items.

---

## Use quantized HNSW for lower memory

`QuantizedHnswIndex` stores uint8 vectors in the HNSW graph, reducing memory by 4x. Float32 vectors are retained for snapshot fidelity.

```ts
import { QuantizedHnswIndex } from "@coda/search";

const index = new QuantizedHnswIndex<string>({
  capacity: 50_000, // max vectors (inserts beyond this are silently rejected)
  // m, efConstruction, efSearch inherited from HnswConfig defaults
});

// Pass as the vectorIndex option to HybridSearch
const search = HybridSearch.create<MyDoc>({
  // ... other config
  embeddingProvider: myEmbedder,
  vectorIndex: index,
  onError: console.error,
});
```

Memory comparison at 10K documents with 1024-dimensional vectors:

| Index                                | Memory | Precision loss           |
| ------------------------------------ | ------ | ------------------------ |
| `VectorIndex` (brute force, float32) | ~40MB  | None                     |
| `HnswIndex` (HNSW, float32)          | ~55MB  | ~1% (ANN approximation)  |
| `QuantizedHnswIndex` (HNSW, uint8)   | ~14MB  | ~5% (quantization + ANN) |

Use `QuantizedHnswIndex` when memory matters more than the last 4% of precision.

---

## Measure search quality

Use the built-in evaluation metrics to measure and guard retrieval quality.

```ts
import {
  ndcg,
  mrr,
  precisionAtK,
  recallAtK,
  type RelevanceMap,
} from "@coda/search";

// Define ground truth: which documents are relevant to this query?
const expected: RelevanceMap = new Map([
  ["LEDGER", 3], // highly relevant
  ["ACCOUNT", 2], // relevant
  ["CONTRACT", 1], // marginally relevant
]);

// Get ranked results from your search
const ranked = results.map((r) => r.id);

// Compute metrics
const quality = {
  ndcg: ndcg(ranked, expected, 10), // 0-1, higher is better
  mrr: mrr(ranked, expected), // 0-1, where is the first correct result?
  precision: precisionAtK(ranked, expected, 5), // fraction of top-5 that are relevant
  recall: recallAtK(ranked, expected, 10), // fraction of relevant docs found in top-10
};
```

For CI regression detection, assert aggregate metrics stay above a threshold:

```ts
// In a vitest test
const avgNdcg = queries.map((q) => ndcg(search(q), q.expected, 10)).reduce(avg);
expect(avgNdcg).toBeGreaterThan(0.85); // regression guard
```

See [Example 10](../examples/10-evaluation/) for a complete walkthrough.

---

## Use sub-path imports

Import only what you need to minimize bundle size:

```ts
// Full API (everything)
import { SearchEngine, HybridSearch, tokenize } from "@coda/search";

// Engine orchestration only
import { SearchEngine, EngineState } from "@coda/search/engine";

// Event system only
import { InMemoryEventBus, SearchEventType } from "@coda/search/events";

// Snapshot persistence only
import { serializeSnapshot, deserializeSnapshot } from "@coda/search/snapshot";
```

---

## Next steps

- [Getting Started](getting-started.md) — end-to-end tutorial
- [Concepts](concepts.md) — theory behind BM25, HNSW, RRF, glossary matching
- [Pipeline Architecture](pipeline.md) — how expanders, stages, and signals compose
- [Tuning Guide](tuning.md) — parameter guidance for production
- [Extending](extending.md) — custom stages, signals, expanders, scorers
- [Examples](../examples/) — runnable standalone scripts