# Adding a New Search Index ## Purpose This guide walks you through adding a new searchable data source to the search service. By the end you will have a working `IndexStrategy` that plugs into the existing engine and gets polling, snapshots, health checks, and graceful shutdown for free. **Audience:** developers familiar with TypeScript who need to index a new data source (MySQL tables, a service catalog, a REST API, etc.). ## Prerequisites - Read `apps/search/src/engine/index-strategy.ts` -- the four-method interface you will implement. - Understand that `HybridSearch` fuses BM25 keyword scoring, vector similarity, and glossary boosting via RRF. Your strategy creates and owns one of these. - Look at `GraphQLStrategy` (simple, single-phase) and `SnowflakeStrategy` (two-phase with degraded mode) for real examples. ## What You Get for Free The `IndexEngine` wraps your strategy and handles: - **Snapshot persistence** -- saves/loads documents + vectors to S3 automatically. - **Warm start** -- restores vectors from snapshot so startup skips re-embedding. - **Health status** -- `isReady()` / `isDegraded()` exposed to `/health/ready`. - **Degraded mode** -- tracks a background embed promise and clears the flag when it resolves. - **Graceful shutdown** -- `destroy()` aborts in-flight work, calls `teardown()`, and releases resources. - **Hash tracking** -- stores the latest content hash for change detection. SearchEngine exposes a four-verb API: `init()`, `refresh()`, `search()`, `destroy()`. It does **not** own polling -- the caller decides when to call `refresh()` (timer, webhook, cron, manual trigger). The `startPolling()` utility from `@coda/async` is a convenience wrapper the service uses to schedule refreshes on an interval, but it is not part of SearchEngine itself. You only write domain logic. The engine owns the mechanical lifecycle. ## Step-by-Step Guide We will build a hypothetical `MySQLStrategy` that indexes tables from an Aurora MySQL database. ### Step 1: Define your document type Create `apps/search/src/mysql/schema-loader.ts`. The `fqn` serves as a stable unique ID for `HybridSearch`. ```ts export interface MySQLTableEntry { fqn: string; // "mydb.users" database: string; name: string; comment: string; columns: { name: string; type: string; comment: string }[]; keywords: string[]; // pre-tokenized for BM25 changedOn: string; // ISO timestamp for change detection } ``` ### Step 2: Create your strategy class Create `apps/search/src/mysql/mysql-strategy.ts`: ```ts export class MySQLStrategy implements IndexStrategy { readonly name = "mysql"; private tableSearch: HybridSearch | null = null; private tables: MySQLTableEntry[] = []; private lastHash: string | null = null; private abortController = new AbortController(); constructor(private readonly opts: MySQLStrategyOptions) {} } ``` ### Step 3: Implement `init(snapshot)` Fetch your data, populate `HybridSearch`, and optionally restore vectors from a snapshot. ```ts async init(snapshot: SnapshotData | null): Promise { this._createSearch(this.opts.embeddingProvider); this.tables = await fetchAllTables(this.opts.pool); this.lastHash = computeHash(JSON.stringify(this.tables.map(t => t.fqn).sort())); if (snapshot && this.tableSearch) { // Warm start: restore cached vectors, keyword-index everything const snapMap = new Map( snapshot.documents.map((doc, i) => [doc.fqn, snapshot.vectors[i]]) ); await this.tableSearch.addFromSnapshot( this.tables, this.tables.map(t => { const v = snapMap.get(t.fqn); return v?.length ? v : undefined; }), ); } else if (this.tableSearch) { await this.tableSearch.add(this.tables); } return { hash: this.lastHash, degraded: false }; } ``` For large data sources where embedding takes minutes, return `{ degraded: true, embedPromise }` instead -- see Common Patterns below. ### Step 4: Implement `fetchAndApply(signal)` Fetch changes since the last poll. Return `null` when nothing changed. Pass `signal` through to `HybridSearch` so the engine can abort during shutdown. ```ts async fetchAndApply(signal: AbortSignal): Promise { if (!this.tableSearch) return null; const changes = await fetchChangedTables(this.opts.pool, this.lastPollTime); if (changes.added.length === 0 && changes.removed.length === 0) return null; if (changes.added.length > 0) await this.tableSearch.add(changes.added, signal); if (changes.removed.length > 0) this.tableSearch.remove(changes.removed.map(t => t.fqn)); // Update local state... this.lastHash = computeHash(JSON.stringify(this.tables.map(t => t.fqn).sort())); return { hash: this.lastHash }; } ``` ### Step 5: Implement `snapshotData()` The engine calls this to build a snapshot. Return documents and their vectors directly -- the strategy owns the search index and can export vectors via `HybridSearch.toSnapshot()`. ```ts snapshotData(): SnapshotSpec { if (!this.tableSearch) return { documents: [], vectors: [] }; const snap = this.tableSearch.toSnapshot(); return { documents: snap.docs, vectors: snap.vectors }; } ``` ### Step 6: Implement `teardown()` Abort any in-flight work. Close connection pools here if applicable. ```ts teardown(): Promise { this.abortController.abort(); return Promise.resolve(); } ``` ### Step 7: Add domain-specific methods These are consumed by your RPC handler, not part of `IndexStrategy`. Create a `SearchPipeline` during initialization to get reranking and graph augmentation. ```ts async search(query: string, limit = 10, after?: { score: number; id: string }) { if (!query.trim() || !this.pipeline) { return { queryId: randomUUID(), results: [], scores: [], related: [] }; } return this.pipeline.run(query, { limit, after }); } private _createPipeline(): void { this.pipeline = new SearchPipeline({ search: async (q, opts) => { if (!this.tableSearch) return []; return (await this.tableSearch.search(q, { limit: opts.limit, after: opts.after })).map(r => r.item); }, projectToDocument: (t) => `${t.database}.${t.name}: ${t.comment}`, rerankProvider: this.opts.rerankProvider, }); } ``` ### Step 8: Register in the engine factory Add a builder function in `apps/search/src/engine/engine-factory.ts` and call it from `buildAllEngines()`: ```ts async function buildMySQLEngine( config: SearchConfig, embeddingProvider: EmbeddingProvider, rerankProvider: RerankProvider | undefined, ): Promise { const strategy = new MySQLStrategy({ pool: createMySQLPool(config), embeddingProvider, rerankProvider, }); const engine = new IndexEngine({ strategy, snapshotStore: createSnapshotStore(config, "mysql"), modelId: config.searchEmbeddingModel, }); return { name: "mysql", engine, strategy, pollIntervalMs: config.mysqlPollIntervalMs, pollJitterMs: config.mysqlPollJitterMaxMs, }; } ``` Then add the condition in `buildAllEngines()`: ```ts if (config.mysqlHost) { builders.push(buildMySQLEngine(config, embeddingProvider, rerankProvider)); } ``` Add the strategy reference to `SearchServiceContext` in `index.ts` so handlers can access it. ### Step 9: Create an RPC handler Add a proto definition in `packages/search-api/`, then create `apps/search/src/handlers/search-mysql.ts`. Follow `search-snowflake.ts`: validate the request, check `ctx.engines.get("mysql")?.isReady()`, call `strategy.search()`, map results to protobuf. Register in `server.ts`: ```ts router.service(SearchService, { // ...existing handlers... searchMySQL: (req) => searchMySQLHandler(req, ctx), }); ``` ### Step 10: Health check integration This is automatic. `/health/ready` iterates `ctx.engines` and reports status for each. Because you called `engines.set("mysql", engine)`, your index appears automatically: ```json { "status": "ready", "indexes": { "graphql": "ready", "snowflake": "ready", "mysql": "ready" } } ``` ## Strategy Checklist - [ ] `init()` handles both cold start (no snapshot) and warm start (with vectors) - [ ] `fetchAndApply()` returns `null` when nothing changed (avoids unnecessary snapshots) - [ ] `fetchAndApply()` passes `signal` to `HybridSearch` calls for abort support - [ ] `snapshotData()` returns documents and vectors from `HybridSearch.toSnapshot()` - [ ] `teardown()` aborts background work via `AbortController` - [ ] Config values added to `apps/search/src/config/load-config.ts` - [ ] Builder registered in `engine-factory.ts`, strategy added to `SearchServiceContext` - [ ] Polling interval and jitter are configurable (caller schedules `refresh()` via `startPolling()` from `@coda/async` or similar) - [ ] Tests cover cold start, warm start, poll with changes, and poll with no changes ## Common Patterns **Two-phase initialization** (Snowflake pattern): for data sources where embedding thousands of documents takes minutes, split `init()` into two phases. Phase 1 populates the keyword index immediately (using `addFromSnapshot` with `undefined` vectors for new items). Phase 2 embeds in the background. Return `{ degraded: true, embedPromise }` so the engine tracks the work and clears the flag when done. Keyword search serves queries within seconds while vectors catch up. **Graph augmentation**: if your data has relationships (foreign keys, type references), build a `Graph` during initialization and pass a `graphAugment` function to `SearchPipeline`. The pipeline appends related items automatically. **Glossary integration**: pass domain glossary entries to `HybridSearch` via the `glossaryEntries` config. The search engine expands queries with synonyms and boosts matching documents. See `@coda/extensions` for existing glossary data. **Language threading**: the search pipeline is multilingual-aware. Accept-Language is read by the search service's RPC handlers (`search-graphql.ts`, `search-snowflake.ts`), normalized via `normalizeLocale` (from `@coda/search`), and threaded as `language` through `SearchEngine.search` → `SearchPipeline.run` → `HybridSearch.search`. Inside the pipeline, `tokenize(query, language)` selects the right `Intl.Segmenter`, Snowball stemmer, and stop-word set per call. For new indexes: 1. If your strategy's `getKeywords()` runs at index time, pass a `language` argument to `tokenize()` only when the document fields are genuinely in that language. Most schema-search documents are in English even when consumers ask in other languages — the localized-glossary-term injection (D10) handles cross-language matching automatically. 2. If your domain has a glossary, use the manifest-based loader (`loadGraphqlGlossary` from `@coda/extensions` — analogous for other domains) so `localizedTerms` reach the index. The static synchronous exports skip the i18n overlays. 3. When invoking the search service from server code, forward the caller's `Accept-Language` to the RPC client. The tool-handler convention is to pluck `headers["Accept-Language"]` from the handler's `headers` argument and pass it as the `locale` parameter to `searchGraphQL` / `searchSnowflake`. For overlay file conventions and how to add a new language, see `packages/extensions/graphql/graphql-glossary-i18n/README.md`. ## Testing Mock the data source and use a deterministic embedding provider: ```ts import { createEmbeddingProvider } from "../embedding"; const embeddingProvider = createEmbeddingProvider({ modelId: "deterministic/default", }); const strategy = new MySQLStrategy({ pool: mockPool, embeddingProvider, }); const result = await strategy.initialize(null); // cold start expect(result.degraded).toBe(false); const poll = await strategy.fetchAndApply(new AbortController().signal); expect(poll).toBeNull(); // no changes mockPool.setChanges({ added: [newTable], removed: [] }); const poll2 = await strategy.fetchAndApply(new AbortController().signal); expect(poll2?.hash).not.toBe(result.hash); // detected change ``` > **Note:** In production, `IndexEngine` owns the `AbortController` and aborts it during `destroy()`, making in-progress refreshes cancellable during graceful shutdown. In strategy unit tests, create your own `AbortController` to test abort handling. Key patterns: - **Mock your data source**, not `HybridSearch` -- let the real search engine run against deterministic embeddings so you test the full pipeline. - **Inject the data-fetching function** via an options field (like `SnowflakeStrategy.fetchCatalog` or `GraphQLStrategy.introspect`) so tests control data without mocking network calls. - **Test snapshot round-trips**: initialize cold, extract `snapshotData()`, then initialize again with that snapshot to verify warm start produces the same hash.