> **Superseded.** This TRD describes the original in-process search implementation. The current architecture is documented in [Search Architecture](../../architecture/search.md) and the search service TRD at [Search Service](search-service.md). # Semantic Schema Search — TRD ## Status Implemented — 2026-03-22 ## Overview ### The problem When a user asks the AI agent "show me royalty payments for Sony," the agent needs to find the right GraphQL field to query. The field is called `abacusContract` — not "royalties," not "payments," not anything a keyword search would match. This is the **vocabulary gap**: users think in business language, but schemas use internal naming conventions. The current search implementation (`server/src/ai/tools/graphql/search.ts`) uses pure keyword tokenization. It splits camelCase names, strips stop words, and scores on token overlap. This works when the user's words happen to appear in the schema (e.g., searching "contract" finds `abacusContract`), but it fails systematically in four ways: 1. **Vocabulary mismatch** — "royalties" vs. `abacusContract`. No amount of token splitting bridges this gap because the words are semantically related but lexically different. 2. **Conceptual queries** — "how are releases distributed?" spans multiple types (`Product`, `DistributionDelivery`, `DistributionDeliveryType`). Keyword search finds fragments but not the full picture. 3. **Wrong field selection** — search finds the right root query but picks the wrong fields or misses nested types needed for a complete query. 4. **Argument confusion** — the agent does not know what values to pass for required arguments, leading to trial-and-error query construction. The result: multi-round failures where the agent searches, picks the wrong field, builds a broken query, gets an error, searches again, and eventually either stumbles onto the answer or gives up. This burns tokens, wastes time, and degrades user trust. ### The solution Hybrid search combining three signals — **vector embeddings**, **keyword matching**, and a **curated domain glossary** — backed by an in-memory graph for relationship traversal. - **Vector embeddings** understand that "royalties" and "contract" are semantically close, even though they share no characters. - **Keyword matching** handles exact or near-exact name matches that embeddings might underweight. - **A curated glossary** maps known business vocabulary (e.g., "royalties" -> `abacusContract`) with 100% precision, and can include curated query examples that short-circuit the search-then-build cycle entirely. - **A schema graph** auto-derived from introspection captures structural relationships (which types connect to which), so the agent can traverse from a root query field to deeply nested types. ### What is vector search? (brief primer) Vector search converts text into arrays of numbers (called "embeddings") where semantically similar text produces similar number arrays. The embedding model is trained on large text corpora and learns that "royalties" and "contract payments" are related concepts, even though they share no letters. At index time, every schema entry (field name + description + argument names) is converted to a 384-dimensional vector. At search time, the user's query is converted to a vector in the same space, and we find the schema entries whose vectors are closest (measured by cosine similarity — essentially the angle between two vectors, where 1.0 = identical direction, 0.0 = unrelated). This is not a large language model. The embedding model (`all-MiniLM-L6-v2`) is 23MB, runs locally in-process via ONNX, requires no API keys, and produces vectors in milliseconds. ## Goals ### Goals - Bridge the vocabulary gap between business language and schema naming conventions so the agent finds correct fields in fewer rounds. - Provide a generic, reusable search infrastructure (`server/src/search/`) that works across GraphQL, Snowflake, and future data sources. - Maintain the existing public API (`searchSchema()`, `getTypeInfo()`, `isSchemaLoaded()`) so all current consumers and tests continue to work without modification. - Keep search latency sub-millisecond after initial indexing. - Support graceful degradation: if the embedding model fails to load (CI, network issues, ONNX error), search falls back to keyword + glossary with no downtime. - Enable curated glossary examples that short-circuit multi-round query building for known high-value queries. ### Non-goals - Snowflake integration — the infrastructure supports it, but wiring Snowflake schema into hybrid search is a separate effort. - Replacing the embedding model with a cloud-hosted one (Bedrock Titan, OpenAI). The in-process model is sufficient for the current schema size. - Building a persistent vector database. All vectors are held in memory and rebuilt on startup or schema change. - User-facing search UI. This is agent-internal infrastructure. ## Architecture ### Component diagram ``` ┌─────────────────────────────────┐ │ common/ package │ │ │ │ ┌────────────┐ ┌───────────┐ │ │ │ bfs.ts │ │ Graph │ │ │ │ (algorithms)│ │ (graph/) │ │ │ └────────────┘ └───────────┘ │ └──────────────┬──────────────────┘ │ imported by ┌──────────────▼──────────────────┐ │ server/src/search/ │ │ │ │ ┌─────────────────────────────┐ │ │ │ embedding/ │ │ │ │ provider.ts (interface) │ │ │ │ transformers.ts (MiniLM) │ │ │ │ vector-index.ts (cosine) │ │ │ └─────────────────────────────┘ │ │ ┌──────────────┐ ┌───────────┐ │ │ │ glossary.ts │ │ hybrid- │ │ │ │ (loader + │ │ search.ts │ │ │ │ matcher) │ │ (combiner)│ │ │ └──────────────┘ └───────────┘ │ └──────────────┬──────────────────┘ │ used by ┌──────────────▼──────────────────┐ │ server/src/ai/tools/graphql/ │ │ │ │ search.ts (GraphQL adapter) │ │ schema-index.ts (lifecycle, │ │ polling) │ │ │ │ server/src/ai/skills/ │ │ graphql-explore/handler.ts │ └──────────────────────────────────┘ ``` ### Data flow — index time ```mermaid sequenceDiagram participant Startup as Server Startup participant Intro as introspectAndIndex() participant Search as search.ts (adapter) participant Embed as EmbeddingProvider participant VI as VectorIndex participant G as Graph Startup->>Intro: introspect gateway Intro-->>Search: SchemaState (queryFieldIndex, typeIndex, typeMap) Search->>Search: build search documents (name + desc + args + glossary context) Search->>Embed: embed(documents[]) — batch all entries Embed-->>Search: Float32Array[] (384-dim vectors) Search->>VI: add(entries with vectors) Search->>G: build graph (types as nodes, field return types as edges) Search->>Search: hash schema for change detection ``` ### Data flow — search time ```mermaid sequenceDiagram participant User as User Query participant HS as HybridSearch participant GL as Glossary participant EP as EmbeddingProvider participant VI as VectorIndex User->>HS: "royalty payments for Sony" HS->>GL: matchGlossary(query, candidateIds) GL-->>HS: [{id: "abacusContract", boost: 1.0}] HS->>EP: embed(["royalty payments for Sony"]) EP-->>HS: queryVector (384-dim) HS->>VI: search(queryVector, limit) VI-->>HS: [{item, score}...] ranked by cosine similarity HS->>HS: compute keyword scores (token overlap) HS->>HS: combine: score = 0.6*vector + 0.2*keyword + 0.2*glossary HS-->>User: ranked results with graph neighborhood attached ``` ### Data flow — polling ```mermaid sequenceDiagram participant Timer as setInterval (5 min) participant Intro as introspectAndIndex() participant Hash as computeSchemaHash() participant Build as buildHybridIndexes() Timer->>Intro: introspect gateway Intro-->>Hash: SchemaState Hash-->>Timer: SHA-256 hash alt hash === lastHash Timer->>Timer: skip (no change) else hash !== lastHash Timer->>Build: rebuild vectors + graph Build-->>Timer: swap state, log change end ``` The hash is computed after schema normalization (sorted type names + field signatures), not on raw JSON. This avoids false-positive rebuilds from non-deterministic field ordering in introspection responses. ## Detailed Design ### Embedding model - **Model:** `Xenova/all-MiniLM-L6-v2` via `@huggingface/transformers` - **Dimensions:** 384 - **Size:** ~23MB (ONNX format), cached in `~/.cache/huggingface` after first download - **Runtime:** In-process ONNX inference — no external API, no API keys, no network calls after first download - **Initialization:** Lazy — model loads on first `embed()` call, not at import time - **Configurable:** `SEARCH_EMBEDDING_MODEL` env var overrides the default model ID The model converts text into 384-dimensional vectors where semantically similar text produces vectors that point in similar directions. For example, "royalties" and "contract payments" would produce vectors with high cosine similarity (~0.7-0.8), while "royalties" and "user profile" would produce vectors with low similarity (~0.1-0.2). ### EmbeddingProvider interface ```ts interface EmbeddingProvider { embed(texts: string[]): Promise; readonly dimensions: number; dispose?(): Promise; } ``` Minimal interface with optional `dispose()` for releasing ONNX sessions in tests and during hot-swap. Each implementation handles its own initialization. The `TransformersEmbeddingProvider` implements this with lazy model loading and L2-normalized vector output. ### VectorIndex Brute-force cosine similarity search over in-memory `Float32Array` vectors: ```ts class VectorIndex { add(entries: VectorEntry[]): void; search( queryVector: Float32Array, limit: number, ): Array<{ item: T; score: number }>; clear(): void; get size(): number; } ``` - O(n) linear scan — fast enough for the current schema sizes (hundreds to low thousands of entries) - Emits a warning log when `size > 10,000` to signal when approximate nearest neighbors (ANN) should be considered - No disk persistence — rebuilt on each introspection cycle - ~80 lines of implementation ### Graph\ Generic labeled directed graph in `common/`. No search/embedding/AI dependencies. ```ts interface GraphNode { id: string; kind: string; data: T; } interface GraphEdge { from: string; to: string; relation: string; } class Graph { addNode(node: GraphNode): void; addEdge(edge: GraphEdge): void; getNode(id: string): GraphNode | undefined; findPath(from: string, to: string, maxDepth?: number): string[] | null; neighborhood(id: string, depth?: number): GraphNode[]; get size(): number; } ``` - Adjacency list internally (`Map>`) - BFS for path finding (configurable max depth, default 5) - `neighborhood()` returns all nodes reachable within N hops (default depth 2, max 50 results to prevent context bloat) - BFS algorithms extracted into `common/src/algorithms/bfs.ts` as generic functions parameterized by a `GetNeighbors` callback, so they are testable independently and reusable for future algorithms For GraphQL, the graph is typed as `Graph` with `kind` discriminators: - Query field nodes (`kind: "graphql_query_field"`) link to type nodes via `"returns"` edges - Type nodes link to other type nodes via `"hasField"` edges based on field return types ### Glossary A curated JSON file mapping business vocabulary to schema entries: ```json { "entries": [ { "terms": ["royalties", "royalty payments", "royalty contracts"], "targets": ["abacusContract"], "related": ["AbacusRoyaltyPayment", "AbacusPaymentTerm"], "context": "Royalties in the Abacus system are represented as contracts...", "domain": "royalties", "priority": "primary", "examples": [ { "question": "how much is owed on contract X?", "query": "query ($id: ID!) { abacusContract(id: $id) { balance status terms { amount currency } } }", "variables": { "id": "" } } ] } ] } ``` | Field | Required | Purpose | | ---------- | -------- | ----------------------------------------------------------------------------- | | `terms` | yes | Business vocabulary that maps to this schema concept | | `targets` | yes | Schema entry names that are the primary matches | | `related` | no | Structurally related entries (scored lower) | | `context` | no | Injected into search documents at embed time to enrich vectors | | `domain` | no | Category tag for scoped search and result grouping | | `priority` | no | `"primary"` or `"secondary"` — primary entries get a 1.1x score multiplier | | `prefer` | no | Deprecation redirect: "use X instead" | | `examples` | no | Curated query examples with correct args; short-circuits multi-round failures | The glossary loader reads JSON at startup, logs a warning for missing or malformed files, and produces an empty glossary (zero entries) so search continues with vector + keyword only. Each domain gets its own file in `server/src/search/glossaries/`. ### Scoring formula Each search result receives a hybrid score combining three signals: ``` score = alpha * vector_score + beta * keyword_score + gamma * glossary_score ``` | Signal | Weight | Score range | Source | | -------- | ----------- | ---------------- | -------------------------------------------------------------------- | | Vector | alpha = 0.6 | 0.0 to 1.0 | Cosine similarity between query vector and entry vector | | Keyword | beta = 0.2 | 0.0 to 1.0 | Normalized token overlap (existing tokenize + match logic) | | Glossary | gamma = 0.2 | 0.0, 0.5, or 1.0 | 1.0 if entry is a glossary `target`, 0.5 if `related`, 0.0 otherwise | **Priority boost:** entries with a numeric `priority` multiplier scale the glossary boost (e.g., `priority: 2` doubles the boost). **Worked example:** User searches "royalty payments" | Entry | Vector score | Keyword score | Glossary score | Final score | | ------------------- | ------------ | ------------- | -------------- | ------------------------------------------------------ | | `abacusContract` | 0.75 | 0.0 | 1.0 (target) | (0.6 _ 0.75 + 0.2 _ 0.0 + 0.2 _ 1.0) _ 1.1 = **0.715** | | `AbacusPaymentTerm` | 0.60 | 0.50 | 0.5 (related) | 0.6 _ 0.60 + 0.2 _ 0.50 + 0.2 \* 0.5 = **0.560** | | `userProfile` | 0.10 | 0.0 | 0.0 | 0.6 _ 0.10 + 0.2 _ 0.0 + 0.2 \* 0.0 = **0.060** | Without hybrid search, a keyword search for "royalty payments" returns zero results for `abacusContract` because none of those tokens appear in the field name. The glossary and vector signals together ensure the correct result ranks first. ### Degraded mode If the embedding provider fails to load (network unavailable, ONNX runtime error, CI environment): 1. `EmbeddingProvider.embed()` throws on first call 2. `HybridSearch` catches the error, logs a warning once, and enters degraded mode 3. In degraded mode, vector weight is redistributed to keyword: `score = (alpha + beta) * keyword + gamma * glossary` 4. Search still works — just without semantic matching. Glossary and keyword signals remain active. 5. On next index rebuild (poll tick), embedding is retried. If it succeeds, full hybrid mode resumes. Tests can pass `null` as the embedding provider to force degraded mode, avoiding the 23MB model download. ### Polling for schema changes `setInterval` (default 5 minutes, configurable via `SEARCH_POLL_INTERVAL_MS`): 1. Introspect the gateway 2. Compute a SHA-256 hash of the normalized schema (sorted type names + field signatures) 3. If hash matches the last hash, skip 4. If hash differs, rebuild vectors + graph, swap state atomically, log the change `stopPolling()` clears the interval for clean shutdown and tests. ## Alternatives Explored ### Why hybrid over pure semantic? Pure vector search alone is unreliable for schema discovery. Embedding models can produce false positives (semantically similar but structurally wrong matches) and cannot handle exact-name lookups as well as keyword search. The glossary provides deterministic, manually-verified mappings for the highest-value vocabulary gaps. Combining all three signals gives the best of each approach: semantic understanding, exact matching, and domain expertise. ### Why in-process over API-based embeddings? An API-based embedding service (Bedrock Titan, OpenAI) would add network latency to every search, require API keys and cost management, and introduce a hard dependency on external availability. The in-process `all-MiniLM-L6-v2` model is: - Free (no API costs) - Fast (sub-millisecond inference after model load) - Offline-capable (after first download) - Sufficient quality for schema search (384 dimensions captures enough semantic signal for field-name-level text) The `EmbeddingProvider` interface makes it straightforward to swap in an API-based provider later if the in-process model proves insufficient. ### Why brute-force over HNSW? HNSW (Hierarchical Navigable Small World) is an approximate nearest neighbors algorithm that provides O(log n) search instead of O(n). However: - The current GraphQL schema has hundreds of entries, not thousands. Brute-force cosine over 1,000 entries with 384-dim vectors takes under 1ms. - HNSW adds a native dependency (`hnswlib-js`) with platform-specific binaries, complicating CI and deployment. - The `VectorIndex` interface is designed for a drop-in HNSW replacement: when the warning log fires at 10K entries, swap the implementation without changing any consumers. ## Cost Analysis | Category | Cost | | ------------------- | ------------------------------------------------------------------------------------ | | External API calls | None — embedding runs in-process | | Model download | ~23MB one-time download, cached in `~/.cache/huggingface` | | Memory (embeddings) | ~1.5 KB per schema entry (384 dims \* 4 bytes/float). 1,000 entries = ~1.5 MB | | Memory (graph) | Negligible — adjacency list of string IDs | | Startup time | 1-2 seconds for model load + batch embedding | | Ongoing compute | Sub-millisecond per search; 5-minute polling introspection (only rebuilds on change) | | Engineering effort | ~550-700 lines of new code across 10 files, plus test files | | New dependency | `@huggingface/transformers` (ONNX runtime) | ## Performance Analysis ### Startup | Phase | Time | Notes | | ------------------------------- | -------- | --------------------------------------------------- | | Model load (first ever) | 5-10s | One-time download of 23MB model | | Model load (cached) | 1-2s | Loading ONNX model from disk cache | | Batch embedding (1,000 entries) | <1s | Single batch through the pipeline | | Graph construction | <100ms | Adjacency list from introspection data | | **Total (warm cache)** | **1-2s** | Non-blocking — keyword search available immediately | ### Search (per query) | Phase | Time | Notes | | --------------------------------- | --------- | ---------------------------------------- | | Query embedding | <10ms | Single text through the pipeline | | Cosine similarity (1,000 entries) | <1ms | Brute-force linear scan of Float32Arrays | | Keyword scoring | <1ms | Token overlap, already implemented | | Glossary lookup | <0.1ms | String matching against curated terms | | Score combination + sort | <0.1ms | Simple arithmetic + array sort | | **Total** | **<15ms** | Dominated by query embedding | ### Memory | Component | Size per 1,000 entries | | --------------------------- | ------------------------------- | | Embedding vectors | ~1.5 MB (384 _ 4 bytes _ 1,000) | | ONNX model | ~23 MB (loaded once, shared) | | Graph adjacency list | ~50 KB | | Glossary entries | ~10 KB | | **Total additional memory** | **~25 MB** | ### Polling - Default interval: 5 minutes (`SEARCH_POLL_INTERVAL_MS`) - Introspection request: ~100-200ms - Hash comparison: <1ms - Rebuild (only on change): 1-2s (same as startup) ## Scaling Characteristics ### Current capacity Brute-force cosine similarity is O(n) but operates on contiguous `Float32Array` memory with simple arithmetic — modern CPUs handle this extremely efficiently. Practical performance: | Schema size | Search time | Memory | | -------------- | ----------- | ------- | | 100 entries | <0.1ms | ~200 KB | | 1,000 entries | <1ms | ~1.5 MB | | 5,000 entries | ~2-3ms | ~7.5 MB | | 10,000 entries | ~5ms | ~15 MB | ### HNSW upgrade path When `VectorIndex` logs its >10K warning, the upgrade path is: 1. Add `hnswlib-js` dependency 2. Implement `HnswVectorIndex` conforming to the same `add/search/clear/size` interface 3. Swap the implementation in `HybridSearch.build()` — no changes to any consumer code The `VectorIndex` interface was designed specifically to make this a drop-in replacement. ### Memory growth Each schema entry adds approximately 1.5 KB of embedding storage (384 dimensions \* 4 bytes). Adding a new data source (e.g., Snowflake with 2,000 columns) would add ~3 MB. The ONNX model memory (~23 MB) is fixed regardless of schema size. ## Breakdown Points & Mitigations ### Model load failure **Risk:** The ONNX model fails to download (first run) or load (runtime error, unsupported platform). **Mitigation:** Degraded mode. `HybridSearch` catches the error, logs once, and redistributes vector weight to keyword scoring. Search still works — it just cannot bridge vocabulary gaps that require semantic understanding. The glossary still provides its curated mappings. On the next poll cycle, embedding is retried automatically. **Impact:** Reduced search quality for vocabulary-mismatch queries. Exact-name and glossary-mapped queries are unaffected. ### 10K+ schema entries **Risk:** If the combined schema across all data sources grows past 10K entries, brute-force search time increases noticeably (>5ms). **Mitigation:** `VectorIndex` emits a warning log at 10K. The HNSW upgrade path is designed and documented. The `VectorIndex` interface does not need to change. **Likelihood:** Low in the near term. The GraphQL schema has hundreds of entries. Snowflake integration (future) could add a few thousand. 10K would require indexing many databases. ### Stale embeddings **Risk:** Schema changes between poll intervals are not reflected in search results. **Mitigation:** 5-minute polling with hash-based change detection. The poll interval is configurable via `SEARCH_POLL_INTERVAL_MS`. Schema changes are typically deployed, not continuous, so a 5-minute window is acceptable. If lower latency is needed, the interval can be reduced or a webhook trigger can be added. ### Glossary drift **Risk:** The curated glossary becomes outdated as the schema evolves — entries reference fields that no longer exist, or new vocabulary gaps emerge that the glossary does not cover. **Mitigation:** `matchGlossary()` validates glossary targets against `candidateIds` (actual schema entries). Targets that no longer exist in the schema are silently ignored. New vocabulary gaps are discovered through agent failure logs (multi-round search patterns) and added to the glossary incrementally. The glossary is a JSON file checked into the repo, so changes go through normal PR review. ### Open handle leaks (polling) **Risk:** `setInterval` timer leaks if the server shuts down without calling `stopPolling()`. **Mitigation:** `stopPolling()` is called during server shutdown. Tests call `stopPolling()` in `afterEach`/`afterAll`. Multiple `startPolling()` calls replace the previous timer (clears the old interval before setting a new one). ## Decision Log | Decision | Rationale | Date | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------- | | In-process MiniLM-L6-v2 over API-based embeddings | No API costs, no network dependency, sufficient quality for schema text, sub-ms inference | 2026-03-21 | | Brute-force cosine over HNSW | Schema size is hundreds of entries; brute-force is <1ms; avoids native dependency; upgrade path documented | 2026-03-21 | | 0.6/0.2/0.2 weight split (vector/keyword/glossary) | Vector provides the strongest novel signal; keyword and glossary are complementary safety nets; tunable via config | 2026-03-21 | | Glossary as JSON file, not database | Curated by engineers, version-controlled, reviewable in PRs, no DB migration needed | 2026-03-21 | | BFS algorithms extracted from Graph | Testable independently, reusable for future algorithms (DFS, topological sort) | 2026-03-21 | | Graph in common/ package | Generic data structure with no AI dependencies; reusable across server and other packages | 2026-03-21 | | Degraded mode over hard failure | Search availability is more important than search quality; glossary + keyword provide a usable baseline | 2026-03-21 | | SHA-256 hash on normalized schema, not raw JSON | GraphQL introspection responses have non-deterministic field ordering; normalizing avoids false-positive rebuilds | 2026-03-21 | | Lazy model initialization | Model loads on first `embed()` call, not at import time; avoids blocking server startup if embedding is not immediately needed | 2026-03-21 | ## Dependencies | Dependency | Purpose | Size | Notes | | --------------------------- | ------------------------------- | -------------------- | ------------------------------------------------------------------------------ | | `@huggingface/transformers` | In-process ONNX model inference | ~23MB model (cached) | ESM module, dynamic import in `TransformersEmbeddingProvider` | | `hnswlib-js` (future) | Approximate nearest neighbors | — | Only needed if schema exceeds 10K entries; not added in initial implementation | No new infrastructure dependencies (no vector database, no external API service, no new containers). ## Testing Strategy ### Unit tests | Module | Test file | Coverage | | --------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | BFS algorithms | `common/src/__tests__/algorithms/bfs.test.ts` | `bfsPath` (shortest path, self-loop, no path, maxDepth, cycles, disconnected), `bfsNeighborhood` (depth-1, depth-2, limit cap, isolated node) | | Graph\ | `common/src/__tests__/graph/graph.test.ts` | addNode/getNode/size, addEdge (directed), findPath (delegates to BFS), neighborhood (returns GraphNode objects), edge cases (missing node, overwrite) | | VectorIndex | `server/src/search/__tests__/vector-index.test.ts` | add/size, cosine similarity ranking, limit, clear, >10K warning log | | Glossary | `server/src/search/__tests__/glossary.test.ts` | loadGlossary (valid file, missing file, malformed JSON), matchGlossary (target match, related match, both, no match, candidateId validation, examples) | | HybridSearch | `server/src/search/__tests__/hybrid-search.test.ts` | Glossary target ranked highest, related below target, degraded mode (null provider), keyword-only scoring, primary priority boost, empty query | | GraphQL adapter | `server/src/ai/tools/graphql/__tests__/search-adapter.test.ts` | buildSchemaGraph (nodes for types/query fields, returns edges, hasField edges, path traversal from query field to nested type) | ### Integration tests - Schema-index polling: `startPolling`/`stopPolling` lifecycle, timer creation/cleanup, replacement on multiple calls - End-to-end search with mock schema: verify vocabulary-mismatch queries return correct results through the full hybrid pipeline ### Regression tests - All existing GraphQL tool and skill tests must continue to pass unchanged - The `searchSchema()` public API returns the same `{ queries: QueryFieldEntry[]; types: TypeEntry[] }` shape ### Test environment considerations - Unit tests for modules that use `EmbeddingProvider` mock the interface (no 23MB model download in CI) - `null` embedding provider forces degraded mode for fast, deterministic tests - Polling tests use `jest.useFakeTimers()` and call `stopPolling()` in `afterEach` to prevent open handle leaks ## Rollout Plan ### Phase 1: Infrastructure (Tasks 1-5) Build the generic search infrastructure with no impact on existing behavior: - BFS algorithms + Graph\ in common/ - EmbeddingProvider interface + VectorIndex - TransformersEmbeddingProvider - Glossary loader + matcher - HybridSearch combiner All new code, no existing files modified. ### Phase 2: Integration (Tasks 6-9) Wire the infrastructure into the GraphQL search path: - Create initial glossary seed file (8 domain entries) - Adapt `search.ts` to use HybridSearch with keyword fallback - Add polling, vector/graph lifecycle to `schema-index.ts` - Enhance `graphql-explore` skill with glossary examples + graph neighborhood Existing public APIs unchanged. Keyword search remains as synchronous fallback. Hybrid search available via async path. ### Phase 3: Validation (Tasks 10-11) - GraphQL adapter unit tests - Full test suite verification (server + common) - Fixup commit if needed ### Future phases (not in scope) - Snowflake schema integration with hybrid search - Tool catalog semantic search - HNSW upgrade if schema exceeds 10K entries - Alternative embedding providers (Bedrock Titan, OpenAI, Cohere) ## Open Questions 1. **Glossary curation workflow** — who is responsible for adding new glossary entries when vocabulary gaps are discovered? Should the agent log suspected vocabulary mismatches (multi-round search patterns) to make gaps easier to find? 2. **Weight tuning** — the 0.6/0.2/0.2 split is a starting point. Should we add observability (log the score breakdown for top results) to enable data-driven tuning? 3. **Snowflake timeline** — when should Snowflake schema be wired into hybrid search? The infrastructure supports it, but the glossary entries and search document builders need to be written. 4. **Model alternatives** — `all-MiniLM-L6-v2` is optimized for general English text. Would a code-aware embedding model (e.g., CodeBERT) perform better for schema names that mix camelCase identifiers with natural language descriptions? 5. **Polling interval** — is 5 minutes the right default? Should polling be disabled in development environments where the schema changes frequently during testing?