# Evals & Quality Assurance

How we measure and protect the quality of search results and agent behavior.

## Overview

Quality assurance operates at three layers, each catching different classes of regression:

```
Layer 1: Search Ranking        — "Did the right tables/fields come back?"
Layer 2: Agent Behavior         — "Did the agent call the right tools and avoid hallucination?"
Layer 3: Regression Detection   — "Did anything get worse since the last deploy?"
```

---

## Layer 1: Search ranking benchmarks

The search service includes an offline benchmark suite that scores keyword + glossary matching against golden queries without a running server, embeddings, or network.

### Golden queries

`apps/search/benchmarks/golden-queries.json` contains labeled queries with expected results. Each entry specifies:

- `query` — the natural language search term
- `expected` — the table/field FQN that must appear in results
- `tags` — categories for filtering (e.g., `revenue`, `contract`, `artist`)
- `mustRank` — the position the expected result must appear at or above

### Running benchmarks

```bash
# Run all benchmarks, compare to baseline
tsx apps/search/benchmarks/run.ts

# Run with per-query detail
tsx apps/search/benchmarks/run.ts --verbose

# Run only revenue-related queries
tsx apps/search/benchmarks/run.ts --tag revenue

# Update the baseline after intentional changes
tsx apps/search/benchmarks/run.ts --update-baseline
```

### Metrics

| Metric        | What it measures                                          | Target    |
| ------------- | --------------------------------------------------------- | --------- |
| **MRR@N**     | Mean Reciprocal Rank — how high the expected result ranks | > 0.90    |
| **Recall@N**  | Whether the expected result appears at all in top N       | > 0.95    |
| **Precision** | Fraction of returned results that are relevant            | Monitored |
| **NDCG**      | Normalized Discounted Cumulative Gain                     | Monitored |

Baselines are stored in `apps/search/benchmarks/baselines/`. When a change improves metrics, update the baseline with `--update-baseline`. If MRR drops, investigate before merging.

---

## Layer 2: Agent behavior evals

Agent behavior evals verify that the AI agent calls the right tools, avoids hallucination, and produces useful answers. These are higher-level than search benchmarks — they test the full chat loop.

### Eval dimensions

| Dimension          | What it checks                                                      | Example assertion                                      |
| ------------------ | ------------------------------------------------------------------- | ------------------------------------------------------ |
| `mustCallTools`    | The agent called the expected tools                                 | "revenue question" → must call `get_account_revenue_*` |
| `mustNotSay`       | The agent did not hallucinate errors or claim tools are unavailable | Must not say "not accessible" when tools are available |
| `responseContains` | The answer includes relevant data                                   | Revenue answer contains dollar amounts                 |
| `sqlMustReference` | Generated SQL references the correct tables                         | Snowflake query references `ROYALTY_STATEMENT_DETAIL`  |
| `noFabrication`    | Every data point in the response came from a tool result            | No numbers appear that weren't in tool call results    |

### Running agent evals (planned)

> **Status:** Not yet implemented. The eval harness is tracked in [todos.md](../todos.md) under "Evals." The design below describes the intended approach.

Agent evals will use the chat streaming API against a running server (QA or local with real Bedrock). The planned workflow:

1. Define eval datasets as (question, expected assertions) pairs
2. Run questions through the streaming chat API
3. Assert against tool-call chains and response content
4. Record results in Langfuse for comparison across runs

### Langfuse integration

Langfuse (self-hosted) provides prompt versioning and trace correlation. Agent evals will be defined as Langfuse datasets with (input, expected output) pairs, enabling comparison across prompt versions and model changes.

---

## Layer 3: Regression detection

### Automated monitors

| Monitor                    | Trigger                                          | Response                                       |
| -------------------------- | ------------------------------------------------ | ---------------------------------------------- |
| `query.rerank.uniform`     | WARN-level log: reranker returned uniform scores | Reranker is broken — check model loading       |
| Search MRR regression      | Benchmark MRR drops below baseline               | Block deploy, investigate query changes        |
| Tool call error rate spike | > 5% errors in 5-minute window                   | Check downstream service health                |
| Bedrock throttling         | `ThrottlingException` in logs                    | Check Bedrock quotas, reduce thinking budget   |
| Agent hallucination report | User feedback (thumbs-down with "wrong data")    | Review conversation in Langfuse, add eval case |

### Glossary target freshness

Glossary entries reference table/field FQNs that can become stale when Snowflake schemas change. Stale targets produce search misses — queries match the glossary term but the boosted target no longer exists.

> **Status:** Automated glossary linting is planned (tracked in [todos.md](../todos.md) under "Glossary target validation at startup"). Currently requires manual verification by diffing glossary JSON targets against the live Snowflake schema.

### Adding a new eval case

When a quality issue is discovered:

1. Reproduce the query that failed
2. Add it to `golden-queries.json` (search) or the agent eval dataset (behavior)
3. Add the expected result / assertion
4. Verify the fix makes the new case pass
5. Update baselines

This "test from the failure" approach ensures every discovered issue becomes a regression guard.

---

## See also

- [Search Architecture](../architecture/search.md) — scoring algorithms, RRF fusion, glossary boosting
- [Testing Guide](testing.md) — unit, functional, and integration test tiers
- [Observability](observability.md) — tracing and monitoring for debugging quality issues