# Learn: Coda Evals (how the model is kept honest)

This page teaches how Coda's **eval harness** works and why it is the thing that lets us trust
an LLM inside a system that writes real contracts. It is the conceptual companion to the
[contract-creation learn page](./contract-creation.md), the
[runbook](../../runbooks/coda/contract-creation.md) (the commands), and
[TRD §8](../../technical-projects/coda/TRD.md) (the design).

## Why evals exist at all

Most of the contract-creation workflow is **pure code**: validation, planning, preview, and
execution are deterministic functions of a draft. Pure code is covered by ordinary unit tests
that pass or fail the same way every run.

Exactly **one** step is stochastic: `extract`, the single LLM call that reads a deal memo into
structured fields. An LLM can return slightly different output on the same input, and it can be
*subtly* wrong in ways a type checker never catches: it can read 85% as 80%, copy the wrong
date, or worst of all, invent an account ID that wasn't in the document.

You cannot unit-test "is the model good enough?" with a boolean. You need a **measurement**.
That is what an eval is: a fixed set of inputs with known-correct answers, a scoring function,
and a threshold the score must clear. Evals turn "the model seems fine" into a number you can
watch over time and gate changes on.

> **The core idea:** measure the one part that *can* be wrong, prove everything else with unit
> tests, and the whole workflow becomes trustworthy. Coda's reliability is the product of a
> measured stochastic step and a deterministic everything-else.

## What the harness does

The runner lives at `apps/server/evals/contract-extraction/run.ts`. For each fixture it does
four things:

```mermaid
flowchart LR
    FX["fixture<br/>(memo text + expected draft)"] --> EX["run extract<br/>(real model or fake)"]
    EX --> SC["score fields<br/>vs expected"]
    SC --> GT["check the<br/>validation gate"]
    GT --> PF{"score ≥ threshold<br/>AND gate OK?"}
    PF -->|yes| PASS["PASS"]
    PF -->|no| FAIL["FAIL"]
```

1. **Extract.** Feed the fixture's memo text through the same extraction node the workflow uses
   (Bedrock Claude at temperature 0, forced through the draft schema). A `--fake` mode swaps in
   an extractor that echoes the expected answer, so the harness can self-test to a perfect score
   with no model or AWS credentials.
2. **Score.** Compare the extracted draft against the fixture's expected draft, field by field,
   to a number in 0..1 (`scoring.ts`).
3. **Gate check.** Run the *real* validator on the extracted draft and confirm it reports the
   missing fields and blocking decision the fixture expects.
4. **Verdict.** A fixture **passes** only if `score ≥ threshold` (default **0.9**) **and** the
   gate check holds. The process exits 0 only when every fixture passes, so it drops cleanly
   into CI or a pre-push hook.

Output is per-fixture (`name … 96.4% PASS`, with `↳` lines for each mismatch), then a summary:
`Fixtures: 7/7 passed` and a mean field accuracy.

## How scoring works (and what it rewards)

Field accuracy is **weighted**, not a flat count, because not all fields are equally dangerous
(`scoring.ts`):

- **Identifiers and key fields weigh 2×.** `account_id`, `signing_entity_id`,
  `run_controller_id`, `contract_name`, and `lifecycle_term_start` count double. Getting an ID
  wrong is far worse than missing an optional note.
- **Rates weigh 2× and compare with tolerance.** `term_rate` / `commission` match within
  ±0.001, so floating-point noise never fails a correct answer.
- **Arrays compare as sets.** Country lists, store lists, and transaction types are
  order-insensitive.
- **Conditions compare after canonical ordering.** When every condition carries an explicit
  priority, both sides sort by it first, because the order is then semantically irrelevant.
- **Hallucinated identifiers are punished at the heavy weight.** If the model emits an
  `account_id` / `signing_entity_id` / `run_controller_id` that the document never stated, that
  is scored as a heavy miss and flagged `(HALLUCINATED)` in the output.

That last rule is the most important one in the whole harness. **Inventing an ID is the single
most dangerous failure mode** in a system that writes real contracts, so the metric is shaped to
make the model lose the most points for exactly that. The eval doesn't just reward "mostly
right", it actively steers the prompt toward "leave it blank when unsure."

## The gate check: does the workflow ask instead of guess?

A high field score is not enough. The workflow's safety depends on the model handing
incomplete drafts to the **human review gate** rather than papering over gaps. So each fixture
also declares what the validator should say, and the harness asserts it:

- `expectedMissingFields`: the draft paths that must be reported missing.
- `expectedBlocking`: whether the draft must be blocked from execution.

For example, the `missing-lifecycle-start` fixture is a term sheet that omits the start date and
every platform ID. The *correct* model behaviour is to extract the rate and contract name and
leave the IDs blank, and the *correct* workflow behaviour is to block, listing `account_id`,
`signing_entity_id`, `run_controller_id`, `lifecycle_term_start`, and `default_term_attachments`
as missing. The eval fails if the model guesses an ID **or** if the gate fails to ask for one.

## The fixtures: adversarial on purpose

The seven fixtures in `fixtures/` are written to attack the failure modes that matter, not just
the happy path:

| Fixture | What it stresses |
|---------|------------------|
| `clean-single-term` | The happy path: a complete, well-formed memo extracts cleanly. |
| `distractor-prose` | A memo padded with irrelevant narrative; the model must not be pulled off the real fields. |
| `missing-lifecycle-start` | Omitted start date and all IDs; the workflow must **ask, not guess** (gate blocks). |
| `rate-commission-mismatch` | Rates that don't sum to 100; extraction must copy them **faithfully** and validation must block. |
| `multi-term-multi-condition` | Genuinely different catalogues and rate variations; correct term-vs-condition modelling. |
| `renew-periodically` | Fixed-period renewal; `renewal_type` and `renewal_years` set correctly. |
| `revenue-type-splits` | Streaming / sync / ad-sales splits as **conditions on one term**, not separate terms. |

Note what `rate-commission-mismatch` teaches: the model is graded on **fidelity to the
document**, not on fixing it. It must extract 85% + 20% exactly as written, and the deterministic
validator is what catches that they don't reconcile. Extraction reports; validation judges. The
eval enforces that division of labour.

## How this increases reliability

Putting it together, the eval harness buys several concrete guarantees:

1. **Regression safety on the riskiest change.** The extraction prompt and the draft schema are
   the parts most likely to drift. Any edit to them is gated by re-running the eval, so a "small
   prompt tweak" that quietly degrades ID accuracy can't ship unnoticed.
2. **Hallucination is measured, not hoped against.** Invented IDs lose the most points and are
   labelled in the output, which keeps continuous pressure on the "never fabricate" behaviour
   that protects the live system.
3. **The human gate is proven to fire.** The gate assertions confirm that incomplete documents
   reach a human with the right questions, which is the workflow's central safety property.
4. **A green eval plus green unit tests covers the whole workflow.** Because everything after
   extraction is pure and unit-tested, and extraction is eval-scored, there is no untested gap
   between "the model read the memo" and "the contract was written."
5. **A trend, not a vibe.** Mean field accuracy is a number that can be tracked release over
   release, so reliability becomes observable rather than anecdotal.

## Running and extending it

Run from the repo root (see the [runbook](../../runbooks/coda/contract-creation.md) for detail):

```bash
pnpm eval:contract-extraction              # real model (needs AWS creds)
pnpm eval:contract-extraction -- --fake    # offline self-test, expects 100%
pnpm eval:contract-extraction -- --only=rate-commission-mismatch
pnpm eval:contract-extraction -- --threshold=0.85
```

**To add a case**, drop a JSON file in `fixtures/` with `input.text`, the `expected` draft, and
(optionally) `expectedMissingFields` / `expectedBlocking`. New fixtures are picked up
automatically. The best fixtures encode a *specific* danger you want to guard against forever:
a memo phrasing that once fooled the model, a tricky split, a missing field that must trigger
the gate.

## What evals do and don't cover

- **They cover** the stochastic step: did the model read the document correctly, and does the
  validator react correctly to what it read.
- **They do not cover** the deterministic write path (planning, the four mutations, execution).
  That is unit-tested and additionally guarded at runtime by dry-run and the QA-only gate. Evals
  measure judgement; tests prove mechanics.
- **They are bounded by the fixture set.** Evals are only as good as the cases in `fixtures/`,
  which is why adding a fixture for every new failure mode is part of maintaining the workflow,
  not an afterthought.