# ows-ai-evals

A centralised AI eval service for testing MCP (Model Context Protocol) surfaces and other AI-powered tools across the PDEGO platform.

## What this is

As PDEGO ships MCP servers (starting with the [apollo-mcp pilot](https://app.notion.com/p/37697177520f81e2b622d7465b5a8b76)), we need a way to test them automatically. Standard integration tests cover the deterministic layers (auth, API, MCP protocol), but there are two additional non-deterministic layers that need a different approach:

- **Layer 4 — Tool selection:** Given a natural language prompt, does the LLM call the right tool?
- **Layer 5 — Answer correctness:** Does the LLM's final answer meet whatever criteria the caller cares about?

This service handles both. MCP repos send a single POST request with their prompt config; the service runs each prompt N times against a fixed Bedrock model, checks tool selection and any configured LLM-as-judge criteria, and returns a structured results object with pass rates per prompt.

The caller resolves any ground-truth values itself (e.g. by querying its own GraphQL API) and bakes the expected value directly into a judge's instructions — this service has no GraphQL client and doesn't know about any MCP's data model.

Every prompt run is traced to Datadog LLM Observability as a workflow span, tagged with the caller's pipeline context (`mcp_name`, `prompt_id`, `pipeline_id`, `pipeline_name`, `build_id`). Tool-selection and judge results are submitted as Datadog evaluations, so pass rates are queryable/filterable per prompt over time — not just per pipeline run.

## Architecture

Evals run against a real LLM (plus LLM-as-judge calls per prompt), so a run can take long enough
to blow past an HTTP client/proxy timeout. `/run-eval` is async: it kicks off the work and returns
immediately with a `job_id`; the caller polls a second endpoint for the result.

```
MCP repo (pytest)          ows-ai-evals (this service)              External
─────────────────          ───────────────────────────              ────────
POST /run-eval        →    1. create job record (DynamoDB)      →   DynamoDB (job store)
{ config, prompts }        2. schedule background execution
                      ←    { job_id, status: "pending" }              (returns immediately)

                           [in the background, per prompt/run/judge:]
                           3. call Bedrock with prompt + tools    →   AWS Bedrock (Claude)
                           4. run each configured judge           →   AWS Bedrock (Claude)
                           5. tag + submit evaluations             →   Datadog (LLM Obs)
                           6. write result to job record            →   DynamoDB (job store)

GET /run-eval/{job_id} →   read job record                       →   DynamoDB (job store)
                      ←    { job_id, status, result | error }
assert pass_rates
```

One service call (plus polling) per pipeline run. The calling repo has zero LLM logic and no
AWS/Datadog credentials — just config in, poll, results out, assert.

The job runs in the same container as the API (a background task, not a separate worker
process/queue) — status is persisted to DynamoDB so a poll can land on any replica, not just the
one that happened to run the job.

## Request shape

```json
{
  "mcp_name": "pdego-audience",
  "mcp_endpoint": "http://localhost:8080/mcp",
  "auth_token": "eyJhbGciOi...",
  "runs": 3,
  "pipeline": { "pipeline_name": "pdego-audience-ci", "pipeline_id": "1234", "build_id": "42" },
  "prompts": {
    "top_fan_growth": {
      "prompt": "Find the artist in my roster with the most new fans this week",
      "expected_tools": ["get_artist_fans"],
      "judges": [
        {
          "name": "correct_artist",
          "instructions": "Does the answer correctly identify Taylor Swift as the artist?"
        }
      ]
    }
  }
}
```

- `expected_tools: []` means the prompt should **not** trigger any tool call.
- `expected_tools` (non-empty) means every listed tool must be called at some point during the run — extra tool calls beyond that list are allowed and don't fail the check. This matters for MCPs like Apollo's dynamic mode, where the model calls exploration tools (`introspect`, `search`, `validate`) before the tool that actually produces the answer (`execute`) — list only the tool(s) that must be called, not the full exploration path.
- `judges` is optional — omit it for prompts that only test tool selection.
- `judge.name` is submitted directly as the Datadog evaluation label, so it must start with a letter and contain only letters/numbers/underscores (validated at the schema level).
- `runs`/`auth_token`/`auth_user` can be set per-prompt to override the top-level default.
- `auth_token` is a bearer JWT for the target MCP (e.g. the same token you'd put in `Authorization: Bearer ...` when hitting it locally) — this service passes it straight through, it doesn't mint or validate it.
- `auth_user` is a fallback: if no `auth_token` is set, the service resolves one via Auth0 ROPG for the named user (currently stubbed, pending Secrets Manager wiring). At least one of `auth_token`/`auth_user` must be set, per-prompt or top-level.

## Response shape

`POST /run-eval` returns `202` immediately:

```json
{ "job_id": "59fa5743-0814-4428-9b3a-49dbd2eaafdc", "status": "pending" }
```

Poll `GET /run-eval/{job_id}` (same `X-API-Key` header required) until `status` is no longer
`pending`/`running`:

```json
{
  "job_id": "59fa5743-0814-4428-9b3a-49dbd2eaafdc",
  "status": "succeeded",
  "result": { "mcp": "test-mcp", "model": "us.anthropic.claude-haiku-4-5-20251001-v1:0", "prompts": { "...": "..." } },
  "error": null
}
```

`result` is the same shape the old synchronous response used to return, just wrapped — it's only
populated once `status` is `succeeded`. On `failed`, `error` holds the exception message instead.
A `404` means the job doesn't exist or has expired (`JOB_TTL_SECONDS`, 6h by default).

## Current state

### Done
- FastAPI service scaffolding (cookiecutter-fastapi template)
- API key auth (Secrets Manager-backed, with dev bypass via `DEV_API_KEY`)
- `/admin/refresh-api-key` endpoint for key rotation
- Judges-based request/response schemas (`EvalRequest`, `EvalResponse`, `PromptConfig`, `JudgeConfig`, `JudgeResult`)
- `/run-eval` endpoint wired up end-to-end, async: returns `202` + `job_id`, actual work runs as an
  in-process background task
- `GET /run-eval/{job_id}` — polls job status/result, backed by DynamoDB (`clients/jobs.py`,
  `services/jobs.py`) so any replica can serve the poll regardless of which one ran the job.
  Result is stored as a single JSON string attribute rather than native DynamoDB Map/List types,
  since it's only ever read back whole (see `clients/jobs.py` docstring for why)
- Real Bedrock client (`clients/llm.py`) — `run_prompt` for tool selection, `run_judge` for LLM-as-judge verdicts. Verified live against `us.anthropic.claude-haiku-4-5-20251001-v1:0`
- Real MCP client (`clients/mcp.py`) — Streamable HTTP transport (`initialize` → `notifications/initialized` → `tools/list`), authenticated via the caller-supplied `auth_token`. Fetches the target MCP's real tool manifest so Bedrock picks from actual tools, not a stub. Tool-selection only — doesn't execute `tools/call`, since correctness here is "did the LLM pick the right tool," not "was the tool's real output correct"
- Datadog LLM Observability — each prompt run is a tagged workflow span; Bedrock `converse` calls are auto-instrumented as nested LLM spans via `ddtrace`'s botocore integration (needs `ddtrace-run` / `make trace-dev`)
- Evaluation submission — `tool_selection` + one label per judge name, tagged with `source`, `mcp_name`, `prompt_id`, `pipeline_id`/`pipeline_name`/`build_id`. Verified live end-to-end
- Unit tests for auth, infra, jobs, and eval endpoints/service (74 passing, fully mocked — no live AWS/Datadog needed)
- Local DynamoDB (`dynamodb-local`, via `make up`) so the async job flow works end-to-end without
  the real table existing yet

### Stubbed — pending
| Stub | Waiting on |
|------|-----------|
| `_get_auth_token()` | Auth pattern confirmed — ROPG via Auth0, needs Auth0 test user credentials in Secrets Manager. Only used as a fallback when the caller doesn't supply `auth_token` directly |

### Open questions
- **Model access** — only Haiku has Bedrock model access enabled on the dev AWS account so far; Sonnet is visible via `list-foundation-models` but blocked on `converse` (`AccessDeniedException` — Marketplace subscription not enabled)
- **MCP endpoint** — `qa-apollo-mcp` Fargate service not yet deployed; local dev possible via `make run-local` in the `apollo-mcp` repo
- **Auth token generation** — ROPG flow confirmed as the pattern (matches existing integration test suites); needs Auth0 credentials stored in Secrets Manager under `ai-evals/users/{user_key}`
- **Real jobs table** — no `ows-ai-evals` infra exists yet in `terraform-infra` at all; the
  `ows-ai-evals-jobs` DynamoDB table + IAM permissions for the app's task role need provisioning
  there before `/run-eval` works in qa/prod (works today in local dev against `dynamodb-local`)

## Running locally

`cp .env.shadow .env` first either way (fill in `DD_API_KEY` if testing LLM Obs; needs AWS creds
with Bedrock access).

### Dockerized (recommended — includes the async job store)

```bash
make up      # builds + starts ai-evals-dev, dynamodb-local, and a one-shot jobs-table-init step
make down    # tears it all down
```

`jobs-table-init` creates the local jobs table before `ai-evals-dev` starts, so `make up` fails
loudly if that setup breaks rather than the first `/run-eval` call failing instead. Serves on
`:8888` (mapped from the container's `:5000`). `DYNAMODB_ENDPOINT_URL` in `.env` points at
`dynamodb-local` — that hostname only resolves inside the compose network, so this path is
Docker-only; it's unset for the deploy target and in real environments, where the app talks to the
real DynamoDB table instead.

### Bare host (`make trace-dev`)

```bash
make trace-dev           # runs under ddtrace-run so Bedrock calls are auto-instrumented; serves on :5000
```

`uv run python dev.py` also works but skips `ddtrace-run`, so Bedrock calls won't be traced. Since
`dynamodb-local` isn't reachable as `dynamodb-local:8000` from the host, `/run-eval` will fail at
job creation unless you either run `docker compose up dynamodb-local` separately and override
`DYNAMODB_ENDPOINT_URL=http://localhost:8000` in your shell, or point at a real provisioned table.

### Test the endpoint

```bash
curl -X POST http://localhost:8888/run-eval \
  -H "X-API-Key: dev-key" \
  -H "Content-Type: application/json" \
  -d '{
    "mcp_name": "test-mcp",
    "mcp_endpoint": "http://localhost:8080/mcp",
    "auth_token": "dev-jwt",
    "runs": 1,
    "pipeline": {"pipeline_name": "manual-test", "pipeline_id": "1"},
    "prompts": {
      "capital_check": {
        "prompt": "What is the capital of France? Answer in one word.",
        "expected_tools": [],
        "judges": [
          {"name": "mentions_paris", "instructions": "Does the answer correctly state that the capital of France is Paris?"}
        ]
      }
    }
  }'
# => {"job_id": "...", "status": "pending"}

curl http://localhost:8888/run-eval/<job_id> -H "X-API-Key: dev-key"
# poll until status is "succeeded" or "failed"
```

(Use port `:5000` instead of `:8888` if running via `make trace-dev` on the bare host — see
`scripts/sample-run-eval.sh` for a runnable version of this.)

## Running tests

```bash
uv run pytest tests/unit/ -v      # unit tests (no live services needed)
uv run pytest tests/integration/  # requires live service on localhost:5000
```

## Setup

Requires Python 3.14 and [uv](https://docs.astral.sh/uv/getting-started/installation/).

```bash
brew install pyenv && pyenv install 3.14 && pyenv local 3.14
uv sync
```