# TASK: `skill-eval-runner` — a reusable skill-eval harness + `skill-eval-creator` skill

> **Status:** Brain dump for a fresh Claude session to spin up the project. Nothing is
> implemented yet. This file is the hand-off; read it top-to-bottom, then read the reference
> source called out in [§2](#2-what-exists-today-the-prototype-to-generalize), then propose a
> concrete plan before writing code.

---

## 1. The vision (why we're doing this)

We built a one-off eval harness for a single skill (`python-service-scan-authz-baseline` in the
`pp-pilot` repo). Other teams saw it and want the same thing for **their** skills. The ask is to
**generalize the harness into a reusable Python library**, publish it to our **internal PyPI**, and
ship a companion **`skill-eval-creator` skill** that teaches Claude how to author new eval suites
against the library.

The workflow people specifically like and want preserved:

- **Iterate cheaply** on a skill with a small/fast model (Anthropic **Haiku**, or GitHub Models
  **gpt-4.1-mini**), running `--no-baseline` (skill-only) for speed.
- **Validate** the change holds on a **bigger model** (Sonnet / gpt-4.1) with the baseline
  (without-skill) run included, to confirm the skill still buys a real lift.

Two deliverables:

1. **`skill-eval-runner`** — a pip-installable Python library + CLI (`skill-eval`). The harness itself.
   **Scaffold it with the org-standard [`cookiecutter-python-library`](https://github.com/theorchard/cookiecutter-python-library)
   pattern — not by hand.** This is a confirmed decision (see [§9](#9-packaging--internal-pypi-org-standard-pathuse-the-cookiecutter));
   it dictates the Poetry toolchain and the `python-<name>` repo layout.
2. **`skill-eval-creator`** — a Claude Code skill that knows the library's config schema + CLI and
   helps a team scaffold an eval suite (fixtures, `evals.json`, assertions) for an existing skill,
   then drive the iterate→validate loop.

**This task is just the planning doc.** The demo we want out of the spin-up session: take the
library, point it at a *second* skill (not the authz one), and show the full iterate→validate loop
working from a clean `pip install skill-eval-runner`. We have two concrete second skills to target —
real PDE marketplace skills from other teams (see [§13](#13-worked-examples-improving-real-pde-skills)).

---

## 2. What exists today (the prototype to generalize)

All paths below are in the `pp-pilot` repo. Note `skills/` is a symlink to `.github/skills/`.

**Read these first — they ARE the spec:**

- Runner source: `/Users/tcalhoun/work/pp-pilot/.github/skills/python-service-scan-authz-baseline/evals/runner/`
  - `runner_common.py` — **the whole engine** (~510 lines): file tools, agent runner, grader,
    benchmark aggregation, CLI `main()`. Everything generic-able lives here.
  - `claude_runner.py` — Anthropic provider entry point (`ChatAnthropic`, reads `ANTHROPIC_API_KEY`).
  - `copilot_runner.py` — GitHub Models provider entry point (`ChatOpenAI` against
    `https://models.github.ai/inference`, reads `GITHUB_MODEL_API_TOKEN`).
  - `pyproject.toml` — deps: `langchain`, `langgraph`, `langchain-openai`, `langchain-anthropic`,
    `langchain-core`, `environs`; dev: `ruff`. (Prototype package is already named `skill-eval-runner`
    — the name we kept; see §11.)
  - `.env.shadow` — credential template (`ANTHROPIC_API_KEY`, `GITHUB_MODEL_API_TOKEN`).
- Make targets (the UX people actually use): `/Users/tcalhoun/work/pp-pilot/.github/skills/python-service-scan-authz-baseline/Makefile`
  — `eval-claude`, `eval-haiku`, `eval-copilot`, `eval-copilot-mini`, each with `-case` and
  `-skill-only` variants. `EVAL_MODEL`/`HAIKU_MODEL`/`COPILOT_MINI_MODEL` overrides.
- Eval suite + docs: `.../evals/evals.json`, `.../evals/README.md`, `.../evals/PERFORMANCE.md`,
  fixtures under `.../evals/files/<service>/`.

### How it works today (architecture to preserve)

1. **Eval suite** = `evals.json` with an `evals: [...]` array. Per case:
   `id`, `name`, `service`, `prompt`, `expected_output` (grader context, not matched),
   `files` (glob patterns relative to skill root, injected as `<context>` + readable via tools),
   `assertions` (list of independently-graded true/false claims).
2. **Two configs per case**: `with_skill` (full `SKILL.md` injected as a `<skill>` block in the
   system prompt) and `without_skill` (plain baseline). The **delta** is the headline metric.
3. **Agent**: `langchain.create_agent(model, tools, system_prompt)` with read-only file tools
   (`read_file`, `list_directory`) scoped to the skill dir, plus a **`write_auth_report`** tool the
   agent calls once with the final artifact.
4. **Grader**: a second LLM call, `model.with_structured_output(...)`, scores each assertion
   PASS/FAIL with quoted evidence. Strict prompt ("no benefit of the doubt").
5. **Outputs**: `evals/workspace/iteration-N/eval-<id>-<service>/<config>/` with
   `outputs/response.md`, `timing.json` (tokens, duration, cost), `grading.json`. A top-level
   `benchmark.json` aggregates mean/stddev pass-rate, time, tokens, cost, and the with−without delta.
6. **Pricing**: hardcoded `PRICING` table, longest-substring match on model name → cost estimate.
7. **CLI flags**: `--case`, `--model`, `--grade-only`, `--no-baseline`, `--iter`, `-d/--debug`.

---

## 3. What's skill-specific today and must be generalized

The engine is ~90% generic already. The coupling to the authz skill:

| Coupling | Where | Generalization |
|---|---|---|
| `write_auth_report` tool + hardcoded `AUTH.md` artifact name | `make_tools`, `run_agent` | Config-driven **artifact contract**: tool name + filename, OR "no artifact, grade the final message." See [§5](#5-the-generic-agent--report-contract). |
| Baseline system prompt mentions "Python microservice authorization" | `run_agent` | Make both system prompts configurable (templates with `{skill}` slot); ship sensible defaults. |
| Paths assume `<skill>/evals/{evals.json,workspace,files}` and `<skill>/SKILL.md` | module-level constants | Resolve from a **suite config** (skill path, evals path, workspace path) with the current layout as the default convention. |
| File tools scoped to `SKILL_DIR` | `make_tools` | Scope to a configurable fixture/skill root. |
| Provider entry points are separate scripts | `claude_runner.py`, `copilot_runner.py` | Fold into a **provider registry** behind one CLI (`skill-eval --provider anthropic|github-models`). Keep `make_model` factory pattern. |
| `PRICING` table | `runner_common.py` | Keep built-in table, allow override/extension via config; warn (don't crash) on unknown models. |
| Package name `skill-eval-runner` | `pyproject.toml` | **Keep** the name `skill-eval-runner`; add console script `skill-eval` (`[tool.poetry.scripts]`). |

Keep as-is (already general and good): the two-config delta model, glob file injection, strict
structured-output grader, the iteration/workspace layout, the benchmark stats.

---

## 4. Package shape (as scaffolded by the cookiecutter)

> **Scaffolded 2026-06-04** via the org cookiecutter into `python-skill-eval-runner/` (lives here in
> the collab planning home for now; moves to its own `python-skill-eval-runner` GitHub repo per §9).
> The cookiecutter dictates a **flat package layout** (`skill_eval_runner/`, *not* `src/`) and derives
> the import package name from `library_name` → **`skill_eval_runner`**. We follow it, not the earlier
> `src/skill_eval/` sketch. Generated baseline (Poetry + ruff + mypy-strict + pytest CI) is green.

```
python-skill-eval-runner/          # cookiecutter output (repo root)
  pyproject.toml                   # name = "skill-eval-runner"; [tool.poetry.scripts] skill-eval = "skill_eval_runner.cli:main"
  Makefile                         # env / lint / fmt / test_unit / test_cov  (cookiecutter targets)
  .bumpversion.cfg                 # semver + git tag (Jenkins publish-pypi-package-v2)
  .github/workflows/main.yml       # CI: ruff + mypy-strict + pytest matrix (3.11/3.12 × ubuntu/macos)
  skill_eval_runner/               # flat package (NOT src/) — cookiecutter convention
    __init__.py                    # public API: run_suite(), grade(), load_config(); __version__
    config.py                      # EvalSuiteConfig + loader (pydantic): paths, artifact, models, prompts, pricing
    suite.py                       # evals.json schema (pydantic models, validation)
    files.py                       # expand_files() — glob fixture injection + sandbox guard
    pricing.py                     # pricing table + cost calc (extensible)
    agent.py                       # run_agent() + tool factory (generic artifact contract, §5)
    grading.py                     # grader (structured output)
    benchmark.py                   # stats + benchmark.json
    workspace.py                   # iteration dirs, output writers, run_and_save/grade_only
    cli.py                         # argparse; subcommands below; console entry `main()`
    providers/
      __init__.py                  # registry: name -> Provider (make_model factory + default/small/large models)
      anthropic.py
      github_models.py             # OpenAI-compatible
      bedrock.py                   # AWS Bedrock (langchain_aws.ChatBedrockConverse) — see §11
  tests/unit/                      # unit tests over pure fns + a fake/stub model (no API calls)
  examples/                        # one subdir per real skill we improve — see §13
    scaffold-suite-app/            # eval suite for the suite-toolkit skill (pdego-marketplace#19)
    migrate-subgraph-integration-tests/  # eval suite for the graphql-toolkit skill (#16)
  README.md
```

### CLI sketch (presets encode the workflow people like)

```
skill-eval iterate   [--model ...] [--case ID] [--provider ...]   # skill-only, small model default
skill-eval validate  [--model ...] [--case ID] [--provider ...]   # full + baseline, big model default
skill-eval run       [--no-baseline] [--grade-only] [--iter N] ... # the raw current behavior
skill-eval grade     --iter N [--case ID]                          # re-grade existing outputs
```

- `iterate` defaults: provider-appropriate small model (Haiku / gpt-4.1-mini), `--no-baseline`.
- `validate` defaults: big model (Sonnet / gpt-4.1), baseline ON.
- `EVAL_MODEL` env override preserved; per-provider default models preserved.
- Config discovery: `skill-eval` looks for `skill-eval.toml` (or `[tool.skill_eval]` in pyproject) in the
  cwd / skill dir, falling back to the current `<skill>/evals/` convention so existing layouts work.

### Config sketch (`skill-eval.toml`)

```toml
[skill_eval]
skill_file = "SKILL.md"              # injected as <skill> for with_skill runs
suite      = "evals/evals.json"
file_root  = "."                     # base for evals.json `files` globs + read-tool sandbox.
                                     #   Default "." = the skill dir (matches the prototype, where
                                     #   `files` patterns are like "evals/files/<svc>/main.py").
workspace  = "evals/workspace"

[skill_eval.artifact]
mode = "tool"                        # "tool" | "final_message"
tool_name = "write_report"           # generic rename of write_auth_report
filename  = "response.md"

[skill_eval.models]
small = "claude-haiku-4-5"           # used by `iterate`
large = "claude-sonnet-4-6"          # used by `validate`
```

> **Note on `file_root`** (resolved during extraction): the prototype scoped the read tools *and*
> the glob-injection base to the **skill root**, with `files` patterns written relative to it
> (`evals/files/...`). We keep that as the default (`file_root = "."`) rather than the earlier
> `fixtures_root = "evals/files"` sketch, so the existing authz suite runs unchanged (the §10.2
> regression anchor).

---

## 5. The generic agent → report contract

The trickiest generalization. Today the agent must call `write_auth_report`; the grader reads that
artifact. Other skills produce different artifacts (a CSV, a patch, a plan, plain prose). Proposal:

- **`mode = "tool"`** (default): the harness registers a single `write_report(content)` tool with a
  configurable name/description; the agent calls it once; that content is graded. This is what makes
  outputs deterministic (don't grade chatter). Allow a configurable artifact filename/extension.
- **`mode = "final_message"`**: no artifact tool; grade the last non-tool assistant message. Simpler
  for skills whose output is just prose.
- **Custom tools (stretch)**: allow a suite to register extra read-only tools (e.g. a `grep` tool,
  or a fake API) via an entry-point/plugin hook. Keep v1 to the two read tools + the report tool.

Whatever we choose, the grader stays artifact-agnostic — it grades a string against assertions.

---

## 6. The plugin (env bootstrap + `skill-eval-creator`)

The Claude Code plugin has **two responsibilities**:

### 6.0 Bootstrap capability (env setup)

A skill that gets a developer from zero to running evals:

- Ensure the right Python via **`pyenv`** (install the version, set a local `.python-version`).
- Create/activate a virtualenv (or `poetry install` if the consuming repo uses Poetry).
- **Install the runner from internal PyPI** (`pip install skill-eval-runner` against
  `pypi.theorchard.io` — needs VPN; document the index URL / `pip.conf` or `--index-url`).
- Verify with `skill-eval --help` (or equivalent) so the user knows it's ready.

### 6.1 `skill-eval-creator` capability

A skill (ships via our plugin mechanism — see [§7](#7-distribution-library-vs-plugin))
that authors eval suites for an *existing* skill. It should:

1. **Scaffold** the suite layout (`evals/`, `evals.json`, `evals/files/<fixture>/`, `skill-eval.toml`)
   and a starter `README.md`/`PERFORMANCE.md`.
2. **Build fixtures**: read the target skill's `SKILL.md`, identify the input shapes it operates on,
   and synthesize realistic fixtures (one per meaningfully-distinct posture/scenario).
3. **Write assertions** following the hard-won rules in [§8](#8-lessons-to-bake-in-opinions). It must
   know they have to be *atomic and independently gradeable*.
4. **Drive the loop**: tell the user to run `skill-eval iterate` (small model) until skill-only is
   near-perfect, then `skill-eval validate` (big model) for the with/without delta; interpret
   `benchmark.json`; update `PERFORMANCE.md`.
5. **Know the config schema + CLI** so it generates valid `skill-eval.toml` and `evals.json`.

Reference the existing skill's docs as the gold-standard example to imitate:
`.../evals/README.md` (methodology) and `.../evals/evals.json` (assertion style).

---

## 7. Distribution: library vs plugin (answering the user's "library or plugin?")

Both, with a clean split:

- **The runner is a Python library** on internal PyPI. Teams add it as a dev dependency and run the
  `skill-eval` CLI. This is the right shape — it's code, it has deps, it has a CLI.
- **`skill-eval-creator` is a Claude Code skill**, distributed through whatever plugin/marketplace
  mechanism PDE already uses (cf. the `pp-pilot`, `sdlc`, `dd-plugins` plugins already installed in
  this environment). The skill *uses* the library but is shipped separately.

They're co-designed and versioned together (the skill references library CLI/flags), but a team can
adopt the library without the skill.

---

## 8. Lessons to bake in (opinions, hard-won on the authz skill)

These came out of real iteration and should be encoded in both the `skill-eval-creator` skill's
guidance and the library README:

- **Iterate small, validate big.** Haiku/mini + `--no-baseline` for the tuning loop; Sonnet/gpt-4.1
  + baseline for sign-off. Cost ratio is ~10×; don't burn the big model on iteration.
- **Assertions must be atomic.** One claim each, independently gradeable. We hit a *compound*
  assertion ("references personas AND notes the prerequisite") that turned an all-or-nothing flaky
  score — splitting it fixed the signal. The grader prompt even says assertions should be
  "specific, independently-gradeable claims."
- **Test useful output, not phrasing.** Some assertions graded *attribution/restatement* ("cite
  main.py", "restate JWT-always-required") of facts the model already used correctly. These produce
  grader-strictness noise. Prefer assertions about substance.
- **Don't overfit to the last assertions.** With few fixtures (we had **n=1** for FastAPI),
  run-to-run misses are *sample variance*, not skill gaps. The fix for variance is **more fixtures**,
  not tuning the skill to satisfy a stochastic grader. Forcing verbatim lines can even make the skill
  *wrong* on edge cases. Goodhart: once you tune the skill to the eval, the eval stops being an
  independent check.
- **Strengthen the skill only when the requirement is genuinely useful.** We made a real onboarding
  note reliable by having the skill emit it verbatim in the report template — legitimate because the
  note has user value. Don't do this for grader-phrasing misses.
- **The with−without delta is the headline.** A high absolute score with a small delta means the
  skill isn't doing much. Always run the baseline at validation time.
- **Negative assertions are valuable** ("output does NOT include migration steps for the health
  endpoint"). Keep them.
- **Keep a living `PERFORMANCE.md`** per skill: model, date, cost, the with/without/delta table, and
  honest notes about known variance.
- **Grader non-determinism is real** — assertion counts and individual verdicts drift run-to-run.
  Report ranges, not single numbers; don't chase the last point.

---

## 9. Packaging & internal PyPI (org-standard path — use the cookiecutter)

**Scaffold the library with the internal `cookiecutter-python-library` tool, not by hand.**
Local checkout: `/Users/tcalhoun/work/cookiecutter-python-library` (repo:
`theorchard/cookiecutter-python-library`).

- Run `make cookies` (requires Poetry installed). Prompts: `library_name` (omit the `python-`
  prefix, e.g. `skill-eval-runner`) and `description`.
- This **dictates the toolchain: Poetry** — migrate the prototype's `uv`/`pyproject` setup to the
  cookiecutter's Poetry layout rather than keeping uv.
- Move the cookiecutter output into a new GitHub repo named **`python-<library_name>`**, created via
  `terraform-infra` → `terraform-github`. Set `branch_protection_enforce_admins = false` so Jenkins
  can push the version bump + git tags.
- **Internal PyPI**: `pypi.theorchard.io` (VPN-only). Publishing is via the Jenkins
  `publish-pypi-package-v2` pipeline (semver inputs: major/minor/patch/rc/release). Register the new
  repo by adding its name to the `poetry_repos` list in `python-deployment-utils`
  `pypi/docker/entrypoint.sh`.
- Pin langchain ecosystem ranges carefully — `langchain>=1.x` had API churn (`create_agent` moved).
- Console entry point: e.g. `skill-eval = skill_eval.cli:app` (confirm final name — see §11).
- CI: lint (ruff config exists), unit tests against a **fake/stub model** (no API calls), and a
  smoke test of the `grade` subcommand on a committed fixture iteration.

---

## 10. Suggested milestones for the spin-up session

1. **Extract & rename**: copy `runner_common.py` into `src/skill_eval/*`, split by concern, rename
   package to `skill-eval-runner`, get `ruff` + a trivial unit test green.
2. **De-couple**: config loader, generic artifact contract ([§5](#5-the-generic-agent--report-contract)),
   provider registry behind one CLI. Make the **existing authz suite run unchanged** through the new
   CLI (regression anchor — same `benchmark.json` shape, similar numbers).
3. **CLI presets**: `iterate` / `validate` / `run` / `grade`.
4. **Worked examples**: build an eval suite under `examples/<skill>/` for each of the two real PDE
   marketplace skills in [§13](#13-worked-examples-improving-real-pde-skills), proving the full loop
   from a clean install. One skill already ships hand-written evals (formalize + improve them); the
   other has none (scaffold from scratch with `skill-eval-creator`).
5. **`skill-eval-creator` skill**: author it, dogfood it by regenerating the demo suite.
6. **Internal PyPI publish** + a short adopter README ("add dev dep, drop `skill-eval.toml` +
   `evals/`, run `skill-eval iterate`").

---

## 11. Open questions for the human

**Resolved (via the cookiecutter / org standards — see §9):**
- Internal PyPI = `pypi.theorchard.io` (VPN); publish via Jenkins `publish-pypi-package-v2`.
- Build tooling = **Poetry** (cookiecutter-dictated). Repo = standalone `python-<name>` via
  terraform-infra (the `technicalelvis/collab` scaffold here is just the planning home for TASK.md).
- **Naming — DECIDED (2026-06-04): `skill-eval-runner`.** Applied consistently: cookiecutter
  `library_name = skill-eval-runner`, repo `python-skill-eval-runner`, import package
  `skill_eval_runner`, CLI `skill-eval`, companion skill `skill-eval-creator`. (Matches the
  prototype's existing package name; the `pde-eval-*` draft is retired.)

**Still open:**
- **Skill/plugin distribution**: which plugin/marketplace does the plugin ship through? Match the
  mechanism behind the existing `pp-pilot`/`sdlc` plugins.
- **Providers**: **DONE (2026-06-04)** — Anthropic, GitHub Models, **and AWS Bedrock** all ship.
  `providers/bedrock.py` registers a `Provider` whose `make_model` builds
  `langchain_aws.ChatBedrockConverse`; auth is the **boto3 default credential chain** (no API-key
  env var), and the region comes from `BEDROCK_REGION` → `AWS_REGION` → `AWS_DEFAULT_REGION` →
  `us-east-1`. Defaults are **cross-Region inference profile IDs** (`us.anthropic.claude-sonnet-4-6`,
  `us.anthropic.claude-haiku-4-5`); the exact version-dated profile ID must match what's enabled in
  the account / the Bedrock IAM policy — override via `--model` / `EVAL_MODEL` / `[skill_eval.models]`.
  No new `PRICING` rows were needed: the longest-substring matcher already resolves
  `us.anthropic.claude-sonnet-4-6-…` to the existing `claude-sonnet-4` entry.

  **AWS access (Terraform).** Running `--provider bedrock` needs an identity with **inference-only**
  Bedrock permissions — `bedrock:InvokeModel` + `bedrock:InvokeModelWithResponseStream` on the
  foundation-model **and** `inference-profile` ARNs for each Claude model used (cf.
  `terraform-infra/qa/ows-coda/iam.tf`). Locally that's your AWS profile/SSO role with that policy;
  in CI/containers it's the task role. Region is constrained to **us-east-1 / us-west-2** (Bedrock
  model availability). Bedrock model access must also be enabled in the account.

  An **anticipated Terraform scaffold** for this lives in `terraform-infra/permissions-platform/{prod,qa}/skill-eval-runner/`
  (staging copy mirroring the real monorepo layout — `terraform fmt`-clean; move into
  `theorchard/terraform-infra` when ready). It provisions the inference-only Bedrock managed policy
  per PP account and attaches it to the shared cross-account **`generic-engineer-role`** (so
  engineers get Bedrock without a bespoke role; CI can attach the exported `bedrock_policy_arn`
  too). Inference-profile ARNs build from `aws_caller_identity` (no hardcoded account IDs). See
  that dir's `README.md`.
- **Artifact contract default**: tool-based (`write_report`) vs final-message — which is the better
  default for the broadest set of skills?
- **Backward compatibility**: keep the old `make eval-*` targets working in `pp-pilot`, or migrate
  that skill to the published library as the first real adopter? (Recommend: migrate it — it becomes
  the proof the generalization didn't regress.)

---

## 12. Definition of done for the demo

- `pip install skill-eval-runner` from internal PyPI in a clean venv.
- A team drops `skill-eval.toml` + `evals/evals.json` + `evals/files/...` next to their `SKILL.md`.
- `skill-eval iterate` runs skill-only on a small model; `skill-eval validate` runs full + baseline on a
  big model and emits `benchmark.json` with a with−without delta.
- The `skill-eval-creator` skill can scaffold all of that for a brand-new skill from scratch.
- The original authz suite still runs through the library with equivalent results (no regression).

---

## 13. Worked examples: improving real PDE skills

An engineer on another team asked how `skill-eval-runner` + the eval skill can be used to improve the
skills shipped in these two `theorchard/pdego-marketplace` PRs. So we ship an **`examples/`
directory with one subdir per skill**, each a self-contained, runnable demonstration of the eval
skill leveraging the library to measure and improve that skill. These double as the spin-up demo
([§10.4](#10-suggested-milestones-for-the-spin-up-session)) and as adopter-facing tutorials.

The two were deliberately chosen because they sit at **opposite ends of the eval-maturity spectrum**:

### 13.1 `examples/scaffold-suite-app/` — *improve an existing eval suite*

- **Source:** `pdego-marketplace#19` — adds the **`suite-toolkit`** plugin with the
  **`scaffold-suite-app`** skill (scaffolds a `frontend-<name>` Orchard Suite app: pnpm + Biome +
  the latest `@theorchard/suite-*` packages, via an interactive questionnaire).
- **Why it's a good example:** the PR **already ships a hand-written `evals/evals.json`** (71 lines)
  — so this subdir shows how to **port an existing, ad-hoc eval suite onto the library**, add a
  `skill-eval.toml`, run it through `skill-eval iterate`/`validate`, and **improve the assertions**
  using the [§8](#8-lessons-to-bake-in-opinions) lessons (atomic claims, substance over phrasing,
  the with−without delta). The artifact contract here is interesting: the skill's output is a
  scaffolded file tree, not prose — a good test of the generic artifact contract
  ([§5](#5-the-generic-agent--report-contract)).

### 13.2 `examples/migrate-subgraph-integration-tests/` — *create an eval suite from scratch*

- **Source:** `pdego-marketplace#16` — adds the **`graphql-toolkit`** plugin with the
  **`migrate-subgraph-integration-tests`** skill (migrates Apollo Federation subgraph integration
  tests onto `@theorchard/graphql-integration`, handling the full jest/vitest × jwt/no-jwt matrix).
- **Why it's a good example:** the PR ships **no evals at all** — so this subdir is the end-to-end
  **`skill-eval-creator` from-scratch story**: read the skill's `SKILL.md` + reference templates,
  synthesize fixtures (one per cell of the jest/vitest × jwt/no-jwt matrix — a naturally good fixture
  set, addresses the n=1 variance lesson), write atomic assertions, and drive iterate→validate.

### What each subdir contains

```
examples/<skill>/
  README.md          # what the skill does, what we're evaluating, how to run it
  skill-eval.toml      # config pointing at the fixtures/suite/workspace
  evals/
    evals.json       # the eval cases (ported for 13.1, authored for 13.2)
    files/<fixture>/ # fixtures synthesized per scenario
  PERFORMANCE.md     # with/without/delta table per model + date (living doc)
```

> **Note on coupling:** the example skills live in `pdego-marketplace`, not in this repo. The
> `examples/` subdirs vendor only the **eval artifacts** (suite, fixtures, config, perf notes) plus a
> pointer to the source PR/skill — not the skills themselves. Keep them runnable in isolation so a
> fresh `pip install skill-eval-runner` + `cd examples/<skill>` + `skill-eval iterate` just works.