# python-skill-eval-runner
Reusable skill-eval harness: a pip-installable library and CLI for running iterate-then-validate evals against Claude Code skills.

## What it does

`skill-eval-runner` measures whether a Claude Code skill actually improves a model's
output. For each eval case it runs an agent twice — **with** the skill's `SKILL.md`
injected and **without** it (the baseline) — grades each run's artifact against atomic
pass/fail assertions, and reports the **with−without delta** as the headline metric.

The workflow it's built around:

- **`skill-eval iterate`** — tune cheaply on a small/fast model (Haiku / gpt-4.1-mini),
  skill-only (no baseline), until skill-only scores are near-perfect.
- **`skill-eval validate`** — sign off on a bigger model (Sonnet / gpt-4.1) with the
  baseline included, to confirm the skill still buys a real lift.

```
skill-eval iterate   [--case ID] [--model ...] [--provider anthropic|github-models]
skill-eval validate  [--case ID] [--model ...] [--provider ...]
skill-eval run       [--no-baseline] [--grade-only] [--iter N] ...   # raw flags
skill-eval grade     --iter N [--case ID]                            # re-grade existing outputs
```

Outputs land under `<workspace>/iteration-N/eval-<id>-<target>/<config>/` with
`outputs/<artifact>`, `timing.json`, and `grading.json`; a top-level `benchmark.json`
aggregates mean/stddev pass-rate, time, tokens, cost, and the delta. `timing.json`
records the agent and grader token usage and cost separately (`cost_usd` vs
`grader_cost_usd`) plus their sum (`total_cost_usd`) — since the grader is a second
model call, possibly on a different model; the benchmark's cost rolls up `total_cost_usd`.

## Configuration

Drop a `skill-eval.toml` (or a `[tool.skill_eval]` table in `pyproject.toml`) next to
the skill. All paths resolve against the config file's directory; every field has a
default matching the conventional `<skill>/evals/` layout, so a suite can run with no
config at all.

```toml
[skill_eval]
skill_file = "SKILL.md"              # injected as <skill> for with_skill runs
suite      = "evals/evals.json"
file_root  = "."                     # base for evals.json `files` globs + read-tool sandbox
workspace  = "evals/workspace"

[skill_eval.artifact]
mode      = "tool"                   # "tool" (agent calls a write tool) | "final_message"
tool_name = "write_report"
filename  = "response.md"

[skill_eval.models]
small   = "claude-haiku-4-5"         # agent model for `iterate`
large   = "claude-sonnet-4-6"        # agent model for `validate`
default = "claude-sonnet-4-6"        # agent model for `run` / `grade`
grader  = "claude-sonnet-4-6"        # grader model (optional; defaults to the agent model)
```

The agent and grader models are resolved independently. Setting `grader` lets you
generate cheaply while grading reliably — e.g. a Haiku agent (`iterate`) judged by a
Sonnet grader, which avoids the small-model grader flakiness (truncated/duplicated
verdicts) you get when one cheap model does both. Override per run with
`--model` / `EVAL_MODEL` (agent) and `--grader-model` / `EVAL_GRADER_MODEL` (grader).

Credentials are read from the environment (or a local `.env`):

- `anthropic` → `ANTHROPIC_API_KEY`
- `github-models` → `GITHUB_MODEL_API_TOKEN`
- `bedrock` → the **boto3 default credential chain** (no API key). Pick the account with
  `AWS_PROFILE`, and set `BEDROCK_REGION` (falls back to `AWS_REGION` / `AWS_DEFAULT_REGION`,
  then `us-east-1`):

  ```sh
  AWS_PROFILE=<your-bedrock-profile> BEDROCK_REGION=us-east-1 \
    skill-eval validate --provider bedrock
  ```

## Installation

There are two ways to install `skill-eval-runner` as a dependency in your application.

### Install using pip (`requirements.txt`)

1. Add `-i https://pypi.theorchard.io/pypi/` to the top of `requirements.txt`.
2. `env/bin/pip install -r requirements.txt`

### Install using Poetry (`pyproject.toml`)

#### pypi.theorchard.io

1. Update `pyproject.toml` to include:

```toml
[[tool.poetry.source]]
name = "pde"
url = "https://pypi.theorchard.io/pypi/"
priority = "supplemental"
```

2. Add this to the `[tool.poetry.dependencies]` section in `pyproject.toml`:
```toml
[tool.poetry.dependencies]
...
skill-eval-runner = {version = "^0.1.0", source = "pde"}
```

3. Run `poetry lock`

#### git+ssh

1. Add this to the `[tool.poetry.dependencies]` section in `pyproject.toml`:

```toml
[tool.poetry.dependencies]
...
skill-eval-runner = { git = "ssh://git@github.com/theorchard/python-skill-eval-runner#v0.1.0"}
```

2. Run `poetry lock`

## Contributing

### Dependencies

This library uses Poetry for dependency management. Please use `poetry add` when adding new dependencies. Always add/commit changes made to the `pyproject.toml` and `poetry.lock` files.

Avoid being overly-restrictive when adding installation requirements. [This](https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/) has a good overview of considerations when specifying what is required to install skill-eval-runner. A few key quotes:

> It’s best practice to indicate any known lower or upper bounds
> It is not considered best practice to use install_requires to pin dependencies to specific versions, or to specify sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from gaining the benefit of dependency upgrades.

Do indicate if a dependency belongs to development (and thus, not required when the library is installed) using `poetry add <the new dependency> --group dev`.

On a regular basis, run `poetry update` to update to the latest versions of all dependencies in the `poetry.lock` file. Specific dependencies can be updated with `poetry update <the specific dependencies>`.

### `make` targets

You can always use `make help` to view all the documented targets for working in this library. Some common ones are:

```
make env
make lint
make fmt
make test_unit
```

### Packaging/Versioning

Use semver and [Jenkins Pipeline](https://pipeline.theorchard.io/job/publish-pypi-package-v2/) to build/publish new versions of this library to the private PyPI repository at [pypi.theorchard.io](https://pypi.theorchard.io) (currently only available if you are on our VPN).

After a release tag has been pushed to Github, use the Github UI to Create a Release from the tag. Start from the autogenerated notes, and add salient details to the change(s) being added. This will be used by clients of the library to determine what to expect when upgrading (easy vs breaking?).