# Evals — `python-service-scan-authz-baseline` Structured evaluations for the `python-service-scan-authz-baseline` skill. Each eval case runs the skill against a synthetic Python microservice fixture and grades the output against a set of assertions, with a baseline run (no skill) for comparison. ## How it works Each eval case spawns a [`create_react_agent`](https://langchain-ai.github.io/langgraph/reference/prebuilt/#langgraph.prebuilt.chat_agent_executor.create_react_agent) with two file tools (`read_file`, `list_directory`) scoped to this skill directory. - **`with_skill`** — the full `SKILL.md` is injected as a `` block in the system prompt. - **`without_skill`** — plain system prompt, no skill context (baseline). After each agent run, a second LLM call grades the output against the assertions in `evals.json` and writes a `grading.json`. A final aggregation step writes a `benchmark.json` comparing pass rates, token usage, and duration between the two configurations. ## Test cases | # | Fixture | Based on | What it tests | |---|---------|----------|---------------| | 1 | `ows-catalog` | ows-product | Fully enforced (`verify_access=True`, `access_log_only=False`) + inline `verify_grass_access`. Tests that the skill identifies active enforcement, coverage gaps (catch-all only), and grass calls. | | 2 | `ows-releases` | ows-track | Enforcement **disabled** (`verify_access=False`) despite a rules file registered via `set_rules_validator`. Tests that the skill correctly flags disabled middleware. | | 3 | `ows-media` | ows-assets | No legacy auth at all; `PdpAuthorizationBackend` **already instantiated**. Tests that the skill doesn't over-prescribe legacy setup and focuses on endpoint-level PP calls. | | 4 | `ows-content` | ows-product-digital | `access_log_only=config.ONLY_LOG_ACCESS_ERRORS` (environment-dependent enforcement) + `only_for_identity` JWT decorator. Tests that the skill flags config-variable risk and mixed auth patterns. | ## Requirements - Python 3.11+ - [`uv`](https://docs.astral.sh/uv/) (`brew install uv` or `pip install uv`) - A provider credential (see below) The runner installs its own virtualenv under `evals/runner/.venv` — no manual setup needed. ### Credentials setup The runners read credentials from `runner/.env`. Copy the shadow file and fill in your keys: ```bash cp .github/skills/python-service-scan-authz-baseline/evals/runner/.env.shadow \ .github/skills/python-service-scan-authz-baseline/evals/runner/.env # then edit runner/.env and add your keys ``` `runner/.env` is gitignored. You only need the key for the provider you plan to use. ## Usage Two runners are available — **GitHub Models** (`copilot_runner.py`, default), **Anthropic** (`claude_runner.py`), and a **mini/haiku** variant of each for cheaper iteration. Pick whichever you have credentials for; the eval logic is identical across all of them. ```bash # Full run — all 4 cases × with_skill + without_skill, then grade and benchmark make eval-copilot make eval-claude # Single case only make eval-copilot-case CASE=enforced-access-rules make eval-claude-case CASE=enforced-access-rules # Skip the baseline (with_skill only — faster for iterating on the skill) make eval-copilot-skill-only # GitHub Models gpt-4.1 make eval-copilot-mini-skill-only # GitHub Models gpt-4.1-mini (cheaper) make eval-claude-skill-only # Anthropic Sonnet make eval-haiku-skill-only # Anthropic Haiku (~$0.16, recommended) # Run a single case make eval-haiku-case CASE=enforced-access-rules # Append results to an existing iteration instead of creating a new one make eval-copilot ARGS="--iter 1" # Use a different model make eval-copilot ARGS="--model openai/gpt-4o-mini" make eval-haiku HAIKU_MODEL=claude-haiku-4-7 # override Haiku model version # Enable debug logging (shows which files expand_files resolved, etc.) make eval-haiku ARGS="-d" make eval-haiku-case CASE=enforced-access-rules ARGS="-d" # Format / lint runner source make fmt make lint ``` ## Output structure Results are written to `evals/workspace/` (gitignored) and organized by iteration: ``` evals/workspace/ └── iteration-1/ ├── benchmark.json ← aggregated pass rate / token / time stats ├── eval-enforced-access-rules/ │ ├── with_skill/ │ │ ├── outputs/response.md ← agent's migration plan │ │ ├── timing.json ← tokens + duration_ms │ │ └── grading.json ← per-assertion PASS/FAIL with evidence │ └── without_skill/ │ ├── outputs/response.md │ ├── timing.json │ └── grading.json ├── eval-logging-only-enforcement-disabled/ ├── eval-no-legacy-auth-pp-already-wired/ └── eval-conditional-enforcement-mixed-patterns/ ``` ### `grading.json` shape ```json { "assertion_results": [ { "text": "Output states that verify_access=True and access_log_only=False", "passed": true, "evidence": "\"enforcement is fully active (verify_access=True, access_log_only=False)\"" } ], "summary": { "passed": 6, "failed": 1, "total": 7, "pass_rate": 0.857 } } ``` ### `benchmark.json` shape ```json { "run_summary": { "with_skill": { "pass_rate": {"mean": 0.83, "stddev": 0.06}, "time_seconds": {...}, "tokens": {...} }, "without_skill": { "pass_rate": {"mean": 0.33, "stddev": 0.10}, "time_seconds": {...}, "tokens": {...} }, "delta": { "pass_rate": 0.50, "time_seconds": 13.0, "tokens": 1700 } } } ``` The `delta` tells you what the skill costs (more tokens, more time) and what it buys (higher pass rate). ## Iterating on the skill The workflow is to: 1. Run `make eval-copilot-mini-skill-only` or `make eval-haiku-skill-only` to score the skill against the eval dataset. 2. For any missed assertion, either update the `assertions` in `evals.json` (if the expectation was wrong) or update the skill to handle the case (if the skill output was wrong). 3. Iterate until `skill-only` scores are near-perfect. 4. Run `make eval-copilot` or `make eval-haiku` to compare scores with and without the skill using a stronger model. See the [agentskills.io eval guide](https://agentskills.io/skill-creation/evaluating-skills) for the complete methodology. ## `evals.json` field reference Each entry in the `evals` array describes one test case: | Field | Type | Description | |-------------------|------------------|----------------------------------------------------------------------------------------------------------------------------| | `id` | string | Slug used to select the case via `make eval-copilot-case CASE=` (or the equivalent `-case` target for any provider) and as the output directory name. | | `name` | string | Human-readable description of what the case tests. | | `service` | string | Name of the fixture service directory under `evals/files/`. | | `prompt` | string | The user prompt passed to the agent. | | `expected_output` | string | Prose description of what a correct response should contain. Used as grader context, not matched literally. | | `files` | array of strings | Glob patterns (relative to the skill root) for files attached to the agent as context. Supports `*` wildcards. | | `assertions` | array of strings | Specific, independently-gradeable claims the response must satisfy. Each assertion is graded true/false by the LLM grader. | ## Providers ### GitHub Models (default) Uses the GitHub Models API (`models.inference.ai.azure.com`) — an OpenAI-compatible endpoint backed by your Copilot subscription. **Requires a fine-grained PAT with `models:read` — a classic PAT will not work.** 1. Go to [github.com → **Settings → Developer settings → Fine-grained tokens**](https://github.com/settings/personal-access-tokens) 2. Click **Generate new token** 3. Under **Permissions → Models** → set to **Read-only** 4. Generate, copy, and export: GITHUB API Token Screenshot ```bash # runner/.env GITHUB_MODEL_API_TOKEN=github_pat_... ``` > ⚠️ **Daily rate limit:** GitHub Models enforces a hard limit of **150 requests per day** > per user per model (`UserByModelByDay`). A full eval run uses 8 agent calls (4 cases × > 2 configs) plus 8 grading calls = **16 requests**. If you hit the limit, the `429` error > will tell you to wait up to 23 hours. To stay within quota: > - Use `make eval-copilot-skill-only` (halves requests to 8) > - Use `make eval-copilot-case CASE=` to run a single case (2 requests) > - Switch to `make eval-claude` for unlimited iteration (Anthropic resets per minute) **Iteration** — use gpt-4.1-mini (cheaper and faster): ```bash make eval-copilot-mini # full run make eval-copilot-mini-skill-only # with-skill only, skip baseline ``` **Full eval / validation** — use gpt-4.1: ```bash make eval-copilot make eval-copilot-skill-only ``` ### Anthropic ```bash # runner/.env ANTHROPIC_API_KEY=sk-... ``` **Iteration** — use Haiku (cheaper and faster): ```bash make eval-haiku # full run (~$0.32) make eval-haiku-skill-only # with-skill only, skip baseline (~$0.16) ``` **Full eval / validation** — use Sonnet to confirm changes hold on a larger model before publishing: ```bash make eval-claude # full run (~$1.34) make eval-claude-skill-only # with-skill only ```