# Performance

## 2026-06-16 — `claude-sonnet-4-6`, total cost $2.85

Run after adding two new behaviors to the skill: (1) `verify_rules_access_standalone` standalone
mode detection, and (2) existing `auth_backend` singleton — don't override, create a new
`migration_auth_backend`. Added one new fixture (`ows-track-standalone`) and one new assertion to
`no-legacy-auth-pp-already-wired`.

| Case                                                                        | With skill      | Without skill   | Delta    |
|-----------------------------------------------------------------------------|-----------------|-----------------|----------|
| Fully enforced access_rules middleware (Flask)                              | 12/12 (100%)    | 6/12 (50%)      | +50%     |
| Logging-only mode with enforcement disabled (Flask)                         | 12/12 (100%)    | 8/12 (67%)      | +33%     |
| No legacy auth — PP backend already wired (Flask)                           | 13/13 (100%)    | 10/13 (77%)     | +23%     |
| Conditional enforcement with mixed auth patterns (Flask)                    | 13/13 (100%)    | 8/13 (62%)      | +38%     |
| FastAPI service with mixed PP migration states                               | 13/13 (100%)    | 8/13 (62%)      | +38%     |
| standalone-access-rules: verify_access=False + verify_rules_access_standalone | 6/6 (100%)    | 5/6 (83%)       | +17%     |
| **Overall**                                                                 | **69/69 (100%)** | **45/69 (65%)** | **+35%** |

All six cases score 100% with skill on Sonnet. The new `standalone-access-rules` case is solid —
6/6 with skill and the single without-skill miss is the standalone-mode posture assertion (expected:
the baseline has no knowledge of the standalone pattern). The new singleton assertion on
`no-legacy-auth-pp-already-wired` passed 13/13.

**Haiku flakiness note (singleton assertion):** The new "create `migration_auth_backend`, don't
override existing singleton" assertion is flaky on Haiku (`claude-haiku-4-5`) — it passed 1/3 runs
at 92% overall. On Sonnet it passes 100% consistently. This is a Haiku capability ceiling, not a
skill or assertion problem; no changes needed.

Haiku skill-only sanity passes (`make eval-haiku-skill-only`, `claude-haiku-4-5`): the full suite
scored ~96–99% across three runs. The `standalone-access-rules` case scored 6/6 (100%) on all three
runs, confirming the new fixture and assertions are clean.

> **Note:** The vast majority of PP migrations are Flask apps, where all five Flask cases now score
> 100% with skill. The FastAPI suite has a single fixture (`ows-product-staging`); its run-to-run
> variance reflects n=1 sample size. The previous two with-skill misses on FastAPI (run-to-run
> attribution variance) resolved this run — we are not tuning the skill to force those last
> assertions to avoid overfitting to one fixture.

---

## 2026-06-02 — `claude-sonnet-4-6`, total cost $2.32

Run after the `MigrationAuthorizationBackend` rewrite, the phase renumber (Phase 1 baseline /
Phase 2 shadow / Phase 3 enforce), and the FastAPI persona-assertion split.

| Case                                                     | With skill      | Without skill   | Delta    |
|----------------------------------------------------------|-----------------|-----------------|----------|
| Fully enforced access_rules middleware (Flask)           | 12/12 (100%)    | 5/12 (42%)      | +58%     |
| Logging-only mode with enforcement disabled (Flask)      | 12/12 (100%)    | 8/12 (67%)      | +33%     |
| No legacy auth — PP backend already wired (Flask)        | 12/12 (100%)    | 10/12 (83%)     | +17%     |
| Conditional enforcement with mixed auth patterns (Flask) | 13/13 (100%)    | 8/13 (62%)      | +38%     |
| FastAPI service with mixed PP migration states           | 11/13 (85%)     | 9/13 (69%)      | +16%     |
| **Overall**                                              | **60/62 (97%)** | **40/62 (65%)** | **+32%** |

The two with-skill misses are both FastAPI scan assertions where Sonnet *used* a fact without
restating/attributing it: it identified `GET /hello/` as excluded but didn't cite `main.py`, and it
described the JWT middleware without an explicit "JWT always required for non-excluded endpoints"
sentence. Both vary run-to-run and are unrelated to the migration-tooling content. The two split
`conftest.py` persona-prerequisite assertions both passed on Sonnet, confirming the assertion split
+ skill strengthening hold on the larger model.

Haiku skill-only sanity passes (`make eval-haiku-skill-only`, `claude-haiku-4-5`): the full suite
scored ~98–100% across runs, and the FastAPI case scored 100% / 100% / 100% across three runs after
the assertion split + skill strengthening. The phase renumber and the assertion split broke no
graded assertions.

> **Note:** The vast majority of PP migrations are Flask apps, where all four cases score 100% with
> skill. The FastAPI suite has a single fixture (`ows-product-staging`), so its remaining
> run-to-run variance reflects an n=1 sample rather than a skill gap — the two misses are facts the
> model used but didn't restate/attribute. We can add more FastAPI fixtures (different posture mixes)
> in the future to decrease the variance of the FastAPI assertions; we are deliberately not tuning
> the skill to force those last assertions, to avoid overfitting to one fixture.