# endpoint_diff

Byte-for-byte before/after comparison of the `tests/integration/endpoints`
suite -- retargeted onto the transfer-ownership profiles and entity ids --
against the local dev server.

It answers one question: **does going from one git ref to another change any
endpoint response?** -- not just the JSON *shape*, but the exact payload bytes.
The two refs are passed explicitly (`run.sh <before-ref> <after-ref>`), so any
two branches or commits can be compared.

It was built to verify that `edd8996d` ("Add historical-ownership-aware
permissions filter (feature-flagged)") leaves every endpoint response unchanged
when the feature flag is off, and is kept as a reusable tool for the next time.

## How it works

1. **Extract** -- `_out/` is wiped clean, `<after-ref>` is checked out and the
   endpoints suite is run once against the dev server with a recorder
   (`recorder_conftest.py`) that wraps `requests.Session.request`. Every HTTP
   call the suite makes is captured and deduped into
   `_out/manifest_original.json` -- a **frozen, code-independent request set**.
2. **Retarget** -- `retarget_manifest.py` rewrites that manifest onto the
   transfer-ownership profiles and entity ids (see [Retargeting](#retargeting)),
   producing `_out/manifest.json` -- the request set every capture replays.
3. **Replay** -- `replay.py` fires that exact manifest at a server and records
   every response to `_out/capture_<label>.json` as three sha256 hashes (raw
   bytes, key-sorted JSON, and order-normalised JSON) -- enough for `compare.py`
   to classify any difference without the body. Pass `--keep-bodies` to also
   store full bodies (far larger captures; only for a small/targeted manifest).
   It imports only `requests`, so it runs identically whatever git revision is
   checked out.
4. **Capture A-B-A** -- the manifest is replayed three times:
   `after_extract` (`<after-ref>`, t0) -> `before` (`<before-ref>`, t1) ->
   `after_replay` (`<after-ref>`, t2). The two `<after-ref>` captures bracket
   the `before` capture in time.
5. **Compare** -- `compare.py` diffs the captures by hash. A `before`/`after`
   difference is blamed on the **commit range** only when `<after-ref>`
   reproduced itself (`after_replay == after_extract`); otherwise the endpoint
   is nondeterministic or the warehouse data drifted, and the difference is
   discounted.

### Why the results are trustworthy

- **Frozen manifest** -- both builds receive byte-identical requests (same URLs,
  headers, bodies), regardless of how the test files differ between the refs.
- **Cache isolation** -- `serve.py` forces `fakeredis`, an in-process cache that
  dies with the server. Each capture starts cold; no cached value leaks from
  one build into another.
- **Determinism floor** -- the after-vs-after pair (`after_replay` vs
  `after_extract`) measures intrinsic nondeterminism (e.g. unordered SQL result
  rows) and data drift. Only differences *above* that floor are attributed to
  the commit range.
- **Single process server** -- reloader off, so it is cleanly killable between
  `git checkout`s.
- **Clean slate** -- `_out/` is wiped at the start of every comparison, so no
  capture, manifest or report from a previous run can be mistaken for this
  run's.

## Retargeting

The recorded manifest carries the endpoints suite's own profiles and products.
`retarget_manifest.py` rewrites every request onto a single fixed input set,
taken from `tests/integration/transfer_ownership/conftest.py`:

- **Profiles** -- each request is fanned out across the four baseline
  InsightsProfiles: employee (`1100`), Thirty Tigers subaccount (`9900028`),
  RGE label (`9900030`), TVT label (`9900032`). The FF-enabled profiles are
  not used -- a flag-off profile is the production code path.
- **Entity ids** -- in both the URL path and the query string: product ids ->
  `956820`, ISRCs -> `USACQ0500001`, (global) participant ids -> `ccf0617f-...`,
  UPCs -> `888880712318`. A POST `{"isrcs": [...]}` body is collapsed to
  `["USACQ0500001"]`.
- **Accounts** -- `/account/<id>/*` and the `account_id` query param are
  retargeted to each profile's backing account (subaccount `7645` / RGE
  `21786` / TVT `15063`), with `account_type` set to match; the employee
  profile is fanned out across all four transfer accounts. The `label_ids` /
  `subaccount_ids` filter params name an account *type*, so they take the
  profile's accounts of that type and are dropped when it has none (a label
  profile has no subaccount).
- **Left untouched** -- date params and ids with no transfer-suite analogue
  (video / channel / store ids). A request that returns empty or 403 is still a
  valid before/after pair.

The fanned-out requests are then deduped (a product-scoped request recorded
against five products collapses to one) and re-id'd. Set
`ENDPOINT_DIFF_NO_RETARGET=1` to skip this step and diff the recorded manifest
as-is.

## Usage

Requires a **clean working tree** (the tool runs `git checkout`). The
version-independent scripts (`serve.py`, `replay.py`, `compare.py`) are staged
into `_out/` before any checkout, so either ref may safely predate this
directory -- the checkout that would remove it cannot break the run.

A fresh comparison always starts clean: the `all` and `extract` phases **wipe
`_out/`** before staging anything, so no capture, manifest or report from a
previous run can leak in. `_out/` is disposable scratch -- nothing durable is
kept there.

```bash
# full run: extract -> before -> after -> compare
tests/integration/endpoint_diff/run.sh <before-ref> <after-ref>

# e.g. compare two branches
tests/integration/endpoint_diff/run.sh \
  before_port_with_ordering_fixes fix-nondeterministic-endpoint-ordering
```

Phases can be run individually. `extract` must run first -- it wipes `_out/`,
records and builds the manifest; `before` / `after` / `compare` then reuse what
it left in `_out/` and do **not** wipe:

```bash
run.sh <before-ref> <after-ref> extract  # checkout <after-ref>, record, retarget, replay -> manifest.json + after_extract
run.sh <before-ref> <after-ref> before   # checkout <before-ref>, replay -> capture_before.json
run.sh <before-ref> <after-ref> after    # checkout <after-ref>,  replay -> capture_after_replay.json
run.sh <before-ref> <after-ref> compare  # -> _out/report.md
```

Knobs (environment variables): `ENDPOINT_DIFF_XDIST` (extract parallelism,
default 15), `ENDPOINT_DIFF_CONCURRENCY` (replay parallelism, default 15),
`ENDPOINT_DIFF_NO_RETARGET` (set to skip retargeting -- diff the recorded
manifest as-is).

## Files

| file | role |
|---|---|
| `serve.py` | dev-server launcher: reloader off, `debug=False`, fakeredis |
| `recorder_conftest.py` | pytest plugin; copied to `tests/integration/conftest.py` during extract, then removed |
| `build_manifest.py` | raw recordings -> `manifest_original.json` (recorded request set) |
| `retarget_manifest.py` | rewrites the manifest onto the transfer-ownership profiles + ids |
| `replay.py` | standalone replayer of `manifest.json` against a server |
| `compare.py` | byte + canonical-JSON diff -> `report.md` / `report.json` |
| `run.sh` | orchestrates the full A-B-A flow between two refs |
| `_out/` | all artifacts (git-ignored); disposable scratch -- wiped at the start of every `all` / `extract` run |

## Output

`_out/report.md` -- verdict, the after-vs-after determinism floor, a category
breakdown of any differing requests (`WHITESPACE` / `ARRAY-ORDER` / `CONTENT` /
`STATUS`), and -- when the captures were taken with `--keep-bodies` -- line-level
body diffs for commit-attributable changes. `compare.py` exits non-zero iff
there is a commit-attributable difference.