# How the testing will actually go

Mental walkthrough of what happens at each step, for preserved context.
Read alongside `README.md` (which has the exact commands) and the
lambda-assets `.claude/cdam-3806-fargate-worker-design.md` (which has the
design rationale).

## Full test flow, linear

1. **Atlantis applies [terraform-infra#33140](https://github.com/theorchard/terraform-infra/pull/33140)**
   → ECR + cluster + service (desired=0) + IAM + SG + log group created.
   No tasks running. ECR repo empty.

2. **Build and push the worker image** (`build_and_push.sh v1`)
   → ECR has `:v1` and `:latest`, both pointing at the same digest.

3. **Scale service to 3** (`set_desired.sh 3`)
   → ECS launches 3 tasks from task def revision 1. Each task pulls
   `:latest` from ECR, starts the Python worker, registers SIGTERM handler,
   fetches task metadata, starts cycling enable/work/disable.
   Worker logs begin flowing to CloudWatch.

4. **Run experiments 1 → 4 in order**, each documented in README.

5. **Teardown** (final step, after findings recorded)
   → scale to 0, delete terraform directory, PR, atlantis destroys
   everything. Prune local images. Archive or delete results.

## What happens during experiment 1 (saturated deploy)

Starting state:
- 3 tasks running revision 1, each cycling ~250ms (200ms work + 50ms idle)
- Each task enables protection, works 200ms, disables, idles 50ms, repeats
- ECS scheduler sees 3 protected tasks, desired = 3. No scaling actions
  pending.

**Trigger**: `update_env.sh 200 50 5` — registers revision 2 (same env values,
different revision number) and points the service at rev 2.

**ECS scheduler response** (expected timeline):
- T+0s: service.task_definition updated to rev 2. New deployment entry
  appears with status=PRIMARY; old deployment flips to status=ACTIVE.
- T+0–30s: ECS launches new tasks per deployment_maximum_percent=200, so
  up to 3 new tasks come up in addition to the 3 old ones (total 6).
- Each new task starts at rev 2, pulls the image, begins cycling. New
  tasks log with `"revision":"2"`.

**The critical question**: what happens to the 3 old rev-1 tasks?

- **Path A (bare protection works)**: ECS catches an unprotected window
  on each rev-1 task, sends SIGTERM during their ~50ms idle gap. Workers
  see `sigterm_received`, finish their shutdown, exit. Rev-1 tasks
  terminate one by one. Deploy completes in seconds to minutes.
- **Path B (deployment blocked)**: when total protected count reaches
  desired (3) AND new tasks try to enable, AWS returns DEPLOYMENT_BLOCKED.
  Whichever revision sees it first, that task exits. Eventually balance
  settles. Old tasks may or may not be the ones exiting.
- **Path C (stall)**: rev-1 tasks are protected, ECS can't SIGTERM them
  within the tiny unprotected window. Their `ExpiresInMinutes=5` window
  gets reset on every re-enable, so protection never expires. New tasks
  keep hitting DEPLOYMENT_BLOCKED and exiting. Deploy never converges.

Only way to distinguish: watch the timeline. If deploy finishes < 5 min,
it's A or B. If it finishes 5–30 min, ExpiresInMinutes expiry is the
mechanism (path C "self-heals" once protection expires). If it never
finishes, full stall — but our 5-min ExpiresInMinutes means even full
stall resolves within a few cycles of that.

**Production implication** — at production `ExpiresInMinutes=30`:
- If the experiment shows path A/B: we're fine, deploys converge fast.
- If experiment shows path C + expiry-driven convergence: at prod,
  deploys would take 30+ min to self-resolve. Version polling is the
  right addition to bound that.

## What happens during experiment 2 (idle deploy)

Starting state:
- 3 tasks running revision 3 (after experiments 1 transitioned us to
  rev 2, and experiment 2's setup bumped us to rev 3 with idle values)
- Each task: enable → 20s work → disable → 20s idle → repeat
- The unprotected window between enable cycles is now ~20s

**Trigger**: `update_env.sh 20000 20000 5` → registers rev 4.

**Expected timeline**:
- T+0s: rev 4 PRIMARY, rev 3 ACTIVE
- T+0–20s: new rev-4 tasks come up, begin cycling
- T+~20s: each old rev-3 task completes its current cycle,
  disables protection, enters idle
- T+~20–40s: ECS sees rev-3 tasks unprotected, sends SIGTERM
- Tasks exit cleanly via the SIGTERM handler

Deploy should converge in 1–3 minutes. If it doesn't, something is
broken that experiment 1 didn't surface (e.g., SIGTERM not reaching
Python — unlikely given PID-1 setup).

## What happens during experiment 3 (saturated scale-in)

Starting state:
- 3 tasks at rev 5 (or whatever we're at after experiment 2's trailing
  transition), saturated cycle

**Trigger**: `set_desired.sh 1` — API call sets service.desiredCount = 1.
No task def change, no new revision.

**Expected timeline**:
- T+0s: desiredCount = 1, runningCount = 3. ECS scheduler sees
  `protected_count (3) > desired (1)`.
- T+0–250ms: the next task to call `enable()` hits
  `protected_count_after_attempt (would be 4 or stays at 3 depending
  on AWS's counting)` which exceeds desired. Actually AWS counts
  currently-enabled protection, not attempts. When a task finishes
  its current cycle and calls enable() again, protected=3 already,
  desired=1, 3>1, DEPLOYMENT_BLOCKED returns.
- Worker logs `enable_failed` with `reason=DEPLOYMENT_BLOCKED`, exits.
- Now protected=2, desired=1, still 2>1. Next worker to call enable()
  hits DEPLOYMENT_BLOCKED, exits.
- Now protected=1, desired=1, 1>1 is false. Remaining worker succeeds
  and continues cycling.

Should converge in <1 minute. If it doesn't fire, the mechanism differs
from our model (investigation needed).

## What happens during experiment 4 (idle scale-in)

Mirror of experiment 3 but workers are idle (long unprotected windows).

- T+0s: desiredCount = 1
- T+0–20s: ECS sees 2 tasks in idle state (unprotected), sends SIGTERM
  to 2 of them
- Workers handle SIGTERM, exit

Should converge in <1 minute. DEPLOYMENT_BLOCKED should NOT fire —
SIGTERM is the primary path here.

## Data we're capturing

Per experiment, three JSONL streams + a findings template:

1. **service-state.jsonl** — every 2s: `runningCount`, `desiredCount`,
   `pendingCount`, `deployments[].{id, status, taskDefinition,
   runningCount, rolloutState, rolloutStateReason}`

2. **service-events.jsonl** — ECS scheduler event messages (including
   rare ones like *"(service ...) was unable to scale in due to task
   scale-in protection"*)

3. **task-state.jsonl** — every 2s for each task (RUNNING + up to 20
   recently-STOPPED): `lastStatus`, `desiredStatus`, `stopCode`,
   `stoppedReason`, timestamps

4. **worker-logs.jsonl** — per-iteration enable/work/disable/exit events
   with task metadata (family, revision, task_arn) so old vs new are
   distinguishable.

## Interpreting results — the decision tree

After all 4 experiments:

```
experiment 1 converged < 5 min?
├── yes → bare task protection handles saturated deploys
│         → version polling in lambda-assets#618 is optional defence
│         → could remove it to match AWS's sample pattern, BUT: still
│           useful as a belt-and-suspenders observability hook. Leave.
│
├── converged 5–30 min (protection expiry drove it)
│         → at prod ExpiresInMinutes=30, deploys would be slow
│         → version polling is the right optimization
│         → KEEP version polling; document this in PR description
│
└── didn't converge even after expiry (path C locked in)
          → bare pattern fundamentally insufficient
          → version polling is LOAD-BEARING, not optional
          → KEEP version polling; consider more aggressive patterns
            (SCRIPT_COUNT-style iteration limit)

experiment 3 DEPLOYMENT_BLOCKED fired?
├── yes → scale-in path works as modeled
│         → no production changes needed
│
└── no  → mechanism differs from our model
          → investigate before prod rollout

experiments 2 and 4 converged fast?
├── yes → SIGTERM baseline path works
│         → confirms the idle-window fallback is reliable
│
└── no  → something is broken upstream of the experiment
          → e.g., SIGTERM handler not firing, PID-1 issue, etc.
          → fix before running saturated experiments
```

## What "done" looks like

- 4 results directories with complete JSONL data + filled-in findings.md
- A short "executive summary" findings.md at the top level that combines
  all 4 into verdicts for CDAM-3806 decisions
- Teardown checklist (in main README) executed to completion
- The verdicts inform whether lambda-assets#618 keeps or drops version
  polling before it goes to prod