# Experiment 1: saturated-queue rolling deploy — findings

**Run date/time (UTC):** 2026-04-17T16:22:49Z

**Setup confirmed:**
- [x] Service at desired_count = 3
- [x] Steady state confirmed before trigger (single PRIMARY deployment)
- [x] Worker cycling verified (iteration ~98–100 at start, ~250ms cycle)
- [x] tail_state.sh running
- [x] tail_logs.sh running

**Trigger:**
- Command: `./scripts/force_deploy.sh` (force-new-deployment without registering new task def; `update_env.sh` was blocked by `iam:PassRole`)
- Trigger time (UTC): `2026-04-17T16:22:49Z`

**Convergence (from ECS service events and describe-services):**
- New PRIMARY deployment created at trigger; 3 new tasks went PENDING → RUNNING within ~15s
- At `16:24:13`: all 3 old tasks exited (within 260 ms of each other); ECS reported "stopped 1 running tasks" and "unable to scale in due to 2 tasks under protection"
- At `16:25:36`: deployment completed, steady state
- **Total: 2:47 from trigger to steady state**

**Old-revision task fate (worker-logs.jsonl):**
- 3 old tasks, all exited via `enable_failed` + `reason=DEPLOYMENT_BLOCKED`
- Timestamps: 16:24:13.49, 16:24:13.65, 16:24:13.76 (clustered within 260 ms)
- Iteration numbers at exit: 1402, 1412, 1402 (≈1,400 cycles each before DEPLOYMENT_BLOCKED fired)
- All exited cleanly via `worker_exit` `reason=loop_exited`

**DEPLOYMENT_BLOCKED observations:**
- Total `enable_failed` events: **3** (exactly one per old task)
- From revision 1 (old); no enable attempts from new tasks failed
- Time from trigger to first DEPLOYMENT_BLOCKED: ~84 s
- All three old tasks hit it in the same ~260 ms window

**SIGTERM observations:**
- Total `sigterm_received` events: **0**
- ECS did not send SIGTERM to any task during this deploy — task protection + DEPLOYMENT_BLOCKED did all the work

**Surprises / notes:**
- **DEPLOYMENT_BLOCKED actually happened, clustered sharply.** It didn't trickle — all three old tasks went in the same sub-second window after 84 s of the deploy being "unable to scale in." ECS appears to gate the condition check across the whole service and release it atomically.
- **ECS reported "2 tasks under protection" at the same second it managed to stop 1.** Consistent with: protected count was 3, ECS's scale-in logic needed to remove 3, protection kept blocking it until whatever internal counter flipped.
- **New tasks (revision 1, same task def as old due to `force_deploy`) never saw DEPLOYMENT_BLOCKED.** They cycled through without issue.
- **~84 s latency** from trigger to DEPLOYMENT_BLOCKED firing is notable — the `protected_count > desired_count` trigger isn't instantaneous. Likely tied to ECS scheduler tick + some internal dampening.

**Verdict:**
- [x] Expected outcome observed: deploy completed cleanly via DEPLOYMENT_BLOCKED. Bare task protection pattern (no version polling) handled the saturated rolling deploy in this environment.

## Decision implications for CDAM-3806

- **Version polling is NOT load-bearing** for the saturated-deploy case — at least at this scale (3 tasks, 250 ms cycle, `ExpiresInMinutes = 5`). DEPLOYMENT_BLOCKED fired reliably and convergence was ~3 min.
- **Version polling is still useful as** (a) defensive layer if ECS's DEPLOYMENT_BLOCKED timing changes at higher scale, (b) observability — lets us correlate "we noticed a new task def" with worker exits.
- Caveats for production interpretation:
  - Production `ExpiresInMinutes = 30`; experiment was `= 5`. Larger value means more time between re-enables where AWS's counter sees higher protected count. Could either speed up or slow down the same mechanism — need to watch first prod deploy.
  - Experiment was 3 tasks; prod will be 0–5. Different counts might stress different parts of the AWS scheduler.
  - Workers here had identical code in old vs new tasks (same image, same task def revision). Real deploys with code changes shouldn't change the mechanism but worth verifying.

## Raw data

- `service-state.jsonl` — ~85 entries (multi-line pretty-printed; `jq -s` to parse)
- `service-events.jsonl` — ~72 entries
- `task-state.jsonl` — ~144 entries
- `task-definitions.jsonl` — 1 task def revision (rev 1) captured
- `service-deployments.jsonl` — empty (DescribeServiceDeployments API didn't return data; investigate later)
- `worker-logs.jsonl` — thousands of entries across 6 tasks

## Teardown confirmation

- [ ] Service still at desired_count = 3 (will remain through experiment 2)
- [ ] Results kept in-place for experiment-2 comparison