# ECS task protection experiment — findings summary

Run date: **2026-04-17**. Full matrix of scale, deploy-config, and phase-timing
variants complete.

## Experiments run

### Saturated rolling deploys

| Scenario | N | Config | 1st DEPLOYMENT_BLOCKED | Spread | Convergence | Mechanisms observed |
|---|---|---|---|---|---|---|
| Exp 1 | 3 | 200% / no jitter | 84 s | 260 ms | 2:47 | 3 DEPLOYMENT_BLOCKED |
| Repeat A | 3 | 200% / no jitter | 94 s | 160 ms | 3:10 | 3 DEPLOYMENT_BLOCKED |
| Repeat B | 3 | 200% / no jitter | 92 s | 217 ms | 2:55 | 3 DEPLOYMENT_BLOCKED |
| Scale=5 | 5 | 200% / no jitter | 89.5 s | 233 ms | 3:01 | 5 DEPLOYMENT_BLOCKED |
| Scale=10 (no retry) | 10 | 200% / no jitter | ~90 s | — | 3:19 | 9 DEPLOYMENT_BLOCKED + 4 throttle enable_errors |
| Scale=10 retry | 10 | 120% / no jitter | 93.6 s | 423 s (waves) | 10:34 | 12 DEPLOYMENT_BLOCKED (wave pattern) |
| Scale=20 | 20 | 200% / no jitter | 96.2 s | 277 ms | 3:14 | 20 DEPLOYMENT_BLOCKED |
| Scale=50 | 50 | 200% / no jitter | 94.9 s | 81 s | 3:14 | 48 DEPLOYMENT_BLOCKED + 19 SIGTERM + 10 throttle errs |
| Scale=50 random jitter | 50 | 200% / random jitter 0-250ms | 89.4 s | 10.6 s | ~3 min | 48 DEPLOYMENT_BLOCKED + 4 SIGTERM + 10 throttle errs |
| Scale=50 bimodal | 50 | 200% / bimodal 50% phase offset | 92.8 s | 12.2 s | ~3 min | 32 DEPLOYMENT_BLOCKED + 15 TASK_STOPPING + 17 SIGTERM + 18 throttle errs |
| Scale=100 | 100 | 200% / no jitter | 98.3 s | 15.1 s (78 events) | **DID NOT CONVERGE** (100+ min) | 40 DEPLOYMENT_BLOCKED + 38 TASK_STOPPING + 31 SIGTERM + 296 enable_errors + 437 disable_errors (throttling cascade) |

### Scale-in

| Scenario | N | Result |
|---|---|---|
| Exp 3: saturated 3→1 | 3→1 | TASK_STOPPING_OR_STOPPED on 2 of 3 tasks within 31 ms. 12 s convergence |

### Additional

| Scenario | N | Config | Result |
|---|---|---|---|
| Exp 1b: REAL task-def revision change | 20 | 200% / rev 4 → rev 5 via `update_env.sh` | 20/20 old tasks exited via DEPLOYMENT_BLOCKED, 0 SIGTERMs, 0 throttle errors, converged 196 s |

### Pending (deemed inferable, not run)

- Exp 2 (idle deploy), Exp 4 (idle scale-in), Exp 5 (long-running work), Exp 6 (scale to 0) — inferable from the empirical data we have

## Key findings

### 1. The ~90 s scheduler tick is scale-invariant and config-invariant

Across 10 saturated-deploy runs spanning N=3 to N=100, 200% and 120% deploy configs, synchronized and jittered phases:

**Time to first DEPLOYMENT_BLOCKED lands in 84-98 s every time** — range 14 s, mean ~91 s, ~6% CV.

This is an AWS-side scheduler property, not something we control. Safe to treat as a constant: plan deploy convergence around "90 s tick + task exit time + new-task stabilization" ≈ **3 minutes in steady state**.

### 2. Spread is synchronization-dependent, not scale-dependent

Early runs (N=3/5/20) all had <300 ms spread across DEPLOYMENT_BLOCKED firings. Scale=50 synchronized had **81 s spread** — a major anomaly. But with random phase jitter at the same scale, spread collapsed to **10.6 s**.

Bimodal phase offset test confirms: spread correlates with how synchronized workers are, not with N. Clumped-phase scenarios have many workers aligned on the same "unprotected window" moments; random phase scatters this out.

**Production relevance:** real workers are message-driven and naturally random-jittered. Production should look like the random-jitter row — tight spread, few SIGTERMs.

### 3. ECS respects the task protection contract

Detailed analysis of all SIGTERM events at scale=50 synchronized:

- 19/19 SIGTERMs arrived **after `disable_success`** (protection was off)
- 0/19 arrived while protection was enabled
- Time since last disable: 367 ms to 13.8 s

**ECS waited for protection to release before SIGTERM.** What looked like "ECS giving up and overriding protection" was actually ECS being opportunistic — stopping tasks during natural unprotected windows. The unprotected windows were elongated by agent endpoint throttling at scale (retry chains up to 4+ seconds on disable/enable).

### 4. Both DEPLOYMENT_BLOCKED and TASK_STOPPING_OR_STOPPED fire on saturated deploy at scale

At scale=3/5/20 we only saw DEPLOYMENT_BLOCKED. At scale=50 we started seeing both. At scale=100 the mix was nearly even:

- 40 DEPLOYMENT_BLOCKED (service-wide counter tipped)
- 38 TASK_STOPPING_OR_STOPPED (ECS selected specific tasks to stop)

**Implication:** `TASK_STOPPING_OR_STOPPED` is NOT just a scale-in signal — it also fires during rolling deploys when ECS picks specific tasks to terminate. Worker code must handle both reason codes identically (release + exit).

### 5. Agent endpoint has a rate limit that matters only at extreme cycle rates

At 200 ms cycles × 10+ concurrent workers, we started seeing `ThrottlingException` on `UpdateTaskProtection`. Retry logic (3 retries, exponential 0.5/1/2 s) handled most of it at scale ≤50.

At scale=100 with 200% deploy config (200 concurrent during deploy), the retry budget was insufficient: 296 enable_errors + 437 disable_errors, cascading worker exits, deploy stuck at 98/100 running after 100+ minutes.

**Production relevance:** this is an experimental artifact only. Production cycles are message-duration (seconds), so per-worker request rate is ~0.2 req/s. At max=5 workers, the combined rate is ~100× below any plausible throttle threshold.

### 6. 200% vs 120% deploy config changes mechanics, not fundamentals

- 200%: single big wave, all new tasks start simultaneously, all old tasks observe DEPLOYMENT_BLOCKED roughly together
- 120%: rolling waves, ~2 new tasks per ~90-110 s wave, convergence scales roughly linearly with N
- Scale=10 at 120% took 634 s (vs ~180 s at 200%) — 3.5× slower

**Recommendation for spatial-validation:** keep module default 200%. At max=5 transient overshoot is trivial. Vector's 120% is justified for their 128-768 max scale.

### 7. Version polling is not load-bearing

**Empirically tested** (exp 1b, 2026-04-17): real task-def revision change at N=20 → 20/20 old tasks exited via DEPLOYMENT_BLOCKED, 0 SIGTERMs, converged cleanly in 196 s. This is the scenario version polling was designed to handle; we now know DEPLOYMENT_BLOCKED handles it without polling.

**Also audited**: 14 imagined failure modes, none broke the "DEPLOYMENT_BLOCKED + SIGTERM-on-unprotected-window" pair. Adversarial phase test (bimodal 50%) still converged cleanly via these two mechanisms.

Version polling adds:
- Faster voluntary exit during saturated deploys (seconds vs ~90 s)
- Explicit "task def superseded" log line
- Theoretical defense against unknown-unknown AWS bugs (but it uses the same control plane as the scheduler, so not truly independent)

Version polling cost:
- 38 lines in service_deployment.py + tests
- 1 DescribeServices call per message iteration
- Extra failure mode (what if DescribeServices is slow/throttled?)

**Empirical verdict: removable.** Both tested (exp 1b) and audited. Keep it if the observability log line or defensive posture has operational value beyond the empirical safety case.

## Production code changes (committed in lambda-assets#618 at 3898b34)

1. **Restored `TASK_STOPPING_OR_STOPPED` in `src/task_protection.py`** as distinct `ProtectionResult.TASK_STOPPING` variant — empirically verified across scale-in (exp 3) and saturated deploy at scale ≥50
2. **Widened exit condition in `src/app.py`** from `protection == DEPLOYMENT_BLOCKED` to `protection not in (ENABLED, UNAVAILABLE)` — any non-successful enable means task should not process the message
3. Version polling retained (not removed) pending final decision
4. All 121 unit + integration tests pass

## Production config recommendations

- `deployment_maximum_percent = 200` (module default, keep) — fast convergence, negligible transient cost at max=5
- `deployment_minimum_healthy_percent = 100` — keep (explicit in terraform)
- `ExpiresInMinutes = 30` for task protection — appropriate headroom above Lambda-15min-timeout ancestor constraint; will need bump to ~60-120 min if/when Phase 2 render+correlation features add minute-scale work
- Task CPU/memory `1024 / 2048` — keep as-is (inherited from daemon-transcoding; may need bump for Phase 2 render+correlation)
- SQS `VisibilityTimeout = 1800 s` (30 min) — matches ExpiresInMinutes; will need to scale together

## Open follow-ups

- ~~Version polling decision: remove or keep?~~ Decided and removed in lambda-assets#618 at `6dfa22b` based on empirical data (exp 1b + 10+ scale runs + adversarial audit)
- Phase 2 sizing (render + correlation) requires benchmarking actual render duration and memory against real Atmos files; don't pre-guess
- **CloudWatch alarm on deployment duration** — no ECS-layer time-based timeout exists (circuit breaker is failure-count based, not time based). Worth adding an alarm like `service in IN_PROGRESS > 30 min` → PagerDuty/Slack. See EXPERIMENTS.md "The residual gap" section.
- **CloudWatch alarm on `SERVICE_DEPLOYMENT_FAILED` event** — surfaces circuit-breaker-triggered failures to operators visibly
- **Operational runbook** for "deploy stuck > 15 min" — documented remediation steps

## Infrastructure state

- Service `desired_count = 0`
- Task def rev 4 active (bimodal test config)
- v4 image in ECR (`:latest` and `:v4` tags)
- Role names renamed to match `dev-*-task-role` pattern (terraform-infra#33152 applied)
- All raw data in `results/` subdirectories

## Timeline

- Experiment infra landed via terraform-infra#33140 (merged)
- IAM role rename via terraform-infra#33152 (merged + applied)
- Scratch code published via theorchard/collab#2637
- Production fixes committed to lambda-assets#618 at `3898b34` (pushed)

## Teardown checklist

See `manual-changes.md`. Plan:

1. Ensure `desired_count = 0` (done)
2. Open terraform-infra PR to destroy `dev/claude-ecs-task-protection-experiment/` module (later)
3. Merge collab PR for historical record
4. Clean local Docker images + `/tmp/docker-experiment-config/`