# ECS Task Protection Experiment — Comprehensive Record **Goal:** Determine empirically whether the ECS Fargate task-protection pattern is safe for long-running SQS-driven workers during rolling deploys and scale-in, and whether defensive layers like periodic task-definition-version polling are necessary. **Context:** CDAM-3806 converts `spatial-audio-validation` from per-task Fargate invocations to a long-running SQS-gated worker using SFN `waitForTaskToken` callback. Preserving in-flight messages across deploys requires task protection. The design question: is task protection alone sufficient, or do we also need defensive polling for new task-def revisions? **Run date:** 2026-04-17. --- ## Table of contents 1. [Methodology](#methodology) 2. [Infrastructure](#infrastructure) 3. [Experiment catalog](#experiment-catalog) 4. [Aggregate findings](#aggregate-findings) 5. [What we know vs inferred vs unknown](#what-we-know-vs-inferred-vs-unknown) 6. [Production implications](#production-implications) 7. [Gaps and caveats](#gaps-and-caveats) 8. [References](#references) --- ## Methodology ### Harness A minimal stdlib-only Python worker in `worker/worker.py` exercises the ECS task-protection agent endpoint in a tight loop: ``` while not shutdown: PUT /task-protection/v1/state { ProtectionEnabled: true, ExpiresInMinutes: N } sleep(WORK_DURATION_MS / 1000) PUT /task-protection/v1/state { ProtectionEnabled: false } sleep(IDLE_MS / 1000) ``` Every event (enable/work/disable/idle/error) emits a structured JSON log line via stdout. The ECS awslogs driver forwards to CloudWatch. **Cycle configuration (set via env vars):** - `WORK_DURATION_MS` = 200 for saturated profile (fast cycling, max request rate) — represents an over-stressed worker - `IDLE_MS` = 50 for saturated - `EXPIRES_MINUTES` = 5 for first experiments (quick clock), 30 later to match production value - `STARTUP_JITTER_MS`, `IDLE_JITTER_MS` added later for random phase spread - `PHASE_MODE` added later for bimodal (50%-offset) phase groups **Retry logic** added partway through: 3 retries with exponential backoff (0.5/1/2s) on `ThrottlingException` or HTTP 5xx responses, matching the production `task_protection.py` policy. ### Data capture Five data streams per experiment, written to `results//`: - `worker-logs.jsonl` — every structured event from every worker, pulled via `aws logs filter-log-events` after the deploy converges (+45 s buffer for CloudWatch ingestion lag) - `service-state.jsonl` — service desired/running/pending + rolloutState snapshots every 2 s via `tail_state.sh` - `service-events.jsonl` — ECS scheduler event messages - `task-state.jsonl` — per-task lifecycle including stopCode, stoppedReason - `task-definitions.jsonl` — task def snapshots (once per revision encountered) - `trigger.txt` — trigger timestamp (epoch ms + ISO), convergence time, pre/post revision (for revision-change runs) ### Trigger mechanisms - **`force_deploy.sh`**: `update-service --force-new-deployment` — creates a new deployment against the *same* task-def revision. Used for most runs; fast and isolates the deploy mechanism from task-def change noise. - **`update_env.sh`**: registers a new task-def revision with modified env vars, then `update-service --task-definition `. Used only in exp 1b. Requires `iam:PassRole` on the task role (fixed via terraform-infra#33152 partway through the session). ### Analysis Python scripts (inline in task output) parse JSONL streams and compute: - Time from trigger to first `enable_failed` (DEPLOYMENT_BLOCKED latency) - Spread across all failing tasks - Mechanism distribution (DEPLOYMENT_BLOCKED vs TASK_STOPPING_OR_STOPPED) - SIGTERM timing correlated with preceding disable/enable state - Throttling error counts and retry-inflated latency --- ## Infrastructure **Repo:** [theorchard/terraform-infra](https://github.com/theorchard/terraform-infra) → `dev/claude-ecs-task-protection-experiment/` **Resources:** - ECR repo (force_delete enabled, v1-v4 images pushed over session) - CloudWatch log group (7-day retention) - ECS cluster (Container Insights disabled to minimize background CloudWatch noise) - IAM task role + execution role (renamed to `dev-*-task-role` / `-task-execution-role` in PR #33152 to match the PassRole allowlist) - IAM policy: `ecs:UpdateTaskProtection`, `ecs:GetTaskProtection`, `ecs:DescribeServices`, `logs:*` - Security group (egress-only) - ECS service with `ignore_changes = [desired_count, task_definition]` so manual updates from experiment scripts don't fight terraform **Deploy configuration during experiments:** - `deployment_minimum_healthy_percent = 100` - `deployment_maximum_percent`: switched between 200 (default) and 120 (vector-like) - `availability_zone_rebalancing`: default (ENABLED) **Key PRs in the timeline:** - [terraform-infra#33140](https://github.com/theorchard/terraform-infra/pull/33140) — initial experiment infrastructure (MERGED + APPLIED) - [terraform-infra#33152](https://github.com/theorchard/terraform-infra/pull/33152) — IAM role rename + EXPIRES_MINUTES bump to 30 (MERGED + APPLIED) - [collab#2637](https://github.com/theorchard/collab/pull/2637) — this experiment code (OPEN, this document) - [lambda-assets#618](https://github.com/theorchard/lambda-assets/pull/618) — production worker code (OPEN) - [terraform-fargate#241](https://github.com/theorchard/terraform-fargate/pull/241) — adds `deployment_circuit_breaker` support to the module (OPEN) - [terraform-infra#33124](https://github.com/theorchard/terraform-infra/pull/33124) (QA) and [#33125](https://github.com/theorchard/terraform-infra/pull/33125) (PROD) — spatial-validation service config consumers (OPEN) --- ## Experiment catalog ### Saturated rolling deploys All runs: `WORK_DURATION_MS=200`, `IDLE_MS=50`, triggered via `force_deploy.sh` unless noted. #### Exp 1 — scale=3, baseline - **Trigger:** 2026-04-17T16:22:49Z - **Pre-steady:** 3 tasks running on rev 1 - **Convergence:** 2:47 (167 s) - **Events:** 3 enable_failed (all DEPLOYMENT_BLOCKED), 3 worker_exit, 0 SIGTERMs - **Time to first DEPLOYMENT_BLOCKED:** 84 s - **Spread across 3 tasks:** 260 ms - **Notes:** original run. ECS reported "2 tasks under protection" mid-deploy, consistent with scheduler waiting for protection to release before stopping. #### Exp 1-repeat-A — scale=3, replicate - **Trigger:** 2026-04-17T17:18:55Z - **Convergence:** 3:10 (190 s) - **Events:** 3 enable_failed (DEPLOYMENT_BLOCKED), 3 worker_exit, 0 SIGTERMs - **Time to first:** 94 s - **Spread:** 160 ms - **Notes:** initial worker-logs pull was incomplete (`aws logs tail` killed too fast after convergence; CloudWatch ingestion lag meant final events hadn't propagated). Re-pulled via `filter-log-events` with 30-s catch-up; saved as `worker-logs-complete.jsonl`. #### Exp 1-repeat-B — scale=3, replicate - **Trigger:** 2026-04-17T17:26:31Z - **Convergence:** 2:55 (175 s) - **Events:** 3 enable_failed (DEPLOYMENT_BLOCKED), 3 worker_exit, 0 SIGTERMs - **Time to first:** 92 s - **Spread:** 217 ms - **Notes:** capture methodology improved — single `filter-log-events` pull after 45-s wait. Used going forward. #### Exp 1-scale=5 - **Trigger:** 2026-04-17T17:31:54Z - **Convergence:** 3:01 (181 s) - **Events:** 5 enable_failed, 5 worker_exit, 0 SIGTERMs - **Time to first:** 89.5 s - **Time to last:** 89.75 s - **Spread:** 233 ms - **Per-task timing offsets:** 0 / 13.6 / 69.7 / 168.3 / 232.8 ms - **Notes:** matches smaller-scale timing closely; scale invariance begins to show. #### Exp 1-scale=10 (no retry worker) - **Convergence:** 3:19 (199 s) — but contaminated - **Events:** 9 enable_failed, 13 worker_exit (more than expected due to premature exits), 0 SIGTERMs, 4 `enable_error` with `ThrottlingException`, 5 `disable_error` - **Notes:** first run where the agent endpoint rate-limited at this scale+cycle. Our experiment worker had no retry at this point. Throttled requests caused worker exits (via widened exit condition in worker.py). 3 new-deployment tasks also exited early due to the same throttling and were replaced by ECS, producing task churn. Prompted adding retry logic to worker. **This data is contaminated for DEPLOYMENT_BLOCKED-timing purposes** but establishes that throttling is real at this cycle rate. #### Exp 1-scale=10, retry worker, 120% deploy config - **Convergence:** 10:34 (634 s) — intentionally slow - **Deploy config:** `deployment_maximum_percent = 120` (vector-like, not default 200) - **Events:** 12 enable_failed, 12 worker_exit, 0 SIGTERMs, 0 throttle errors - **Time to first:** 93.6 s - **Spread:** 423 s (many waves of ~90-110 s each) - **Notes:** first run with retry worker (v2 image). Demonstrates the "wave pattern" at 120%: ECS launches ~1-2 new tasks at a time, waits for corresponding old tasks to release protection via DEPLOYMENT_BLOCKED, repeats. Convergence linear in N. Informs the 120% vs 200% tradeoff discussion. #### Exp 1-scale=20, retry worker, 200% deploy config - **Trigger:** ~2026-04-17T17:31:53Z - **Convergence:** 3:14 (194 s) - **Events:** 20 enable_failed, 20 worker_exit, 0 SIGTERMs, 0 throttle errors - **Time to first:** 96.24 s - **Time to last:** 96.51 s - **Spread:** 277 ms - **Notes:** clean run. Tight spread matches small-scale behavior despite larger N. #### Exp 1-scale=50, retry worker, 200% deploy config - **Convergence:** 3:14 (194 s) - **Events:** 48 enable_failed, 56 worker_exit, 19 SIGTERMs, 10 throttle errors (via enable_error/disable_error events) - **Time to first:** 94.87 s - **Time to last:** 176.32 s - **Spread:** 81.4 s - **Notes:** FIRST run where the <300 ms tight spread breaks down. Spread jumps to 81 s. Also FIRST run with SIGTERMs. Triggered deep analysis of whether ECS was overriding protection (it wasn't — see finding #3 below). All 19 SIGTERMs arrived after `disable_success` with normal latency (~40 ms), followed by a 362 ms to 13.8 s gap before SIGTERM. The gap corresponds to the worker's next `enable()` call being throttle-retry-delayed — extending the unprotected window from ~50 ms to multi-second, which ECS exploited. #### Exp 1-scale=100, retry worker, 200% deploy config - **Convergence:** DID NOT CONVERGE in 100+ minutes — stopped manually at 98/100 running - **Events:** 78 enable_failed (40 DEPLOYMENT_BLOCKED + 38 TASK_STOPPING_OR_STOPPED), 374 worker_exit, 31 SIGTERMs, 296 enable_errors, 437 disable_errors (heavy throttling) - **Time to first enable_failed:** 98.27 s - **Notes:** cascade failure. With 200 concurrent tasks (100 old + 100 new during deploy) × 200 ms cycles, the agent endpoint's rate limit is continuously exceeded. Retry budget exhausted → workers exit via `FAILED` (widened exit condition) → ECS replaces → new workers also throttle → churn continues. The ~90 s scheduler tick still fires (98 s) but overall deploy doesn't converge because tasks die faster than ECS stabilizes. **Relevant for PRODUCTION only if request rate approaches this regime**; prod cycle is message-duration (seconds), so rate is ~100× lower. At max=5 prod workers with 30-s work cycles, request rate is ~0.3 req/s total — far below any throttle threshold. #### Exp 1-scale=50, retry, random phase jitter - **Convergence:** ~3 minutes - **Config:** `STARTUP_JITTER_MS=250`, `IDLE_JITTER_MS=100` — uniform random phase spread across the cycle - **Events:** 48 enable_failed, 51 worker_exit, 4 SIGTERMs, 10 throttle errors - **Time to first:** 89.4 s - **Spread:** 10.6 s - **Notes:** the 81-s spread from the synchronized scale=50 run collapses to 10.6 s with jitter. SIGTERMs drop 5× (19 → 4). Confirms the wide spread was a synchronization artifact, not an AWS-side behavior. Production workers are naturally message-driven/jittered, so real deploys should look like this row, not the synchronized baseline. #### Exp 1-scale=50, retry, bimodal 50% phase offset - **Convergence:** ~3 minutes (post-trigger analysis) - **Config:** `PHASE_MODE=bimodal` — half the workers start at phase 0, half at phase 125 ms (half of the 250 ms cycle) - **Events:** 47 enable_failed (post-trigger: 32 DEPLOYMENT_BLOCKED + 15 TASK_STOPPING_OR_STOPPED), 54 worker_exit, 17 SIGTERMs, 18 throttle errors - **Time to first (post-trigger):** 92.77 s - **Spread (post-trigger):** 12.24 s - **Notes:** adversarial test designed to break DEPLOYMENT_BLOCKED by forcing `protected_count` to oscillate around `desired_count`. Hypothesis: if at any instant ~50% of tasks are unprotected, the service-level check `protected > desired` might never cleanly trip. **Result: hypothesis was wrong.** With 200 ms work + 50 ms idle (80% / 20% split), the two phase groups overlap in "both protected" state ~60% of the cycle, so the condition still holds enough to fire DEPLOYMENT_BLOCKED. 17 SIGTERMs (between sync's 19 and random's 4) — SIGTERM count correlates with *phase clumping*, not adversarial phase per se. #### Exp 1b — scale=20, REAL task-def revision change - **Trigger:** 2026-04-17T19:59:45Z via `run_real_deploy.sh 20 experiment-1b-real-deploy 60` — registered a new task-def revision (rev 4 → rev 5) with `IDLE_MS` changed from 50 → 60 - **Convergence:** 3:16 (196 s) - **Events:** 20 enable_failed, 20 worker_exit, 0 SIGTERMs, 0 throttle errors - **Revisions observed in logs:** rev 4 = 19,730 events (old); rev 5 = 35,507 events (new, longer run) - **Notes:** THE gap exp. Every prior experiment triggered deploys via `force-new-deployment` against the *same* task-def revision, so old and new tasks had identical `taskDefinitionArn`. This was the first run where the rolling deploy transitioned between two *different* revisions. Behavior identical to force-deploy baseline: DEPLOYMENT_BLOCKED fires at ~90 s, all 20 old-revision tasks exit cleanly via that signal. Closes the empirical gap that was keeping the version-polling removal decision open. ### Scale-in #### Exp 3 — saturated scale-in 3→1 - **Trigger:** 2026-04-17T16:29:07Z via `set_desired.sh 1` - **Setup:** 3 tasks running saturated (just after exp 1 completed) - **Convergence:** 0:12 (12 s) - **Events:** 2 enable_failed (both `TASK_STOPPING_OR_STOPPED`), 2 worker_exit (matching 3→1 delta), 0 SIGTERMs - **Time from trigger to first enable_failed:** ~2 s - **Spread across 2 stopping tasks:** 31 ms - **Notes:** distinct mechanism from rolling deploy. No service-level counter check; ECS directly marks the specific task(s) to stop, and those tasks' next `enable()` calls return `TASK_STOPPING_OR_STOPPED`. 14× faster than deploy convergence because there's no ~90 s scheduler tick — the per-task marking flips immediately. This was the run that empirically confirmed `TASK_STOPPING_OR_STOPPED` is a real, current reason code (it had been removed from `task_protection.py` during a prior review pass because we couldn't cite primary docs for it). ### Experiments inferred, not run - **Exp 2 (idle rolling deploy):** worker spending most of time in SQS long-poll, rarely enabling protection. Inferred behavior: ECS sees protection off, sends SIGTERM during the unprotected window (same mechanism observed at scale=50 with saturated cycles). SIGTERM handler releases message + exits. - **Exp 4 (idle scale-in):** same reasoning as exp 2, just with TASK_STOPPING_OR_STOPPED targeting instead of the service-wide counter. - **Exp 5 (long-running work, 30 s):** worker holds protection during work. Inferred behavior: worker enables, works 30 s, disables, enables, etc. DEPLOYMENT_BLOCKED fires on the next enable after the service counter tips (~90 s). Deploy convergence lower-bounded by max work duration. For production mediainfo work (seconds), this is tight. For future Phase 2 render+correlation (minutes), this means deploys take as long as the longest in-flight message. - **Exp 6 (scale to 0):** same mechanism as exp 3 but fully. Last task hits TASK_STOPPING_OR_STOPPED, exits, service is empty. These were deemed inferable based on the already-validated mechanisms (DEPLOYMENT_BLOCKED, TASK_STOPPING_OR_STOPPED, SIGTERM). Running them would add confirmation but no new mechanism-level data. --- ## Aggregate findings ### Finding 1: the ~90 s scheduler tick is scale-invariant and config-invariant Across 10 saturated-deploy runs spanning N=3 to N=100, both deploy configs (200% and 120%), synchronized and random-jittered and bimodal phases, and both force-deploy-same-rev and real-revision-change triggers: | Run | N | Config | 1st DEPLOYMENT_BLOCKED | |---|---|---|---| | Exp 1 | 3 | 200% sync | 84 s | | Repeat A | 3 | 200% sync | 94 s | | Repeat B | 3 | 200% sync | 92 s | | Scale=5 | 5 | 200% sync | 89.5 s | | Scale=10 retry | 10 | 120% sync | 93.6 s | | Scale=20 | 20 | 200% sync | 96.2 s | | Scale=50 | 50 | 200% sync | 94.9 s | | Scale=50 random jitter | 50 | 200% random | 89.4 s | | Scale=50 bimodal | 50 | 200% bimodal | 92.8 s | | Scale=100 | 100 | 200% sync | 98.3 s | | **Exp 1b** | **20** | **200% real revision change** | **≈90 s** | Mean: **91 s** across all runs, range 84-98 s, coefficient of variation ~5%. Tight enough to plan around. Safe to assume ~90 s for production convergence modeling. ### Finding 2: spread is synchronization-dependent, not scale-dependent At N=3/5/20 with synchronized workers, the spread across failing tasks within a single run is always <300 ms. At N=50 synchronized it jumps to 81 s — initially surprising. But adding random phase jitter at the same N=50 collapses spread back to 10.6 s, and bimodal phase offset at N=50 gives 12.2 s. So spread correlates with synchronization clump size, not total task count. **Production relevance:** real workers are naturally desynchronized (message arrival times, per-message work durations vary). Production deploys should behave like the jittered row — tight spread, few SIGTERMs. ### Finding 3: ECS respects the task-protection contract At scale=50 synchronized we observed 19 SIGTERM events, initially read as "ECS giving up on protection and force-stopping tasks." Detailed analysis of each SIGTERM: - **19/19** arrived after `disable_success` (protection was OFF from the worker's perspective) - **0/19** arrived while protection was held - Time between `disable_success` and SIGTERM: 362 ms to 13.8 s For 17/19 of these, the `disable_success` elapsed time was normal (~40 ms), so protection was clearly off. For 2/19, the disable call itself was retry-inflated (4457 ms and 4323 ms) due to throttling. The wide time-between-disable-and-SIGTERM gap was caused by the worker's *next* `enable()` call being retry-delayed (throttling). From ECS's perspective: task is unprotected, scheduler is free to stop it, SIGTERM sent. **ECS never overrode protection.** What looked like "giving up" was "opportunistic stopping during extended unprotected windows caused by throttling retries." At normal cycle rates (no throttling), unprotected windows are ~50 ms and ECS rarely grabs them. ### Finding 4: two distinct failure reasons fire for rolling deploys at scale At N=3/5/20 only `DEPLOYMENT_BLOCKED` fires. At N=50 and N=100 we see both: - Scale=50: 48 DEPLOYMENT_BLOCKED, 0 TASK_STOPPING_OR_STOPPED (surprisingly) - Scale=50 bimodal: 32 DEPLOYMENT_BLOCKED, 15 TASK_STOPPING_OR_STOPPED - Scale=100: 40 DEPLOYMENT_BLOCKED, 38 TASK_STOPPING_OR_STOPPED The service-level counter (DEPLOYMENT_BLOCKED) isn't the only deploy-time signal. ECS also *targets specific tasks* for stopping during a deploy — when it does, those tasks' next `enable()` calls return TASK_STOPPING_OR_STOPPED rather than DEPLOYMENT_BLOCKED. **Implication for worker code:** must handle BOTH reason codes identically (release + exit). This is the motivation for the widened exit condition in `app.py` — `protection not in (ENABLED, UNAVAILABLE)` covers all current AND any future reason codes uniformly. ### Finding 5: agent endpoint has a rate limit that matters only at extreme rates At 200 ms cycles × 10+ concurrent workers, we start seeing `ThrottlingException` on `UpdateTaskProtection`. Retry handles most of it at N ≤ 50. At N=100 with 200% deploy config (200 concurrent tasks), the retry budget is insufficient, causing the cascade described in exp 1-scale=100. **Production relevance:** experimental artifact only. Production cycles are message-duration (mediainfo validation ≈ seconds). At max=5 workers and ~10 s per message, total rate ≈ 0.5 req/s. At max=50 (hypothetical future scale), rate ≈ 5 req/s. Both well under any plausible throttle threshold. ### Finding 6: 200% vs 120% deploy config changes mechanics, not fundamentals - 200%: all new tasks can start simultaneously; old tasks observe DEPLOYMENT_BLOCKED roughly together; single big wave; ~3-minute convergence regardless of N. - 120%: ECS launches only small batches; each batch waits for a DEPLOYMENT_BLOCKED cycle before the next; convergence scales roughly linearly with N. Scale=10 at 120% took 634 s vs ~180 s at 200%. **Recommendation for spatial-validation:** use module default 200%. At max=5 the transient 2× overshoot is trivial; the fast convergence is worth it. Vector's 120% is justified because at max=768 the 2× overshoot would be enormous (ENI pool pressure, connection storms). ### Finding 7: version polling is not required for correctness Empirically tested across scenarios polling was designed to handle: - Exp 1b: real task-def revision change. 20/20 old tasks exited via DEPLOYMENT_BLOCKED without any polling signal. Clean convergence in 196 s. - 10+ saturated-deploy runs across scales 3-100: DEPLOYMENT_BLOCKED fired reliably in every single run. - Adversarial audit: 14 hypothesized failure modes tried, none broke the DEPLOYMENT_BLOCKED + SIGTERM-on-unprotected-window pair. Polling would provide: - Faster voluntary exit (seconds vs ~90 s). Cosmetic, not safety. - Explicit "task def superseded" log line. Replicable via other observability. - Theoretical defense against unknown AWS bugs. But polling uses the same ECS control plane as the scheduler, so not truly independent. **Verdict: removable.** Removing drops 38 lines + one API dependency + one failure mode. Keeping is defensible only if the observability log line has operational value independent of the safety case. --- ## What we know vs inferred vs unknown ### Known (empirically measured) - DEPLOYMENT_BLOCKED fires at ~90 s (±6 s, CV ~5%) after deploy trigger across scales 3-100 and all tested configs - DEPLOYMENT_BLOCKED fires for both force-deploy-same-revision AND real task-def revision change - TASK_STOPPING_OR_STOPPED is a real, currently-returned reason code - Scale-in triggers TASK_STOPPING_OR_STOPPED on the next enable call of targeted tasks (12 s convergence for small N) - SIGTERM fires during unprotected windows; ECS never overrides active protection - Agent endpoint rate-limits with `ThrottlingException` at ~80+ req/s sustained (roughly, from scale=10 observation) - Retry with exponential backoff (3 attempts, 0.5/1/2 s) handles throttling at N ≤ 50 with 200 ms cycles - Saturated deploy converges in ~3 minutes at 200% across all scales (given throttling doesn't cascade) - 120% deploy config produces wave-based convergence scaling roughly linearly with N ### Inferred (not directly measured but implied) - Idle workers exit via SIGTERM during the window where protection is off (from scale=50 observation of the same mechanism) - Long-running work pushes deploy convergence out by the max work duration (because worker only re-enables between messages) - At prod scale (max=5, message-driven cycles), throttling is irrelevant - Spread in production should look like the jittered row (few SIGTERMs, tight DEPLOYMENT_BLOCKED spread) ### Unknown / uncharacterized - **Exact ECS internal logic for the ~90 s tick.** AWS docs say `protected_count > desired_count` but that alone doesn't explain the consistent 90-s delay. Our hypothesis is ECS has a grace period / retry loop after deciding "time to stop old tasks" before it tips the service-level DEPLOYMENT_BLOCKED gate. Not verified against AWS implementation. - **Exact rate-limit threshold on the agent endpoint.** We observed throttling at roughly 80+ req/s sustained but didn't binary-search the threshold. - **Behavior during deploy failures** (bad new code, circuit breaker firing). Not exercised. - **Behavior during cross-AZ / cross-region events.** Not tested. - **What happens if `DescribeServices` is slow or unavailable.** Not tested (affects version polling if we kept it). - **Exact formula for how ECS picks tasks for TASK_STOPPING_OR_STOPPED during a deploy.** We see it at scale but don't know the selection logic. ### Claims we've retracted - "Version polling has strong empirical basis for removal" — initially claimed based only on audit; not true until exp 1b was run. Now accurate. - "SIGTERMs at scale=50 showed ECS giving up on protection" — initial interpretation was wrong. Detailed analysis showed ECS respected protection; SIGTERMs fired during genuinely unprotected windows extended by throttle retries. - "The 81-s spread at scale=50 reveals scale-dependent behavior" — initial interpretation. Correct interpretation: synchronization artifact that disappears with jitter. --- ## Production implications ### Deploy convergence SLO - **~3 minutes for rolling deploy at prod max=5** (steady state, no circuit-breaker firings) - **90 s lower bound** from the scheduler tick; shorter convergence not achievable via task-protection path alone - **Upper bound scales with max work duration** if work exceeds ~90 s (not applicable for current mediainfo-only work; becomes relevant if Phase 2 render+correlation adds minute-scale work) ### Recommended service config for spatial-validation - `deployment_maximum_percent = 200` (module default — keep) - `deployment_minimum_healthy_percent = 100` (explicit in terraform) - `deployment_circuit_breaker_enabled = true` (pending terraform-fargate#241) - `deployment_circuit_breaker_rollback_enabled = true` - `task_cpu = 1024 / task_memory = 2048` (keep current; Phase 2 may need bump) - `ExpiresInMinutes = 30` for task protection (matches prod terraform) - SQS `VisibilityTimeout = 1800 s` (matches expires_minutes) - `WorkerRetryPolicy`: 3 retries, exponential backoff 0.5/1/2 s on `ThrottlingException` / HTTP 5xx ### Worker behavior requirements (committed to lambda-assets#618) 1. `enable_protection()` must return distinct result for each AWS-documented reason code (DEPLOYMENT_BLOCKED, TASK_STOPPING, MISSING, TASK_NOT_VALID), plus generic FAILED for unknown reasons. 2. Worker exit condition: any non-successful enable result releases the message and exits. "Success" means ENABLED or UNAVAILABLE (local dev). 3. SIGTERM handler sets a shutdown flag; loop exits at next top-of-loop check. 4. Message deletion happens in `finally` — SFN handles per-message outcome via `SendTaskSuccess` / `SendTaskFailure`. ### Failure-detection safety net — what each layer actually covers | Failure mode | Catches it | Notes | |---|---|---| | Old tasks don't release protection (normal rolling deploy) | DEPLOYMENT_BLOCKED | Empirically verified at ~90 s across N=3-100 | | Old tasks don't release because they got scale-in marked mid-deploy | TASK_STOPPING_OR_STOPPED | Fires alongside DEPLOYMENT_BLOCKED at larger scale | | Worker stuck mid-cycle, protection off briefly | SIGTERM | Empirically observed during unprotected windows at scale | | Worker crashed without `disable()` | `ExpiresInMinutes` | Auto-expires after 30 min; task becomes eligible for stop | | New code won't start (image pull fail, task-def invalid) | Circuit breaker (when enabled) | **Failure-count based**, not time based | | New code starts but crashes repeatedly | Circuit breaker | Same | | New code passes health checks but has runtime bug | Nothing in stack — surfaces via app monitoring | | | Deploy wall-clock exceeds reasonable limit | `fargateDeploy updateTimeout` | **CI-layer only** — bails the deploy script, not ECS's view of the deployment | | **Hypothetical: DEPLOYMENT_BLOCKED silently doesn't fire AND SIGTERM doesn't trigger AND new tasks are healthy** | **Nothing in the current stack** | No empirical evidence this occurs; listed for completeness | ### Circuit breaker is NOT a deploy timeout Commonly misunderstood. The circuit breaker's failure threshold (`max(3, min(desired_count * 0.5, 200))`, rounded up — not configurable) counts **consecutive failed task launches OR failed health checks**, not elapsed time. A deploy where new tasks launch successfully but the deployment stays `IN_PROGRESS` for other reasons (old tasks holding protection, slow convergence, etc.) will **never trigger the circuit breaker**. ### The residual gap Without version polling, the only safety net against "deploy is stuck because DEPLOYMENT_BLOCKED silently doesn't fire" is `fargateDeploy`'s wall-clock `updateTimeout` — which fails the CI script but leaves the ECS deployment `IN_PROGRESS` in mixed-revision state. This requires manual intervention (re-trigger the deploy, investigate via CloudWatch). This is a **real residual risk**. No empirical evidence it has occurred, but AWS behavior changes (or unknown scheduler bugs) could produce it. Mitigations worth considering as follow-up work (not scope for the initial spatial-validation PR): 1. **CloudWatch alarm on deployment duration** — e.g., `service in IN_PROGRESS > 30 min` → PagerDuty/Slack. This is the actual time-based safety net. 2. **CloudWatch alarm on `SERVICE_DEPLOYMENT_FAILED` event** — catches circuit-breaker-triggered failures visibly, instead of requiring someone to check the console. 3. **Operational runbook** — documented remediation steps for "deploy stuck > 15 min": re-trigger, force stop specific tasks, etc. ### Without circuit breaker (current state of production service) - ECS has no automatic timeout on the deployment - No automatic rollback on failure - Service stays in `IN_PROGRESS` indefinitely in mixed-revision state on any failure mode - Manual intervention required to detect/remediate Circuit breaker will be added by [terraform-fargate#241 + consumer PRs](#infrastructure). Even after it lands, the residual gap above remains unaddressed until a CloudWatch-alarm-on-duration follow-up. --- ## Gaps and caveats ### Remaining untested scenarios - **Production-shape cycle times.** All experiments used 200 ms work cycles. Production is seconds-per-message. Mechanism *should* be identical, but not empirically verified at production rates. - **Deployment circuit breaker firing.** Config is set for prod; we haven't exercised the failure path (bad new code, auto-rollback). - **Network partitions / agent endpoint unavailability.** Worker code treats these as `FAILED` via retry-exhaustion path; not scenario-tested. - **CloudWatch alarm on SERVICE_DEPLOYMENT_FAILED.** Not yet configured for spatial-validation. Worth adding as a follow-up. ### Known limitations of this experiment - **Synthetic worker** — doesn't do real message processing; pure enable/disable cycle. Production behavior may differ for work-duration reasons (e.g., long mediainfo calls blocking the loop longer). - **Single AZ / single cluster / dev account** — didn't test prod-like capacity or cross-AZ failure modes. - **Dev IAM / dev network** — rate limits on the agent endpoint may differ in prod accounts. - **Didn't test under autoscaling activity** — experiments used fixed `desired_count` via manual `update-service` calls. Production has autoscaling which may add its own dynamics (scale-in during deploy, etc.). --- ## References ### Internal - [FINDINGS.md](./FINDINGS.md) — condensed summary of this document - [theorchard/collab#2637](https://github.com/theorchard/collab/pull/2637) — this PR - [theorchard/terraform-infra#33140](https://github.com/theorchard/terraform-infra/pull/33140) — initial experiment infrastructure - [theorchard/terraform-infra#33152](https://github.com/theorchard/terraform-infra/pull/33152) — IAM role rename + EXPIRES_MINUTES=30 - [theorchard/lambda-assets#618](https://github.com/theorchard/lambda-assets/pull/618) — spatial-validation SQS-gated worker (production code) - [theorchard/terraform-fargate#241](https://github.com/theorchard/terraform-fargate/pull/241) — adds deployment_circuit_breaker to module - [theorchard/terraform-infra#33124](https://github.com/theorchard/terraform-infra/pull/33124) (QA), [#33125](https://github.com/theorchard/terraform-infra/pull/33125) (PROD) — spatial-validation service config ### External - [AWS — Task scale-in protection](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-scale-in-protection.html) - [AWS — Protecting your tasks from being terminated (agent endpoint)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-scale-in-protection-endpoint.html) - [AWS — Deployment circuit breaker](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html) - [AWS — API failures and error messages (reason codes)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/api_failures_messages.html) - [AWS — Deploy Amazon ECS services by replacing tasks (rolling update)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html)