# ECS Task Protection Experiment

Empirical harness for measuring how ECS Fargate task scale-in protection
actually behaves during rolling deploys and scale-in. Built to settle the
open design question on CDAM-3806: does the AWS-documented task-protection
pattern, on its own, cover the busy-queue rolling-deploy case — or does the
worker actually need version polling as a deterministic exit trigger?

## Links

- Infrastructure: [theorchard/terraform-infra `dev/claude-ecs-task-protection-experiment/`](https://github.com/theorchard/terraform-infra/tree/master/dev/claude-ecs-task-protection-experiment)
- Initial infra PR (merged): [theorchard/terraform-infra#33140](https://github.com/theorchard/terraform-infra/pull/33140)
- IAM role-rename follow-up PR: [theorchard/terraform-infra#33152](https://github.com/theorchard/terraform-infra/pull/33152)
- Production feature PR using these findings: [theorchard/lambda-assets#618](https://github.com/theorchard/lambda-assets/pull/618)

Tear the terraform module down when the experiment is complete — this is a
measurement harness, not a permanent workload.

## Layout

```
worker/
  worker.py       # stdlib-only enable/work/disable loop + structured JSON logs
  Dockerfile      # python:3.13-slim, CMD is python exec form for PID-1 SIGTERM
scripts/
  build_and_push.sh   # docker build + push to the experiment ECR repo
  set_desired.sh      # change desired_count
  force_deploy.sh     # force a rolling deploy without image change
  update_env.sh       # re-register task def with new WORK_DURATION_MS / IDLE_MS
  tail_state.sh       # snapshot describe-services every 2s to a JSONL file
  tail_logs.sh        # tee CloudWatch logs to a JSONL file
README.md
```

## Prerequisites

1. The terraform-infra PR at `dev/claude-ecs-task-protection-experiment/` is
   applied. This creates the ECR repo, IAM roles, cluster, service (at
   `desired_count=0`), log group, and SG.
2. `awsume dev` has active credentials.
3. Docker is running locally.

## First-time setup

```bash
# Build and push an initial image
./scripts/build_and_push.sh v1

# Bring the service up with 3 tasks (cycling at the default 200ms work / 50ms idle)
./scripts/set_desired.sh 3
```

Wait for all 3 tasks to be RUNNING (about 30–60s). Confirm with:

```bash
aws ecs describe-services --cluster claude-ecs-task-protection-experiment \
    --services claude-ecs-task-protection-experiment --query 'services[0].runningCount'
```

## Experiment methodology

**A "deploy" = registering a new task definition revision, not pushing a
new image.** Each `update_env.sh` call registers a new revision and points
the service at it. Old and new workers then show distinct `revision` fields
in their JSON logs, which lets us tell them apart cleanly.

`force_deploy.sh` triggers a rolling deploy without a task def change,
which ECS handles the same way mechanically but doesn't produce
distinguishable old/new workers in logs. Prefer `update_env.sh` for
measurement runs; `force_deploy.sh` is here for edge-case testing.

`build_and_push.sh` is only needed once (or when you want to change the
worker code). The experiment's worker code is static — we're measuring
ECS behavior, not application code, so we don't need to push new images
between experiments.

## Experiment 1: saturated-queue rolling deploy

**Question**: with all workers tight-looping enable→work→disable at ~250ms
cycle time, does a rolling deploy complete via task protection's natural
mechanisms, or do old tasks keep their protection renewed faster than ECS
can terminate them?

**Preconditions**: service at `desired_count = 3`, env `WORK_DURATION_MS=200
IDLE_MS=50 EXPIRES_MINUTES=5` (saturated profile, the terraform default).

**Time budget**: 35 minutes. If not converged by then, the stall hypothesis
is confirmed — don't wait longer (see "Aborting a stalled experiment" below).

```bash
mkdir -p results/experiment-1-saturated-deploy

# Warmup verification — confirm steady state BEFORE triggering
aws ecs describe-services --cluster claude-ecs-task-protection-experiment \
    --services claude-ecs-task-protection-experiment \
    --query 'services[0].{running:runningCount,desired:desiredCount,deps:deployments[].status}'
# Should show running=3, desired=3, deps=["PRIMARY"]. If not, wait.

# Confirm workers are actively cycling by tailing a few log lines:
aws logs tail /ecs/claude-ecs-task-protection-experiment --since 30s --format short | head -20
# Should see enable_success, work_start/end, disable_success, every ~250ms per task

# Terminal A — snapshot service state
./scripts/tail_state.sh results/experiment-1-saturated-deploy &
STATE_PID=$!

# Terminal B — tail worker logs
./scripts/tail_logs.sh results/experiment-1-saturated-deploy &
LOGS_PID=$!

# Trigger rolling deploy: register a new task def revision (same values
# but revision number bumps, giving us distinct old vs new workers).
./scripts/update_env.sh 200 50 5

DEPLOY_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "Deploy triggered at ${DEPLOY_START}" > results/experiment-1-saturated-deploy/notes.md
echo "Old revision: check task-state.jsonl for taskDefinitionArn at ${DEPLOY_START}" >> results/experiment-1-saturated-deploy/notes.md

# Wait until one of:
#   (a) service-state.jsonl shows deployments[] down to 1 entry (PRIMARY only),
#       runningCount == 3 at the new revision — DEPLOY COMPLETED
#   (b) 35 minutes elapsed — STALL CONFIRMED
# Watch in a third terminal:
#   watch -n 5 "tail -1 results/experiment-1-saturated-deploy/service-state.jsonl | jq .deployments"

DEPLOY_END=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "Deploy ended at ${DEPLOY_END}" >> results/experiment-1-saturated-deploy/notes.md

kill ${STATE_PID} ${LOGS_PID}
```

**Signals to look for in the output:**

| File | Search | Meaning |
|---|---|---|
| `worker-logs.jsonl` | `"event":"sigterm_received"` with `"revision":"N"` (old) | ECS successfully terminated old-rev tasks — bare task protection handled the deploy |
| `worker-logs.jsonl` | `"event":"enable_failed"` with `"reason":"DEPLOYMENT_BLOCKED"` | `DEPLOYMENT_BLOCKED` fired. Check which revision saw it. |
| `service-events.jsonl` | `"was unable to ... due to task scale-in protection"` | ECS tried to scale in / replace but was blocked |
| `service-state.jsonl` | `deployments[]` length > 1 for many minutes | Deploy hasn't reached steady state — potential stall |
| `task-state.jsonl` | old-rev tasks with `stopCode` and `stoppedReason` | How ECS eventually terminated old workers |

**Outcome interpretation:**
- **Deploy completed < 2 min**: bare task protection handles this path. Version polling is pure defence.
- **Deploy completed 2–30 min, old-rev tasks eventually SIGTERM'd**: ECS scheduler catches the unprotected window eventually. Version polling is faster but not load-bearing.
- **Deploy stalled >30 min or old-rev tasks never SIGTERM'd**: version polling is load-bearing. The production design choice is validated.

## Experiment 2: idle rolling deploy (baseline)

**Question**: with workers idle (long sleep between cycles, simulating an
empty queue), does a rolling deploy complete cleanly and fast via SIGTERM?

**Preconditions**: service at `desired_count = 3`. Worker profile doesn't
matter before we switch it.

**Time budget**: 5 minutes. Should converge fast.

```bash
# Switch to idle profile (registers a new task def revision with longer sleeps)
./scripts/update_env.sh 20000 20000 5

# Wait 1-2 min for ECS to settle on the idle-profile revision and for
# workers to pass through their first cycle.
sleep 90

# Warmup verification — confirm tasks are idle-cycling
aws logs tail /ecs/claude-ecs-task-protection-experiment --since 30s --format short | head
# Should see long pauses between log lines (20s each)

mkdir -p results/experiment-2-idle-deploy
./scripts/tail_state.sh results/experiment-2-idle-deploy &
STATE_PID=$!
./scripts/tail_logs.sh results/experiment-2-idle-deploy &
LOGS_PID=$!

# Trigger: register another new revision (same idle values)
./scripts/update_env.sh 20000 20000 5
echo "Deploy triggered at $(date -u +%Y-%m-%dT%H:%M:%SZ)" > results/experiment-2-idle-deploy/notes.md

# Expected outcome: fast deploy completion (<2 min) because workers are
# unprotected for ~20s at a time, giving ECS plenty of SIGTERM window.
# Watch for sigterm_received in worker-logs.jsonl from old-rev tasks.

kill ${STATE_PID} ${LOGS_PID}
```

## Experiment 3: saturated-queue scale-in

**Question**: if workers are cycling fast AND we reduce desired_count,
does DEPLOYMENT_BLOCKED fire on enable(), or does ECS manage to SIGTERM
them some other way?

**Preconditions**: service at `desired_count = 3`. Worker profile
saturated (`200 50 5`).

**Time budget**: 2 minutes. DEPLOYMENT_BLOCKED should fire on first
re-enable attempt after the scale-in signal.

```bash
# Switch back to saturated profile
./scripts/update_env.sh 200 50 5
sleep 90  # wait for new profile to take effect

# Warmup verification
aws ecs describe-services --cluster claude-ecs-task-protection-experiment \
    --services claude-ecs-task-protection-experiment \
    --query 'services[0].{running:runningCount,desired:desiredCount}'
# Should be running=3, desired=3

mkdir -p results/experiment-3-saturated-scale-in
./scripts/tail_state.sh results/experiment-3-saturated-scale-in &
STATE_PID=$!
./scripts/tail_logs.sh results/experiment-3-saturated-scale-in &
LOGS_PID=$!

# Trigger scale-in
./scripts/set_desired.sh 1
echo "Scale-in to 1 triggered at $(date -u +%Y-%m-%dT%H:%M:%SZ)" > results/experiment-3-saturated-scale-in/notes.md

# Expected: within seconds, 2 of 3 workers see `enable_failed` with
# reason=DEPLOYMENT_BLOCKED in their next enable() call.
# Stop after 2 min regardless.

kill ${STATE_PID} ${LOGS_PID}

# Restore
./scripts/set_desired.sh 3
```

## Experiment 4: idle scale-in (baseline)

**Question**: with workers idle, scale-in triggers SIGTERM during the idle
window. Verifies the "easy path" works and gives us a baseline time for
comparison with experiment 3.

**Preconditions**: service at `desired_count = 3`. Worker profile idle.

**Time budget**: 2 minutes.

```bash
# Switch to idle profile
./scripts/update_env.sh 20000 20000 5
sleep 90

mkdir -p results/experiment-4-idle-scale-in
./scripts/tail_state.sh results/experiment-4-idle-scale-in &
STATE_PID=$!
./scripts/tail_logs.sh results/experiment-4-idle-scale-in &
LOGS_PID=$!

./scripts/set_desired.sh 1
echo "Scale-in to 1 triggered at $(date -u +%Y-%m-%dT%H:%M:%SZ)" > results/experiment-4-idle-scale-in/notes.md

# Expected: 2 workers receive SIGTERM during their 20s idle sleep, exit
# cleanly within the Fargate stopTimeout. No DEPLOYMENT_BLOCKED events.

kill ${STATE_PID} ${LOGS_PID}
./scripts/set_desired.sh 3
```

## Aborting a stalled experiment

If experiment 1 or 3 isn't converging in its time budget:

1. `kill ${STATE_PID} ${LOGS_PID}` to stop the snapshot processes
2. `./scripts/set_desired.sh 0` to stop all tasks (ECS will wait for task
   protection to expire before actually terminating — up to `ExpiresInMinutes`
   = 5 min with the experiment's config)
3. Wait up to 5 min for protection to expire
4. Confirm `runningCount = 0` via `describe-services`
5. Scale back up to 3 with `set_desired.sh 3` when ready to continue

The experiment's `ExpiresInMinutes = 5` (in the terraform env vars) makes
this recovery fast. The production worker uses 30 min — we deliberately
chose 5 for the experiment so aborts are cheap.

## Filling in findings

After each experiment, record observations in `findings.md` using the
template in `findings-template.md`. Copy the template into the results
directory for each run.

## Teardown checklist

Follow every step, in order. Terraform does not own the Docker images
I built locally, the experiment results, or this directory — those
are separate cleanups.

1. **Drop ECS desired_count to 0** so tasks stop:

   ```bash
   ./scripts/set_desired.sh 0
   ```

2. **Delete the terraform-infra directory** in a follow-up PR:

   ```bash
   cd ~/projects/terraform-infra
   git checkout -b CDAM-3806-DEV-delete-experiment theorchard/master
   git rm -r dev/claude-ecs-task-protection-experiment/
   git commit -m "CDAM-3806: DEV: Remove ECS task protection experiment"
   git push
   # open PR, atlantis destroys:
   #   ECR repo (all images via force_delete), cluster, service, task defs,
   #   IAM roles + policies, log group, security group, default task def
   ```

3. **Prune local Docker images** we built:

   ```bash
   docker image rm 103233932089.dkr.ecr.us-east-1.amazonaws.com/claude-ecs-task-protection-experiment:v1 2>/dev/null
   docker image rm 103233932089.dkr.ecr.us-east-1.amazonaws.com/claude-ecs-task-protection-experiment:v2 2>/dev/null
   docker image rm 103233932089.dkr.ecr.us-east-1.amazonaws.com/claude-ecs-task-protection-experiment:latest 2>/dev/null
   docker image prune -f
   ```

4. **Archive results** to research or delete:

   ```bash
   # either move to ~/research/ecs-task-protection/
   mv ~/projects/scratches/ecs-task-protection-experiment/results ~/research/ecs-task-protection/
   # or drop them
   rm -rf ~/projects/scratches/ecs-task-protection-experiment/results
   ```

5. **Delete this directory**:

   ```bash
   rm -rf ~/projects/scratches/ecs-task-protection-experiment
   ```

## What's in terraform vs what isn't

**Terraform-managed (deleted by the teardown PR):**
- ECR repo + all images
- ECS cluster, service, all task definition revisions
- Task execution role, task role, inline policy, managed policy attachment
- CloudWatch log group + all streams
- Security group

**Not in terraform (manual cleanup per checklist above):**
- This scratch directory (worker source + scripts + README + results)
- Local Docker images built via `build_and_push.sh`
- Anything registered on AWS outside terraform for one-off debugging

## Notes on what we're NOT testing

- Fargate Spot / Savings Plans behaviour — on-demand only here
- Autoscaling CloudWatch alarms — manual `set_desired.sh` gives us
  deterministic scale-in events without the alarm pipeline as a variable
- Real SQS / SFN / S3 — the worker simulates work with `time.sleep`; the
  only thing under measurement is the task-protection + ECS deploy dance
- Multi-container pods / sidecars — single-container task