# Experiment 3: saturated-queue scale-in — findings

**Run date/time (UTC):** 2026-04-17T16:29:07Z

**Setup confirmed:**
- [x] Service at desired_count = 3 (just after experiment 1 completed)
- [x] Workers cycling saturated profile (200ms work / 50ms idle)
- [x] tail_state.sh + tail_logs.sh running

**Trigger:**
- Command: `./scripts/set_desired.sh 1`
- Trigger time (UTC): 2026-04-17T16:29:07Z

**Convergence:**
- 16:29:09 — 2 workers hit `enable_failed reason=TASK_STOPPING_OR_STOPPED` within 31 ms
- 16:29:09 — ECS event: "stopped 2 running tasks"
- 16:29:19 — steady state reached (service-level event)
- **Total: 12 seconds from trigger to steady state**

**Task fate:**
- 2 of 3 tasks exited via `TASK_STOPPING_OR_STOPPED`
- 1 task remained (matches new desired=1)

**Failure reason observed:**
- `reason=TASK_STOPPING_OR_STOPPED` — **not** `DEPLOYMENT_BLOCKED`
- Elapsed time per failing call: 39 ms (similar to successful calls)

**SIGTERM observations:**
- Total `sigterm_received` events: **0**
- Scale-in operated entirely through the task-protection agent endpoint

## Major finding: scale-in uses a different reason than deploys

- **Experiment 1 (deploy)**: `reason=DEPLOYMENT_BLOCKED`
- **Experiment 3 (scale-in)**: `reason=TASK_STOPPING_OR_STOPPED`

Our earlier research (and my code review pass) had concluded `TASK_STOPPING_OR_STOPPED` was "empirically observed but not documented — might be stale / renamed." It is very much real. It fires when ECS has already decided to stop a specific task and transitioned it to STOPPING state, after which the agent endpoint refuses new protection on that task.

Hypothesis for the timing difference:
- **Deploy path**: ECS sees `protected_count > desired_count` at the service level → `DEPLOYMENT_BLOCKED` on ANY task's next enable attempt. Task identity doesn't matter; it's a service-wide counter.
- **Scale-in path**: ECS picks specific tasks to stop (per the scale-in policy) → marks them STOPPING → those tasks' next enable attempts get `TASK_STOPPING_OR_STOPPED`.

This explains:
- Why experiment 3 converged so much faster (12 s vs 2:47) — direct per-task marking, no service-level counter waiting.
- Why only 2 of 3 tasks got the failure — ECS targeted 2 specific tasks (the amount to remove), not the whole service.
- Why experiment 1 got `DEPLOYMENT_BLOCKED` across all 3 old tasks simultaneously — service-wide gate flipping open.

## Decision implications for CDAM-3806

- **Production task_protection.py should restore `TASK_STOPPING_OR_STOPPED` handling** as a distinct result. I had removed it because we couldn't verify the reason was real. It IS real; the experiment confirms it.
- **Production app.py has a subtle bug**: it currently only releases + exits on `ProtectionResult.DEPLOYMENT_BLOCKED`. Any other FAILED result (including TASK_STOPPING_OR_STOPPED today, or any future reason AWS adds) would cause the worker to continue processing unprotected. Should change to: **any non-ENABLED result triggers release + exit** (except `UNAVAILABLE`, which is the local-dev case and should proceed without protection).
- **Scale-in is load-bearing on `TASK_STOPPING_OR_STOPPED` in practice**, not `DEPLOYMENT_BLOCKED`. The mental model "DEPLOYMENT_BLOCKED covers both" is imprecise — true, but via different code paths.

## Raw data

- `worker-logs.jsonl` — full cycle logs for 3 tasks
- `service-state.jsonl` / `task-state.jsonl` — scale-in progression
- `service-events.jsonl` — ECS scheduler events

## Teardown
- Service now at `desired_count = 1` running 1 task
- Will scale to 0 next (for cost control during findings write-up)