# Resonance Engine

Service for collecting DSP fan data (artists, albums, tracks, playlists, etc.)

## Stack

**Backend** — Python 3.14, FastAPI, Uvicorn, Typer (CLI), pydantic-settings  
**Frontend** — React 19, TypeScript, Vite 6, Biome

## Getting started

```bash
cp .env.example .env
make install
make dev
```

- API: `http://localhost:8000` — interactive docs at `/docs`
- Dashboard: `http://localhost:5173` — Vite dev server with HMR, proxies `/api` to the API

## Environment variables

| Variable        | Default | Description                      |
|-----------------|---------|----------------------------------|
| `ENVIRONMENT`   | `dev`   | `dev` / `staging` / `production` |
| `APP_DEBUG`     | `false` | FastAPI debug mode               |
| `LOGGING_DEBUG` | `false` | Rich console logging             |

## Commands

| Command                | Description                                        |
|------------------------|----------------------------------------------------|
| `make install`         | Install Python and Node dependencies               |
| `make dev`             | Run API + dashboard in parallel                    |
| `make typecheck`       | Type-check Python (`ty`) and TypeScript (`tsc`)    |
| `make lint`            | Lint Python (`ruff`) and frontend (`biome check`)  |
| `make fmt`             | Auto-format Python (`ruff`) and frontend (`biome`) |
| `make test`            | Run pytest                                         |
| `make build-dashboard` | Build frontend → `static/dashboard/`               |
| `make docker-up`       | Build image and run via Docker Compose on `:8080`  |
| `make docker-down`     | Stop Docker Compose services                       |

## Docker

```bash
make docker-up           # :8080
PORT=9000 make docker-up # custom port
```

The Docker image builds the frontend and bundles it into the FastAPI server - no separate Node process in production.

## Fan collection

```
EventBridge cron ──► fan-fanout λ ──► SQS (per DSP) ──► fan-collect λ ×N ──► Kafka ──► Snowflake
   rate(10 min)       plan_fanout       visibility 600s    batch_size=N         + Redis stats
```

A cron tick fires the **fan-fanout** Lambda, which runs `plan_fanout`: for each
active DSP client it picks how many fans to dispatch, then sends them to that
client's SQS queue in batches. The **fan-collect** Lambdas consume one message
each (≈ a batch of fans), call the DSP API per fan, and write results to Kafka
(→ Snowflake) while recording request stats in Redis.

### Dispatch sizing (`compute_fanout_plan`)

The goal: each run keeps the fleet busy for one fanout window — no idle, no
pile-up into the next tick. Sizing is **request-based** — the currency is DSP
requests, not fans — and driven entirely by the last fanouts' **collect-task**
data (not live Redis stats). Per client:

| When                                       | Count                                              |
|--------------------------------------------|----------------------------------------------------|
| **Cold start** (no collect tasks yet)      | `floor(nominal_rps × 0.85 × window / reqs_per_fan)` (seed) |
| **Warm**                                   | `floor(rps × 0.9 × window / reqs_per_fan)`         |

The warm formula sizes the dispatch to the **observed request rate**: a throttled
or slow drain lowers `rps` and shrinks the next run; a faster drain raises it and
fills the window — self-correcting, **no cap**. It targets `FILL_SAFETY` × the
window (90%) so slack absorbs noise/stragglers.

**Signals (`CollectTask.fanout_signals`)** are medianed over the last
`_SIGNAL_FANOUTS` (3) fanouts — one noisy drain can't whipsaw sizing. Each
fanout's signal is self-contained, from its collect tasks:

- `rps = Σrequests / p90-span` — achieved fleet request rate
- `reqs_per_fan = Σrequests / Σfans`
- `latency = Σ(finished − started) / Σrequests` — per-request serial latency
- `throttle_rate = Σrate_limited / Σrequests`

p90 span (not max) so a retried batch (which lands ~one SQS visibility timeout
late) doesn't inflate the run; stale (DLQ'd) batches are dropped. The latest,
still-running fanout uses *now*, so an overrun shrinks the next run mid-flight.
`TestFanoutSizingSimulation` drives the loop end-to-end and asserts it converges
to near window-fill without ratcheting.

**Batch size** = `get_batch_size_for(get_effective_latency_s(latency))` — fans per
SQS message sized so a collect Lambda runs ~`TARGET_BATCH_DURATION_S`.

**Recommended concurrency** (dashboard advisory, `get_recommended_concurrency`) is
an **AIMD controller on the 429 signal**: `rps × latency` is the concurrency that
ran (Little's law); clean headroom (no throttling) → grow (`/ SAFETY_FACTOR`,
~+18%), throttling → back off by the 429 share. So it discovers the DSP's real
ceiling instead of trusting a guess.

`nominal_rps` (per `DSPClient`, in the DB) is **only the cold-start seed** — warm
sizing runs entirely on observed `rps`/`reqs_per_fan`, and concurrency grows past
nominal until the DSP throttles. Redis `StatsBackend` is monitoring-only; it
never feeds the plan.

### Sizing constants

| Constant                       | Default | Meaning                                                                |
|--------------------------------|---------|------------------------------------------------------------------------|
| `FAN_FANOUT_WINDOW_S`          | `600`   | Fanout window — set to match the cron cadence                          |
| `REQUESTS_PER_FAN`             | `9`     | Worst-case DSP calls per fan (refresh + 8 endpoints) — cold-start seed |
| `SAFETY_FACTOR`                | `0.85`  | Cold-start headroom + concurrency grow step (`planner.py`)             |
| `FILL_SAFETY`                  | `0.9`   | Warm target window utilisation (`planner.py`)                          |
| `RAW_REQUEST_LATENCY_S`        | `0.5`   | Default per-request latency before any drain (`planner.py`)            |
| `MIN_LATENCY_S`                | `0.05`  | Observed-latency floor (`planner.py`)                                  |
| `_SIGNAL_FANOUTS`              | `3`     | Fanouts the sizing signal is medianed over (`models.py`)              |
| `TARGET_BATCH_DURATION_S`      | `120`   | Target wall-clock per SQS message → drives `batch_size` (`planner.py`) |
| `MAX_CONCURRENCY`              | `1000`  | Cap on the recommended concurrency (`infra.py`)                        |
| `FAN_COLLECT_WORKER_TIMEOUT_S` | `540`*  | Collect Lambda timeout; the collector stops a margin before it        |
| `FAN_DISPATCH_LOCK_S`          | `7200`  | Re-selection lock after dispatch (lost-message safety net)             |

\* config default; **the deployed value comes from terraform** (see below).

### Terraform infra coupling

`get_recommended_concurrency` surfaces a *suggested* `reserved_concurrent_executions`
on the sizing dashboard (AIMD on the 429 signal). It is a **recommendation only** —
the values that actually run are hard-coded in terraform
(`terraform-infra/fansifter/<env>/resonance-engine/`) and must be synced by hand.

| Knob                             | Where                                     | QA value           | Notes                                                                                            |
|----------------------------------|-------------------------------------------|--------------------|--------------------------------------------------------------------------------------------------|
| Cron cadence                     | `lambda.tf` (`cloudwatch_event_schedule`) | `rate(10 minutes)` | Should match `FAN_FANOUT_WINDOW_S`                                                               |
| `reserved_concurrent_executions` | `lambda.tf` (fan-collect)                 | `5`                | Caps in-flight request rate vs the DSP budget                                                    |
| ESM `batch_size`                 | `lambda.tf`                               | `1`                | One SQS message → one invocation (≈ one app-side batch)                                          |
| `fan_collect_worker_timeout_s`   | `lambda.tf` (local)                       | `900`              | Collect Lambda timeout (and the collect-task drain span the planner sizes on)                    |
| SQS `visibility_timeout`         | `sqs.tf`                                  | `960`              | **Must be** `worker_timeout_s + 60`; failed/timed-out messages redeliver only after this elapses |
| `dlq_max_receive_count`          | `sqs.tf`                                  | `3`                | Retries before a message lands on the DLQ                                                        |

**Invariant:** `fan_collect_worker_timeout_s` is the single source of truth for
the collect timeout and must stay in sync across three places — the fan-collect
Lambda `timeout`, the SQS `visibility_timeout` (`= this + 60`), and the
`FAN_COLLECT_WORKER_TIMEOUT_S` env var. Lowering `reserved_concurrent_executions`
below what the throughput needs risks queue backup; raising it past the DSP rate
limit triggers 429s (which the recommended-concurrency AIMD then backs off from).