# Known Rabbit Holes

Hard-won lessons from development. Read these before touching the related subsystem.

## Spotify Rate Limits

- **429 handling**: Workers MUST handle `Retry-After` header with exponential backoff + jitter. Never retry blindly. Rate limits are per-app — commonly cited at ~180 req/30s, but empirically observed at ~300-360 req/30s (see `spotify-api-rate-analysis.md`). Use ~180 req/30s as a conservative planning budget shared across all workers. At 100 concurrent workers, `Retry-After` values escalate to 28-30s causing Lambda timeouts. Sweet spot for dev backfill: **15 concurrent workers** with **100 fans/batch** — Retry-After stays in the 4-8s range.
- **Lambda Concurrency vs Rate Limits**: More Lambda concurrency does NOT increase aggregate Spotify throughput — the per-app rate budget is fixed. Higher concurrency just spreads rate limit backoff across more workers. At 100 workers, batches timeout (15 min) before completing. At 15 workers, batches complete in ~5.5 min.

## SQS

- **MaximumMessageSize**: The `terraform-sqs` module defaults `MaximumMessageSize` to 2048 bytes. Fan-batch messages at 500 fans are ~133KB, at 100 fans ~27KB. Must explicitly set `sqs_max_message_size = 262144` (256KB) in `sqs.tf`.
- **ESM Over-Claiming**: The Lambda SQS event source mapping (ESM) can claim more messages than it can process concurrently, locking them for the full visibility timeout. Always set `scaling_config { maximum_concurrency }` on the ESM to match Lambda reserved concurrency. Without this, the ESM grabs all messages, processes N at a time, and the rest sit invisible until the visibility timeout expires.
- **Visibility Timeout vs Processing Time**: Visibility timeout should be ~25% above average batch processing time, not the Lambda max timeout. With 420s visibility timeout (7 min) and ~5.5 min processing, messages recycle quickly on failure. At 900s (15 min), a single bad cycle locks messages for 15 min.
- **Batch Request Size Limit**: `send_message_batch` has a 1MB total payload limit (separate from the 256KB per-message limit). Buffering 10 fan-batch messages at 500 fans each (~133KB) exceeds 1MB. The manifest parser handler tracks cumulative byte size and flushes before 1MB.

## DynamoDB

- **Throughput**: Token refresh writes hit WCU on Songwhip. Consider On-Demand mode during collection windows. Monitor closely.

## Lambda

- **Timeouts**: At 100 fans/batch and 15 concurrent workers, batches complete in ~5.5 min well within the 15-min limit. At 500 fans/batch with high concurrency, rate limiting causes timeouts. Prefer smaller batches over higher concurrency.
- **Token Expiry Race**: Multiple workers could try to refresh the same token simultaneously. Use conditional writes (DDB `ConditionExpression`) to prevent conflicts.
- **Stale Tokens**: Some fan tokens may have been revoked. Handle gracefully — log and skip, don't retry indefinitely.
- **Docker Build (Apple Silicon)**: Must use `--platform linux/amd64 --provenance=false`. Without `--platform`, image is arm64 which Lambda rejects. Without `--provenance=false`, Docker Desktop BuildKit adds OCI attestation manifests (`application/vnd.oci.image.index.v1+json`) that Lambda doesn't support — requires `application/vnd.docker.distribution.manifest.v2+json`.
- **Image Deploy Workflow**: The terraform-lambda module ignores image URI changes after creation. After pushing a new `:latest` image, you must run `aws lambda update-function-code` to update the function. On first `terraform apply`, the ECR image must already exist or creation fails.
- **confluent-kafka + Python 3.11 (AL2 glibc)**: The `amazon/aws-lambda-python:3.11` image is AL2 (glibc 2.26). `confluent-kafka>=2.5.0` only ships `manylinux_2_28` wheels (glibc 2.28+) — no compatible prebuilt wheel, and source build fails without matching librdkafka headers. Pin to `confluent-kafka>=2.3.0,<2.5.0` (2.4.0) which provides `manylinux2014` wheels. This matches `theorchard/lambda-fan-response`. If upgrading to Python 3.12 (`public.ecr.aws/lambda/python:3.12`, AL2023/glibc 2.34+), `confluent-kafka>=2.10.0` works (see `theorchard/lambda-audience`).

## Kafka

- **Partitioning**: With 73M fans in a 24-hour window (~845 records/sec), the MSK topic needs enough partitions for parallelism. Use 6–12 partitions to allow the Fargate sink connector to scale out to multiple tasks if needed. A single connector task can handle ~5,000 rec/s, so 1 task suffices at baseline, but partitions enable horizontal scaling.
- **Producer Batching**: The Collector Worker must use asynchronous writes and batching (via `confluent-kafka` Python library) to sustain throughput. Synchronous per-record produces will bottleneck at scale.

## Terraform / IAM

- **IAM CreatePolicy in Dev**: The `generic-engineer-role` in the dev account lacks `iam:CreatePolicy`. Use `aws_iam_role_policy` (inline policies via `iam:PutRolePolicy`) instead of `aws_iam_policy` + `aws_iam_role_policy_attachment` (managed policies). The terraform-lambda module's internal policies work because they also use inline policies.
- **Fargate `task_type=worker` health check**: The `terraform-fargate` module has two separate health check variables: `web_service_health_check_command` (only applies when `task_type` is the default `web_service`) and `health_check_command` (applies to `task_type = "worker"`). For workers, `web_service_health_check_command` is silently ignored and the module falls back to the default `/bin/bash /var/app/conf/healthcheck.sh` from the Docker image. The org's existing sink connectors (`terraform-infra/prod/kafka-infra/snowflake_sink*/`) all use the default `web_service` task type (which creates an ALB + Route53 + WAF). Resonance Engine uses `task_type = "worker"` to avoid that overhead, so it must use `health_check_command` instead. Also note: `health_check_grace_period_seconds` only applies to web services (ALB-level); for workers, only `container_start_period_seconds` (Docker HEALTHCHECK startPeriod) provides a startup grace window.