# Production Readiness Ledger

Trackable checklist of items deferred from dev. Each must be addressed before running the full **73M-fan pipeline** in the dedicated production AWS account.

Source: `CLAUDE.md` Production Readiness Checklist, `docs/token-encryption-strategy.md`.

---

## 1. AWS Account & Infrastructure

- [ ] **Dedicated AWS account provisioned** — Re-run Terraform with updated variables (bucket names, account IDs, region). All infra is codified; `terraform apply` stands everything up.
- [ ] **ECR image tag mutability** (`terraform/manifest_parser.tf:8-9`, `terraform/collector_worker.tf:8-9`) — Switch from `MUTABLE` to `IMMUTABLE`; use digest-based image references instead of `:latest`.
- [ ] **ECR KMS encryption** (`terraform/manifest_parser.tf:16-20`, `terraform/collector_worker.tf:16-20`) — Replace AWS-managed KMS (`aws/ecr`) with customer-managed KMS key via `kms_key` parameter.
- [ ] **S3 SSE-KMS on recollection prefix** — Ensure Phase 5b recollection S3 export uses SSE-KMS with CMK (not SSE-S3) for token files.
- [ ] **SQS SSE-KMS with CMK** (`terraform/sqs.tf`) — Enable SSE-KMS with customer-managed key on the manifest-files queue and its DLQ via the `terraform-sqs` module. Dispatcher splitting embeds refresh tokens directly in SQS message bodies.
- [ ] **SQS DLQ retention review** — Currently 14 days. Tokens in failed messages persist for the full retention period. Audit Lambda DLQ for partial dispatch failures containing tokens.

## 2. Secrets & Credentials

- [ ] **Spotify credentials in Secrets Manager** (`terraform/collector_worker.tf:100-102`) — Migrate `SPOTIFY_CLIENT_ID` and `SPOTIFY_CLIENT_SECRET` from Lambda env vars to Secrets Manager (`terraform-secrets-manager` module). Lambda fetches at cold start via `boto3`.
- [ ] **Snowflake sink connector secrets populated** — `dev/kc-sfsink-resonance-engine/SNOWFLAKE_PRIVATE_KEY` and `SNOWFLAKE_PRIVATE_KEY_PASSPHRASE` in Secrets Manager.
- [ ] **Snowflake service user + RSA key pair** — Create prod service user (e.g., `PROD_KAFKA_CONNECT_RESONANCE_ENGINE`) with appropriate role grants.

## 3. Cross-Account Access & Orchestration

- [ ] **DynamoDB access** — Establish cross-account IAM roles for `dynamodb:ExportTableToPointInTime` from RE account to Songwhip prod (`926734670777`), or co-locate the table.
- [ ] **EventBridge + Step Functions** — Automate daily DDB exports (currently manual). Requires cross-account DDB access first.
- [ ] **DDB token refresh writeback** (`lambdas/collector-worker/handler.py:287-292`) — Write rotated refresh tokens back to Songwhip DDB with `ConditionExpression` to prevent race conditions.

## 4. MSK / Kafka

- [ ] **MSK IAM authentication** (`lambdas/collector-worker/handler.py:231-232`, `terraform/variables.tf`) — Dev CDC cluster uses TLS-only (`kafka_security_protocol = "SSL"`). For prod: implement OAUTHBEARER callback for `AWS_MSK_IAM` auth using `aws-msk-iam-sasl-signer` package, set `kafka_security_protocol = "SASL_SSL"` and `kafka_sasl_mechanism = "AWS_MSK_IAM"`.
- [ ] **MSK topics created** — `resonance-engine.spotify-data` (6 partitions) and `dlq.resonance-engine` (3 partitions) via AKHQ or CLI.
- [ ] **Separate token management topic** — Consider a dedicated Kafka topic for token data with restricted ACLs, separate from the analytics data topic.

## 5. Snowflake

- [ ] **Masking policies applied in prod schema** — Verify `RE_TOKEN_MASK` on `RE_FAN_TOKENS.REFRESH_TOKEN` and `RE_LANDING_CONTENT_MASK` on `RE_SPOTIFY_DATA_LANDING.RECORD_CONTENT` are deployed in the prod schema. Only the READWRITE role + ACCOUNTADMIN should see plaintext.
- [ ] **RE_FAN_TOKENS SELECT restricted** — Restrict SELECT on `RE_FAN_TOKENS` to only the recollection export task's role.
- [ ] **Deduplication guard** (`snowflake/task.sql`) — Replace `INSERT` with `MERGE` keyed on `(SPOTIFY_USER_ID, ARTIST_ID/TRACK_ID, COLLECTED_AT)` to handle Kafka at-least-once delivery duplicates.
- [ ] **Clustering keys** (`snowflake/final_table.sql`) — Add `CLUSTER BY (TO_DATE(COLLECTED_AT), PARTITION_KEY)` on `RE_FAN_TOP_ARTISTS`; `CLUSTER BY (TO_DATE(PLAYED_AT), PARTITION_KEY)` on `RE_FAN_RECENTLY_PLAYED`. Enable auto-clustering only after insert patterns stabilize.
- [ ] **Task session timezone** (`snowflake/task.sql`) — Pin task session timezone to UTC via `ALTER TASK ... SET TIMEZONE = 'UTC'` to ensure `TIMESTAMP_LTZ` consistency.

## 6. Monitoring & Observability

- [ ] **Datadog Lambda monitoring** (`terraform/manifest_parser.tf:87-89`, `terraform/collector_worker.tf:87-89`) — Set `datadog_enabled = true` and `datadog_advanced_enabled = true`.
- [ ] **Datadog DLQ monitor** (`terraform/collector_worker.tf:97`) — Add Datadog monitor for the SQS DLQ when Datadog is enabled.
- [ ] **Datadog sink connector monitor** (`terraform/snowflake_sink.tf`) — Uncomment `sink_datadog_monitor` module block; update cluster name and Slack channels for prod.
- [ ] **CloudWatch log optimization** (`lambdas/manifest-parser/handler.py:138-139`) — Move per-file logging to DEBUG, log batch summaries at INFO to reduce CloudWatch cost.

## 7. Scale & Performance

- [ ] **Concurrency tuning** (`terraform/variables.tf`, `terraform/collector_worker.tf`) — Dev sweet spot: 15 workers / 100 fans per batch. For 73M fans, model concurrency against Spotify rate limits (~180 req/30s). Consider wave-based throttling or longer collection windows.
- [ ] **Fan batch size tuning** (`terraform/variables.tf`) — `fan_batch_size` controls fans per SQS message. 100 works for dev (~5.5 min/batch). For prod, may increase to 200-300 once rate limit behavior at scale is characterized.
- [ ] **Collection window planning** — At ~5-6 fans/sec zero-429 throughput (see `docs/spotify-api-rate-analysis.md`), 73M fans requires multiple Spotify apps or multi-day collection windows. Define target SLA.

---

## Completed in Dev (verify in prod)

These items are done in the dev account. Each needs verification/re-application when the prod account is provisioned.

- [x] **Snowflake masking policies deployed** (2026-02-28) — `RE_TOKEN_MASK` + `RE_LANDING_CONTENT_MASK` in `FANSIFTER_APP_REPORTING.DEV_MMACHADO`. See `snowflake/masking_policy.sql`.
- [x] **Token encryption strategy decided** (2026-02-28) — Infrastructure-layer (Option B) chosen over application-layer KMS. See `docs/token-encryption-strategy.md`.
- [x] **MSK topics created** — `resonance-engine.spotify-data` (6 partitions), `dlq.resonance-engine` (3 partitions) via AKHQ.
- [x] **Snowflake sink connector running** — `kc-sfsink-resonance-engine` on Fargate, Snowpipe Streaming ingestion.
- [x] **Landing table + streams + tasks** — `RE_SPOTIFY_DATA_LANDING`, 3 streams, 4 tasks (parent + 3 children).
- [x] **RE_FAN_TOKENS table + token extraction pipeline** — Phase 5a complete. MERGE with recency guard prevents stale overwrites.
- [x] **159K fan backfill validated** — Pipeline ran end-to-end: S3 export -> Manifest Parser -> SQS -> Collector Worker -> Kafka -> Snowflake.