# Resonance Engine — Project Guide

## Mission

To bridge the gap between artist and advocate by transforming silent listening habits into actionable connection. Resonance Engine empowers creators to identify and celebrate their most dedicated fans through transparent, consent-based data, ensuring that no loyal listener remains underappreciated.

**Tagline:** Turning fan loyalty into lasting resonance.

## What This Project Does

Resonance Engine uses the Spotify API to collect behavioral data from fans who have explicitly consented to data collection. Fan tokens and consent records live in a DynamoDB table (Songwhip). The system performs daily bulk exports from DynamoDB to S3, parses the export manifests, refreshes OAuth tokens as needed, calls the Spotify API for each consented fan, produces results to MSK (Kafka), and sinks them into Snowflake via a Kafka Connect Snowflake sink connector for downstream analytics.

## Project Tracking

**Notion Board:** https://www.notion.so/30f97177520f81adbc3bf1af603081c8

Kanban board organized by:
- **Status**: To Do, In Progress, Done, Blocked
- **Phase**: Breadboard, Phase 1: Infrastructure, Phase 2: Orchestrator, Phase 3: Collector Worker, Phase 4: Data Sink, Phase 5: Architecture Evolution, Documentation
- **Priority**: High, Medium, Low

**Workflow:**
1. At the start of each session, review the Notion board to understand current state
2. Update task status as work progresses
3. Create new tasks for discovered work
4. Document blockers and add context to task cards

## AWS Account Strategy

The Songwhip DynamoDB table (`songwhip-release-tasks-production`) lives in its own **separate production AWS account**. Resonance Engine infrastructure is deployed to a different account. This cross-account boundary shapes the development and deployment approach.

### Current Phase: Dev Account

All Resonance Engine infrastructure runs in the **AWS dev account** (`103233932089`).

#### DDB Export

159,316 spotify-presave records exported from Songwhip DDB to `s3://dev-mymac80/resonance-engine/ddb-export/`. Format: DynamoDB JSON (`.json.gz`). Full details: [`docs/ddb-export.md`](docs/ddb-export.md)

#### AWS Profiles

| Profile | Account | Purpose |
|---------|---------|---------|
| `aws_dev` | `103233932089` | Dev account — pipeline infrastructure |
| `songwhip` | `926734670777` | Songwhip prod — DynamoDB source (read-only) |
| `prod` | `437795906467` | Parent account for role assumption |

#### AWS Credentials in Claude Code

`awsume` sets credentials as environment variables in the user's terminal, which are **not inherited** by Claude Code's shell sessions. Always prefix AWS commands with `eval "$(awsume <profile> -s 2>/dev/null)"` to source credentials inline:

```bash
# Correct — sources awsume credentials into the command's shell
eval "$(awsume aws_dev -s 2>/dev/null)" && terraform plan

# Wrong — credentials are not available in Claude Code's shell
terraform plan  # will fail with ExpiredToken
```

This pattern must be used for **every** Bash tool invocation that requires AWS credentials (Terraform, AWS CLI, etc.).

#### Snowflake

- **Songwhip PostgreSQL** synced via Fivetran at `SONGWHIP_APP_REPORTING.PROD_SONGWHIP_API_PUBLIC`. The **DynamoDB table is NOT synced** — the DDB export is the only source for presave/token data.
- **RE target schema**: `FANSIFTER_APP_REPORTING.DEV_MMACHADO` — all objects prefixed `RE_*`. Full inventory: [`docs/snowflake-objects.md`](docs/snowflake-objects.md)
- **Warehouse**: `DEV_OWS_WH`

### Future: Dedicated AWS Account

A new dedicated AWS account will be provisioned. All infrastructure is Terraform-codified for re-deployment. Cross-account DDB access or co-location TBD. Design uses `environment` variable, no hardcoded account IDs, configurable S3 bucket names.

## Core Principles

- **Consent-first**: Never collect or process data without explicit fan opt-in.
- **Transparency**: Fans must understand what data is collected, how it is stored, and how it is used.
- **Privacy by design**: Minimize data retention. Store only what is needed. Anonymize where possible.
- **Actionable over exhaustive**: Prioritize insights that lead to real artist-fan connection over raw data accumulation.

## Reference Repositories

| Repository | Local Path | Purpose |
|-----------|------------|---------|
| `theorchard/terraform-infra` | `../../../terraform-infra/` | Org Terraform monorepo — reference patterns for Fargate sink connectors at `prod/kafka-infra/snowflake_sink*/`, Lambda modules, IAM conventions. **Always use this relative path** — never scan the home directory. |
| `theorchard/lambda-audience` | — | Kafka producer Lambda patterns (songwhip-event-processor) — Python 3.12/AL2023 multi-stage uv build, `confluent-kafka>=2.10` |
| `theorchard/lambda-fan-response` | — | Fan response Lambdas (smf-fan-response) — Python 3.11/AL2, `confluent-kafka==2.4.0`, simple Dockerfile with gcc install |

## Prior Art: Heavy Rotation POC

The `heavy-rotation/streamlit-test/` project is a working POC that validates core patterns at campaign level (~2,000 fans). Resonance Engine scales this to ~73M fans. Key modules in `heavy-rotation/streamlit-test/lib/`: `spotify_auth.py` (OAuth), `spotify_client.py` (HTTP + 401/429 handling), `wave_processor.py` (parallel fan processing). The POC calls `GET /v1/me/top/artists`; Resonance Engine uses both top artists and recently played.

## Architecture Overview

Three data flows: (0) **Backfill** — one-time batch pipeline seeds `RE_FAN_TOKENS` from DDB export, (2) **Re-collection** — Snowflake exports token files to S3 → same SQS queue → same Collector Worker, (1) **Fan onboarding** (future) — DDB Streams → EventBridge → SQS FIFO. The Collector Worker auto-detects file format (DDB JSON vs Snowflake JSON).

Full diagrams, design decisions, and data flows: [`docs/architecture.md`](docs/architecture.md)

Phased build plan (Phases 1-5d): [`docs/build-plan.md`](docs/build-plan.md)

## Tech Stack

- **IaC**: Terraform
- **Runtime**: Python 3.11+ (Lambdas)
- **API**: Spotify Web API
- **Auth**: Spotify OAuth 2.0 (Authorization Code Flow)
- **Source DB**: DynamoDB (Songwhip — fan tokens & consent)
- **Orchestration**: EventBridge + Step Functions
- **Queue**: SQS with DLQ
- **Streaming**: MSK (Managed Kafka) → Snowflake Sink Connector (Fargate)
- **Warehouse**: Snowflake (Snowpipe Streaming ingestion, Stream + Task for transformation)

## DynamoDB Schema (Songwhip)

**Table**: `songwhip-release-tasks-production`

| Field | Type | Description |
|-------|------|-------------|
| `partitionKey` | String | `group:album{id}` (V1) or `group:prerelease{id}` (V2) |
| `sortKey` | String | `task:spotify-presave:{userId}` |
| `refreshToken` | String | Spotify OAuth refresh token |
| `spotifyUserId` | String | Spotify user ID |

## Spotify API Reference

### Endpoints (Scoped to Consented Fans)

| Endpoint | Scope Required | Purpose |
|----------|---------------|---------|
| `GET /v1/me/top/artists` | `user-top-read` | Top artists by listening affinity |
| `GET /v1/me/player/recently-played` | `user-read-recently-played` | Recent listening history |
| `GET /v1/me/top/tracks` | `user-top-read` | Top tracks by listening affinity |
| `GET /v1/me/tracks` | `user-library-read` | Saved tracks (liked songs) |
| `GET /v1/me/following` | `user-follow-read` | Followed artists |

Token refresh: `POST accounts.spotify.com/api/token` with Basic auth (`base64(client_id:client_secret)`). Rate limits: commonly cited at ~180 req/30s per app, with internal analysis observing ~300-360 req/30s; design for the conservative limit and handle 429 with `Retry-After` + exponential backoff + jitter. See [`docs/spotify-api-rate-analysis.md`](docs/spotify-api-rate-analysis.md).

## Production Readiness

Items deferred from dev. Full trackable checklist with file references: [`docs/production-readiness.md`](docs/production-readiness.md).

**Categories** (22 items across 7 areas):
1. AWS Account & Infrastructure (6) — dedicated account, ECR immutability, KMS encryption, S3/SQS SSE-KMS
2. Secrets & Credentials (3) — Spotify creds to Secrets Manager, Snowflake service user
3. Cross-Account Access & Orchestration (3) — DDB access, EventBridge automation, token writeback
4. MSK / Kafka (3) — IAM auth, topic creation, token topic ACLs
5. Snowflake (5) — masking in prod, SELECT restriction, dedup guard, clustering, timezone
6. Monitoring & Observability (4) — Datadog Lambda/DLQ/sink, CloudWatch optimization
7. Scale & Performance (3) — concurrency tuning, batch sizing, collection window planning

## Known Rabbit Holes

Key gotchas — full explanations in [`docs/rabbit-holes.md`](docs/rabbit-holes.md):
- **Spotify rate limit sweet spot**: 15 concurrent workers, 100 fans/batch (429 Retry-After stays 4-8s)
- **More Lambda concurrency != more Spotify throughput** — per-app rate budget is fixed
- **SQS max message size**: Module defaults to 2048 bytes, must set 262144
- **SQS ESM over-claiming**: Set `scaling_config { maximum_concurrency }` to match reserved concurrency
- **SQS visibility timeout**: ~25% above avg processing time, not Lambda max timeout
- **SQS batch request**: 1MB total limit — flush before hitting it
- **Docker build (Apple Silicon)**: `--platform linux/amd64 --provenance=false`
- **Lambda image deploy**: Module ignores URI changes — use `aws lambda update-function-code`
- **confluent-kafka on Python 3.11/AL2**: Pin `<2.5.0` (needs manylinux2014 wheels)
- **IAM CreatePolicy in dev**: Use `aws_iam_role_policy` (inline), not managed policies
- **Fargate worker health check**: Use `health_check_command`, not `web_service_health_check_command`
- **Kafka producer batching**: Must use async writes via `confluent-kafka` to sustain throughput

## Project Structure

```
resonance-engine/
├── CLAUDE.md
├── README.md
├── terraform/
│   ├── main.tf               # Backend, providers, VPC info, ACM cert, Route53 zone
│   ├── versions.tf           # Terraform + provider version pins
│   ├── variables.tf          # Input variables
│   ├── outputs.tf            # Queue URLs/ARNs, Lambda name/ARN, ECR URL, Fargate service
│   ├── s3.tf                 # S3 bucket data source
│   ├── sqs.tf                # SQS queue + DLQ
│   ├── iam.tf                # IAM policy documents + policy resources
│   ├── manifest_parser.tf    # ECR repo + terraform-lambda module
│   ├── collector_worker.tf   # ECR repo + terraform-lambda + SQS event source
│   └── snowflake_sink.tf     # Fargate Kafka Connect + Secrets Manager (Phase 4b)
├── lambdas/
│   ├── manifest-parser/
│   │   ├── handler.py
│   │   ├── Dockerfile         # Container image (Python 3.11)
│   │   ├── requirements.txt
│   │   ├── requirements-dev.txt
│   │   └── tests/
│   └── collector-worker/
│       ├── handler.py
│       ├── Dockerfile         # Container image (Python 3.11)
│       ├── requirements.txt
│       ├── requirements-dev.txt
│       └── tests/
├── snowflake/
│   ├── landing_table.sql
│   ├── final_table.sql
│   ├── stream.sql
│   ├── task.sql
│   ├── token_table.sql      # RE_FAN_TOKENS (Phase 5a)
│   ├── token_stream.sql     # Dedicated stream for token extraction (Phase 5a)
│   ├── token_task.sql       # RE_EXTRACT_TOKENS child task (Phase 5a)
│   └── masking_policy.sql   # Dynamic Data Masking for refresh tokens
└── docs/
    ├── architecture.md              # Flow diagrams, design decisions, data flows
    ├── build-plan.md                # Phased build plan (1-5d)
    ├── ddb-export.md                # DDB export details & record types
    ├── dev-setup.md                 # Pre-requisites for running pipeline in dev
    ├── production-readiness.md      # Production readiness checklist & tracking
    ├── rabbit-holes.md              # Full explanations of known gotchas
    ├── snowflake-objects.md         # RE_* object inventory & sink connector config
    ├── spotify-api-rate-analysis.md # Empirical rate limit analysis from 159K backfill
    ├── terraform-standards.md       # Org Terraform conventions & code examples
    └── token-encryption-strategy.md # Encryption decision + org pattern analysis
```

## Terraform Standards

Follows conventions from `terraform-infra` monorepo (`../../../terraform-infra/`). Key rules: use `aws_iam_role_policy` (inline) not managed policies, `terraform-lambda@5.2.1` for container images, `terraform-fargate` for sink connectors, `aws_iam_policy_document` data sources (not inline JSON), exact version pinning on modules, `terraform fmt` before committing.

Full standards with code examples: [`docs/terraform-standards.md`](docs/terraform-standards.md)

## Pre-Requisites for Running Pipeline in Dev

1. Set `TF_VAR_spotify_client_id` and `TF_VAR_spotify_client_secret` env vars
2. `terraform apply -var-file=dev.tfvars` (sets Kafka bootstrap servers, SSL, ESM enabled)
3. MSK topics created via AKHQ (already done for dev)
4. Snowflake sink connector secrets populated in Secrets Manager
5. ECR images pushed before first `terraform apply`

Full commands and details: [`docs/dev-setup.md`](docs/dev-setup.md)

## Running Tests

No project-level virtual environment is configured. Create a temporary venv and install dev dependencies:

```bash
python3 -m venv /tmp/re-test-venv
/tmp/re-test-venv/bin/pip install -r lambdas/<lambda-name>/requirements-dev.txt -r lambdas/<lambda-name>/requirements.txt --quiet
/tmp/re-test-venv/bin/pytest lambdas/<lambda-name>/tests/ -v
```

Examples:
```bash
# Manifest Parser tests
/tmp/re-test-venv/bin/pytest lambdas/manifest-parser/tests/ -v

# Collector Worker tests
/tmp/re-test-venv/bin/pytest lambdas/collector-worker/tests/ -v
```

**Note:** `python` is not on PATH — use `python3` (or `python3.13` via Homebrew). Lambda runtime is Python 3.11; tests are compatible with 3.11+.

## Development Guidelines

- All infrastructure changes go through Terraform — no manual console edits
- Keep Lambdas focused and single-purpose
- Write tests for data transformation, manifest parsing, and scoring logic
- Never hardcode credentials — use environment variables or AWS Secrets Manager
- Log consent events explicitly for auditability
- Handle Spotify API rate limits with backoff/retry (exponential + jitter)
- Use DynamoDB conditional writes for token refresh to avoid race conditions
- Python runtime for all Lambdas (consistent with POC patterns)