# Resonance Engine **Turning fan loyalty into lasting resonance.** ## Mission To bridge the gap between artist and advocate by transforming silent listening habits into actionable connection. Resonance Engine empowers creators to identify and celebrate their most dedicated fans through transparent, consent-based data, ensuring that no loyal listener remains underappreciated. ## Overview Resonance Engine is an automated pipeline that collects Spotify behavioral data from consented fans at scale. It exports fan records from DynamoDB, refreshes OAuth tokens, calls the Spotify API, and lands the results in Snowflake for analytics. ## AWS Account Strategy The Songwhip DynamoDB table lives in a **separate production AWS account**. Resonance Engine infrastructure is deployed to a different account. - **Dev (current)**: DynamoDB export completed and staged in dev account S3. Service infrastructure (SQS, Lambdas, IAM) runs in dev (`103233932089`). MSK topics and Snowflake sink connector are managed centrally in `kafka-infra`. EventBridge + Step Functions orchestration is deferred. - **Pipeline entry point**: `s3://dev-mymac80/resonance-engine/ddb-export/01771804222395-8dd0c6a7/` - **159,316 fan tokens** (spotify-presave records with `refreshToken` + `spotifyUserId`) - **Production (future)**: A dedicated AWS account will be provisioned. Terraform re-deploys all infrastructure. Once DynamoDB access is established (cross-account IAM or co-location), the full automated pipeline runs end-to-end. ## Architecture **Dev** — Manual export from Songwhip prod, pipeline runs in dev account: ``` DynamoDB (Songwhip prod) ─── manual export + copy ───> S3 Export (dev) │ S3 Export (.json.gz) → Manifest Parser (Lambda) │ │ streams files, extracts presave fans, │ │ dispatches 100-fan batches to SQS │ ↓ │ SQS (fan batches) → Collector Worker (Lambda) x15 │ │ token refresh → Spotify API → Kafka produce │ ↓ │ MSK (Kafka) topic │ ↓ │ Snowflake Sink Connector (Fargate) │ ↓ │ Snowflake (landing → Stream + Task → analytics tables + token table) ``` **Production (future)** — Fully automated with EventBridge + Step Functions: ``` DynamoDB (Songwhip) → S3 Export → Manifest Parser (Lambda) → SQS ↑ token refresh │ └──────────────── Collector Worker (Lambda) ◄──────────────┘ │ MSK (Kafka) topic │ Snowflake Sink Connector (Fargate) │ Snowflake (landing → Stream + Task → final table) ``` ### Pipeline Phases | Phase | Component | Purpose | |-------|-----------|---------| | 1 | **Terraform** | S3 data source (existing bucket), SQS/DLQ, IAM policy documents + policy resources | | 2 | **Orchestrator** | Dev: manual DDB export + copy to S3. Future: EventBridge → Step Function → DDB Export; Manifest Parser Lambda (streams .json.gz, dispatches fan batches to SQS) | | 3 | **Collector Worker** | Processes fan batches: token refresh, Spotify API calls (top artists + recently played), MSK produce with circuit breaker for rate limiting | | 4 | **Data Sink** | MSK topics + Snowflake Sink Connector (Fargate) → landing table → Stream + Task → analytics tables + token table | | 5a | **Token Table** | `RE_FAN_TOKENS` in Snowflake, extracted via `RE_EXTRACT_TOKENS` task from Kafka messages | | 5b-1 | **Dispatcher Splitting** | Manifest Parser streams .json.gz files and dispatches 100-fan batches (not file paths). Collector Worker processes fan batches directly with Snowflake format normalization and circuit breaker | ## Key Principles - **Consent-first**: All data collection requires explicit fan opt-in - **Transparency**: Fans understand what data is collected and how it is used - **Privacy by design**: Minimize retention, anonymize where possible - **Actionable insights**: Transform listening data into meaningful artist-fan connections ## Project Structure ``` resonance-engine/ ├── terraform/ # Infrastructure as Code │ ├── main.tf # Backend, provider, VPC info, caller identity │ ├── versions.tf # Terraform + provider version pins │ ├── variables.tf # Input variables (env, S3, SQS, Lambda, Kafka, sink config) │ ├── outputs.tf # Queue URLs/ARNs, Lambda name/ARN, ECR URL, Fargate service │ ├── dev.tfvars # Dev-specific overrides (MSK bootstrap, SSL, ESM enabled) │ ├── s3.tf # S3 bucket data source (existing dev-mymac80) │ ├── sqs.tf # SQS queue + DLQ (terraform-sqs module, 256KB max message) │ ├── iam.tf # IAM policy documents + policy resources │ ├── manifest_parser.tf # ECR repo + terraform-lambda module (dispatcher) │ ├── collector_worker.tf # ECR repo + terraform-lambda + SQS ESM (MaxConcurrency=15) │ └── snowflake_sink.tf # Fargate Kafka Connect + Secrets Manager ├── lambdas/ │ ├── manifest-parser/ # Streams DDB exports, dispatches fan batches → SQS │ │ ├── handler.py # Lambda handler (Phase 5b-1 dispatcher) │ │ ├── Dockerfile # Container image (Python 3.11) │ │ ├── requirements.txt │ │ ├── requirements-dev.txt │ │ └── tests/ # 61 unit tests │ └── collector-worker/ # Fan-batch processing, token refresh, Spotify API → MSK │ ├── handler.py # Lambda handler (Phase 3 + 5b-1 fan-batch + circuit breaker) │ ├── Dockerfile # Container image (Python 3.11) │ ├── requirements.txt │ ├── requirements-dev.txt │ └── tests/ # 77 unit tests ├── snowflake/ # Landing table, Stream + Task, final table └── docs/ # Architecture documentation ``` ## Prerequisites - AWS dev account with appropriate permissions - Terraform 1.12.0 - Python 3.11+ - Spotify Developer App credentials - Snowflake account with service user + key pair for sink connector - MSK cluster access (existing org clusters in dev/prod) - Access to Songwhip DynamoDB table (prod account — for manual exports) ## Getting Started ```bash cd resonance-engine # Deploy infrastructure cd terraform awsume aws_dev terraform init terraform plan terraform apply # Build and push Manifest Parser container image cd ../lambdas/manifest-parser ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) REPO=lambda-resonance-manifest-parser VERSION=$(git rev-parse --short HEAD) aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com # --platform linux/amd64: Lambda runs x86_64 (required on Apple Silicon Macs) # --provenance=false: prevents OCI image index manifest that Lambda rejects docker build --platform linux/amd64 --provenance=false \ --build-arg version=$VERSION -t $REPO:latest . docker tag $REPO:latest $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/$REPO:latest docker push $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/$REPO:latest # Update Lambda function code with the new image aws lambda update-function-code \ --function-name lambda-resonance-manifest-parser \ --image-uri $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/$REPO:latest # Manual Lambda invoke (after image update) aws lambda invoke --function-name lambda-resonance-manifest-parser \ --cli-binary-format raw-in-base64-out \ --payload '{"bucket": "dev-mymac80", "export_prefix": "resonance-engine/ddb-export/01771804222395-8dd0c6a7/"}' \ output.json cat output.json ``` ## Status - **DDB Export**: Complete (2026-02-22) — 159K fan tokens staged in dev S3 - **Phase 1**: Complete (2026-02-22) — Infrastructure live in dev. SQS queue + DLQ, IAM policy documents, S3 data source. PRs [#2277](https://github.com/theorchard/collab/pull/2277), [#2278](https://github.com/theorchard/collab/pull/2278) merged and applied. - **Phase 2a**: Complete (2026-02-22) — Manifest Parser Lambda handler. Parses `manifest-files.json` (JSON Lines), reconstructs S3 keys. PR [#2280](https://github.com/theorchard/collab/pull/2280) merged. - **Phase 2b**: Complete (2026-02-22) — Manifest Parser Terraform deployment: ECR repo (KMS encrypted), `terraform-lambda@5.2.1` module, IAM inline role policies (S3 read + SQS send), Dockerfile (Python 3.11). PR [#2281](https://github.com/theorchard/collab/pull/2281). - **Phase 3**: Complete (2026-02-23) — Collector Worker: token refresh, Spotify API calls (top artists + recently played), MSK produce with confluent-kafka. PRs [#2285](https://github.com/theorchard/collab/pull/2285), [#2286](https://github.com/theorchard/collab/pull/2286), [#2287](https://github.com/theorchard/collab/pull/2287). - **Phase 4**: Complete (2026-02-23) — Snowflake sink connector (Fargate), landing table, Stream + Task transforms, analytics tables. Snowpipe Streaming ingestion. - **Phase 5a**: Complete (2026-02-23) — Token table (`RE_FAN_TOKENS`), token extraction stream + task. Collector Worker includes `refresh_token` + `new_refresh_token` in Kafka messages. - **Phase 5b-1**: Complete (2026-02-24) — Dispatcher splitting: Manifest Parser streams `.json.gz` files and dispatches 100-fan batches to SQS (not file paths). Collector Worker processes fan batches directly with circuit breaker for Spotify rate limits and Snowflake format normalization. PRs [#2294](https://github.com/theorchard/collab/pull/2294), [#2295](https://github.com/theorchard/collab/pull/2295). - **Backfill**: In progress (2026-02-24) — 159K fans dispatched as 1,596 batches of 100 fans. 15 concurrent Collector Workers processing at ~9 batches/5 min. - **Next**: Phase 5b-2 (Re-collection infrastructure — Snowflake COPY INTO @stage for scheduled re-collection)