# Pre-Requisites for Running Pipeline in Dev

Before running the backfill or re-collection pipeline, the following must be configured.

## 1. Terraform Variables (Spotify Credentials)

Spotify OAuth credentials are **not** stored in Terraform state or tfvars files. Pass them via environment variables before `terraform plan/apply`:

```bash
export TF_VAR_spotify_client_id="<spotify-app-client-id>"
export TF_VAR_spotify_client_secret="<spotify-app-client-secret>"
eval "$(awsume aws_dev -s 2>/dev/null)" && terraform -chdir=terraform plan -var-file=dev.tfvars
```

The Spotify app is the same one used by Songwhip for presave OAuth. Credentials are in the Spotify Developer Dashboard under the presave app.

## 2. Terraform Apply with dev.tfvars

All dev-specific overrides live in `terraform/dev.tfvars`. This file sets:
- `kafka_bootstrap_servers` — MSK CDC cluster TLS endpoints (port 9094)
- `kafka_security_protocol = "SSL"` — the dev MSK cluster uses TLS-only (no IAM auth)
- `kafka_sasl_mechanism = ""` — not needed for TLS-only clusters
- `collector_worker_esm_enabled = true` — enables the SQS event source mapping

Always pass `-var-file=dev.tfvars` when running Terraform:

```bash
eval "$(awsume aws_dev -s 2>/dev/null)" && terraform -chdir=terraform apply -var-file=dev.tfvars
```

## 3. MSK Topics (AKHQ — One-Time Setup)

Create topics via AKHQ before first run (already done for dev):
- `resonance-engine.spotify-data` (6 partitions)
- `dlq.resonance-engine` (3 partitions)

## 4. Snowflake Sink Connector Secrets (One-Time Setup)

After `terraform apply`, populate Secrets Manager secrets:
- `dev/kc-sfsink-resonance-engine/SNOWFLAKE_PRIVATE_KEY`
- `dev/kc-sfsink-resonance-engine/SNOWFLAKE_PRIVATE_KEY_PASSPHRASE`

## 5. ECR Images Must Exist

The terraform-lambda module ignores image URI changes after creation. Before first `terraform apply`, push container images:

```bash
# Build and push (from repo root)
eval "$(awsume aws_dev -s 2>/dev/null)"
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 103233932089.dkr.ecr.us-east-1.amazonaws.com

# Manifest Parser
docker build --platform linux/amd64 --provenance=false -t lambda-resonance-manifest-parser:latest lambdas/manifest-parser/
docker tag lambda-resonance-manifest-parser:latest 103233932089.dkr.ecr.us-east-1.amazonaws.com/lambda-resonance-manifest-parser:latest
docker push 103233932089.dkr.ecr.us-east-1.amazonaws.com/lambda-resonance-manifest-parser:latest

# Collector Worker
docker build --platform linux/amd64 --provenance=false -t lambda-resonance-collector-worker:latest lambdas/collector-worker/
docker tag lambda-resonance-collector-worker:latest 103233932089.dkr.ecr.us-east-1.amazonaws.com/lambda-resonance-collector-worker:latest
docker push 103233932089.dkr.ecr.us-east-1.amazonaws.com/lambda-resonance-collector-worker:latest
```

After pushing new images, update the Lambda function code:

```bash
eval "$(awsume aws_dev -s 2>/dev/null)"
aws lambda update-function-code --function-name dev-lambda-resonance-collector-worker \
  --image-uri 103233932089.dkr.ecr.us-east-1.amazonaws.com/lambda-resonance-collector-worker:latest
aws lambda update-function-code --function-name dev-lambda-resonance-manifest-parser \
  --image-uri 103233932089.dkr.ecr.us-east-1.amazonaws.com/lambda-resonance-manifest-parser:latest
```