# SMF Rate Comparison Test — Plan

## Context

We've been using the **Songwhip** Spotify app's client ID for the Resonance Engine pipeline. A new data dump of refresh tokens from a different app (**SMF**) is available. The goal is to compare Spotify API rate limits between the two apps to understand if SMF has a different rate budget — this directly impacts production throughput planning.

The previous Songwhip rate analysis (`docs/spotify-api-rate-analysis.md`) found ~10-12 req/sec (~300-360/30s) with 15 concurrent workers. We want an apples-to-apples comparison with SMF credentials.

## Approach: Local First, Then Lambda

### Step 1: Get the file and inspect it

- User downloads the SMF token file from Google Drive to local machine
- Inspect the file to determine format (CSV, JSON, JSONL, etc.) and identify the key fields (`spotify_user_id`, `refresh_token`, etc.)
- Determine record count

### Step 2: Upload to S3

- Upload the file to `s3://dev-mymac80/resonance-engine/smf-tokens/` (new prefix under existing dev bucket)
- Keep original format — the local test script will handle parsing

### Step 3: Write a local rate test script

**File:** `scripts/smf_rate_test.py`

Standalone Python script that:
1. Reads a configurable number of fan tokens from the SMF file (local or S3)
2. Accepts SMF `client_id` and `client_secret` via env vars (`SMF_CLIENT_ID`, `SMF_CLIENT_SECRET`)
3. Sequentially (or with configurable concurrency) refreshes tokens and calls the same two Spotify endpoints:
   - `GET /v1/me/top/artists?limit=50&time_range=medium_term`
   - `GET /v1/me/player/recently-played?limit=50`
4. Logs per-request metrics:
   - HTTP status code
   - Response time (ms)
   - `Retry-After` header value (on 429s)
   - Request sequence number and timestamp
5. Prints a summary at the end:
   - Total requests, success count, 429 count, error count
   - Average/p50/p95 response time
   - Average Retry-After duration
   - Effective throughput (successful requests/sec)

**Reuse from existing code:**
- Token refresh logic pattern from `lambdas/collector-worker/handler.py:_refresh_token()` (lines ~93-140)
- Spotify API call pattern from `handler.py:_call_spotify_api()` (lines ~142-219)
- Rate limit handling pattern (exponential backoff + jitter)

**Key differences from collector worker:**
- No Kafka producer — just logging metrics
- No SQS/Lambda context — plain script
- Configurable concurrency via `ThreadPoolExecutor` (start with 1, then scale up)
- CSV/JSON output of per-request metrics for analysis

### Step 4: Run local comparison

1. Run with **SMF creds** against ~100-500 SMF tokens, single-threaded first
2. Run with **Songwhip creds** against ~100-500 Songwhip tokens (from existing DDB export), single-threaded
3. Compare 429 rates, Retry-After distributions, and throughput
4. If results look promising, scale up concurrency (5, 10, 15 workers) to find the SMF rate ceiling

### Step 5: Lambda pipeline (if warranted)

If local tests show meaningfully different rate limits:
- Format SMF tokens into the collector worker's expected format (fan batch JSON)
- Upload to S3 in compatible format
- Swap Lambda env vars to SMF credentials
- Run through the full pipeline for a realistic scale test
- Compare CloudWatch metrics with the Songwhip backfill analysis

## Files to Create

| File | Purpose |
|------|---------|
| `scripts/smf_rate_test.py` | Standalone rate comparison script |

## Files to Reference (read-only patterns)

| File | What to reuse |
|------|---------------|
| `lambdas/collector-worker/handler.py` | Token refresh, API call, rate limit patterns |
| `docs/spotify-api-rate-analysis.md` | Benchmark numbers to compare against |
| `scripts/create_test_subset.py` | S3 upload pattern |

## Verification

1. **Token validity**: Script should report how many tokens successfully refresh vs fail (revoked/expired) — this tells us if the SMF dump is usable
2. **Rate comparison**: Side-by-side metrics output for SMF vs Songwhip at same concurrency levels
3. **No secrets in code**: Credentials via env vars only, never hardcoded