# Spotify API Rate Limit Analysis

Observed during the 159K fan backfill on 2026-02-24. All data from CloudWatch Logs
for `dev-lambda-resonance-collector-worker` and Snowflake landing table metrics.

## Configuration

| Parameter | Value |
|-----------|-------|
| Concurrent workers (Lambda) | 15 |
| Fans per batch | 100 |
| Visibility timeout | 420s |
| Lambda timeout | 420s |
| Retry-After threshold (circuit breaker) | 10s |
| Max rate limit retries per call | 3 |

## Endpoints Hit Per Fan

Each fan triggers 3 HTTP requests to Spotify:

| # | Host | Endpoint | Purpose |
|---|------|----------|---------|
| 1 | `accounts.spotify.com` | `POST /api/token` | Refresh OAuth token |
| 2 | `api.spotify.com` | `GET /v1/me/top/artists?limit=50&time_range=medium_term` | Top artists (affinity) |
| 3 | `api.spotify.com` | `GET /v1/me/player/recently-played?limit=50` | Recent listening history |

Token refresh (`accounts.spotify.com`) appears to have a separate, higher rate limit and is
not a bottleneck. The analysis below focuses on `api.spotify.com` endpoints.

## Observed 429 Pattern (5-minute sample)

### Volume

| Metric | Count |
|--------|-------|
| Total 429 responses | 672 |
| `/me/player/recently-played` | 326 (48%) |
| `/me/top/artists` | 295 (44%) |
| Unique calls that hit at least one 429 | 468 (11% of all API calls) |

429s are split roughly 50/50 across both endpoints, suggesting a shared per-app
rate budget rather than per-endpoint limits.

### Retry Progression

| Attempt | Count | Interpretation |
|---------|-------|----------------|
| 1st retry | 468 | Initial 429 hit |
| 2nd retry | 203 | Still rate limited after first backoff |
| 3rd retry | 1 | Rare — almost all resolve by attempt 2 |

Of the 468 calls that hit a 429: ~265 recovered on the 1st retry, ~202 on
the 2nd retry, and just 1 required a 3rd. Nearly all recover without
exhausting retries or tripping the circuit breaker.

### Retry-After Distribution

| Retry-After (s) | Occurrences |
|------------------|-------------|
| 1 | 17 |
| 2 | 3 |
| 3 | 8 |
| **4** | **170** |
| **5** | **118** |
| **6** | **81** |
| 7 | 33 |
| 8 | 5 |
| **Average** | **4.8s** |

Distribution covers 435 of 672 total 429 responses. The remaining 237
were not captured in the sampled log window used for Retry-After extraction
(the 672 total comes from the full 5-minute window aggregate count).

Stable 4-5s range indicates we are moderately over the limit (1.2-1.5x), not
severely. At 100 concurrent workers this escalates to 28-30s (observed in earlier
testing), confirming rate limit is per-app and shared across all workers.

## Rate Limit Estimation

### Request Volume Breakdown (5-minute window)

| Metric | Value |
|--------|-------|
| Batches completed | ~22 |
| Fans attempted | ~2,200 |
| Token refresh failures (revoked) | 74 |
| Fans making API calls | ~2,126 |
| Unique API call attempts (2 per fan) | ~4,252 |
| Retry attempts (429 re-sends) | 672 |
| **Total HTTP requests to `api.spotify.com`** | **~4,924** |

### Derived Rates

| Metric | Value |
|--------|-------|
| Total request rate | ~16.4 req/sec (~492/30s) |
| Retry overhead (wasted requests) | ~14% |
| Effective successful rate | ~14.2 req/sec (~425/30s) |
| Fan throughput | ~7.5 fans/sec (~450 fans/min) |

### Estimated True Rate Limit

The Retry-After of 4-5s with stable (non-escalating) values means we are
approximately 1.3x over the limit. Working backwards:

**Estimated `api.spotify.com` rate limit: ~10-12 req/sec (~300-360/30s)**

This is higher than the commonly cited community figure of ~180 req/30s. Possible
explanations:

1. The ~180 figure may be per-endpoint, and our two endpoints share a combined
   budget of ~300-360/30s
2. Spotify may have increased limits since the community benchmarks were published
3. Rate limits may vary by app registration tier or usage history

## Zero-429 Throughput Projections

| Scenario | req/sec | fans/sec | fans/min | Workers |
|----------|---------|----------|----------|---------|
| **Current (with 429s)** | ~16.4 | ~7.5 | ~450 | 15 |
| **Zero 429s (conservative)** | ~10 | ~5 | ~300 | 8-10 |
| **Zero 429s (aggressive)** | ~12 | ~6 | ~360 | 10-12 |

### Options for Zero-429 Operation

1. **Reduce concurrency to 10 workers** — natural per-fan processing time (~1s)
   spaces out requests enough to stay under the limit
2. **Add inter-request delay** — 100-150ms between API calls at 15 workers achieves
   the same effect without reducing parallelism
3. **Accept the 429s** — current backoff pattern is stable, only 14% overhead,
   and the circuit breaker prevents escalation

### Recommendation

The current approach (15 workers with backoff) is pragmatic for the one-time
backfill. The 14% retry overhead is acceptable given the simplicity — no
throttle coordination needed across workers. For scheduled re-collection
(Phase 5b), consider reducing to 10 workers to eliminate 429s entirely and
be a better API citizen.

## Production Scale Implications (73M fans)

At the current effective rate of ~7.5 fans/sec:

| Metric | Value |
|--------|-------|
| Total fans | 73,000,000 |
| Effective rate | 7.5 fans/sec |
| Time to complete | ~2,704 hours (~113 days) |

This confirms that a single Spotify app's rate budget cannot process 73M fans
in a 24-hour window. Production options:

1. **Multiple Spotify apps** — each with its own rate budget, partitioned by fan cohort
2. **Extended collection window** — weekly or bi-weekly collection instead of daily
3. **Priority-based collection** — collect high-value fans daily, others on a rotation
4. **Hybrid approach** — combine options 2 and 3, collecting the most active/stale
   fans first (already implemented via `LAST_COLLECTED_AT ASC` ordering)

---

## SMF App Rate Comparison (2026-03-30)

A second Spotify app (SMF) was tested to compare rate budgets. The SMF app has a
separate set of 11.7M refresh tokens from a different OAuth integration. The test
used the same endpoints and methodology as the Songwhip analysis above.

### Token Dump Profile

| Metric | Value |
|--------|-------|
| Source | SMF app (separate from Songwhip) |
| Format | Flat JSON array of refresh token strings (no user IDs) |
| Total tokens | 11,706,000 |
| File size | 1.5 GB uncompressed |
| Valid token rate | ~41% (remaining are "Invalid client" or revoked) |
| S3 location | `s3://dev-mymac80/resonance-engine/smf-tokens/spotify.json` |

"Invalid client" errors indicate the dump contains tokens issued by other Spotify
apps — only ~41% were issued by the SMF app and can be refreshed with its credentials.

### Local Test Configuration

All tests run from a local machine (macOS) using `scripts/rate_test.py` with
`ThreadPoolExecutor`. Same two `api.spotify.com` endpoints as the Songwhip test
(top artists + recently played), plus token refresh on `accounts.spotify.com`.

### Results by Concurrency Level

| Workers | Tokens | Successful req/s | Fans/s | 429 rate | Mean latency | P95 latency |
|---------|--------|-------------------|--------|----------|-------------|-------------|
| 1 | 500 | 2.7 | 0.9 | **0%** | 293ms | 577ms |
| 15 | 1,000 | 41.8 | 13.9 | **0%** | 280ms | — |
| 50 | 2,000 | 117.7 | 39.2 | **0%** | 330ms | 715ms |
| 100 | 3,000 | 113.8 | 37.9 | **0%** | 639ms | 1,371ms |
| 200 | 5,000 | 103.1 | 34.4 | **0%** | 1,346ms | 3,779ms |

### Key Findings

1. **Zero rate limiting at all concurrency levels** — not a single 429 response
   across 9,510+ requests at 200 concurrent workers. The Songwhip app hit 429s at
   11% of requests with just 15 workers.

2. **Peak throughput: ~118 req/s at 50 workers** (~39 fans/s). Beyond 50 workers,
   throughput plateaus and latency climbs — bottleneck shifts to local machine
   connection pooling, not Spotify rate limits.

3. **SMF rate budget is at least 10x higher than Songwhip** — Songwhip's ceiling
   was ~10-12 req/s; SMF sustains 118 req/s with no 429s.

4. **At Lambda scale, the real ceiling is likely higher** — distributed execution
   across Lambda workers eliminates the local connection pool bottleneck observed
   at 100+ threads.

### Head-to-Head Comparison (15 workers)

| Metric | Songwhip | SMF |
|--------|----------|-----|
| 429 rate | 11% | **0%** |
| Successful req/s | ~14.2 | **41.8** |
| Fan throughput | ~7.5 fans/s | **13.9 fans/s** |
| Mean Retry-After | 4.8s | N/A |
| Effective overhead from retries | 14% | **0%** |

At equal concurrency (15 workers), SMF delivers **2.9x the fan throughput** with
zero retry overhead. The improvement comes entirely from eliminating rate limiting.

### Lambda Infrastructure Tests (2026-03-30)

Ran the SMF tokens through the existing collector worker Lambda pipeline to measure
throughput at distributed scale. Lambda env vars swapped to SMF credentials. Same
SQS queue, same Kafka producer, same collector worker code.

**Token validity note**: ~41% of tokens in the SMF dump are valid for this app.
The remaining ~59% fail with "Invalid client" (issued by a different Spotify app)
or "Refresh token revoked". This means each 100-fan batch yields ~41 successful
Spotify API call sets. Invalid token refreshes still consume wall clock time
(serial HTTP calls to `accounts.spotify.com`).

#### Results by Lambda Concurrency

| Concurrency | Tokens | SQS Messages | Wall Clock | 429s | Kafka msgs/s (Datadog) | Avg Lambda Duration |
|-------------|--------|-------------|------------|------|------------------------|---------------------|
| 15 | 10,000 | 100 | ~4 min | **0** | ~19 msgs/s | ~28s |
| 100 | 50,000 | 500 | ~2.5 min | **0** | — | ~28s |
| 500 | 200,000 | 2,000 | ~8 min | **0** | ~102-120 msgs/s | — |
| 1,000 | 500,000 | 5,000 | ~17 min | **0** | ~102-120 msgs/s | ~211s (p50), ~262s (p95) |

- **Total Lambda invocations at 1000 concurrency**: 4,488 (some messages recycled)
- **Kafka flushes**: 5,005 (matches dispatched messages — zero DLQ)
- **Zero 429s across all concurrency levels** — not a single rate limit response
  in 760,000 tokens processed across all runs

#### Scaling Bottleneck: Spotify Server-Side Latency (Not Rate Limits)

Per-endpoint timing instrumentation was added to the collector worker to identify
the bottleneck. A clean token set (678K pre-validated tokens, ~98% success rate)
was tested at both 15 and 1000 concurrency.

##### Per-Endpoint Latency Comparison

| Endpoint | 15 workers (avg / p95) | 1000 workers (avg / p95) | Slowdown |
|----------|----------------------|-------------------------|----------|
| `accounts.spotify.com` token refresh | 90ms / 140ms | 400ms / **3,100ms** | 4.4x avg, **22x p95** |
| `api.spotify.com` /me/top/artists | 230ms / 400ms | 490ms / **3,200ms** | 2.1x avg, 8x p95 |
| `api.spotify.com` /me/recently-played | 215ms / 380ms | 460ms / **1,350ms** | 2.1x avg, 3.5x p95 |

**All three endpoints degrade under load — not just `accounts.spotify.com`.** Spotify
is not sending 429s but is server-side queuing requests, resulting in higher latency.
This is a **soft throttle**: aggregate throughput caps at ~102-120 fans/sec regardless
of concurrent workers.

Key observations:
- Token refresh p95 hits **3.1s** at 1000 concurrency (vs 140ms at 15) — the
  single largest contributor to batch duration inflation
- API endpoints also degrade ~2x on average, with p95 spikes to 3.2s
- Clean tokens did NOT improve aggregate throughput — **102 msgs/s Kafka output
  was identical** for both dirty and clean token runs at 1000 concurrency
- Lambda avg duration: 28s (15 workers) vs 273s (1000 workers) — driven entirely
  by per-call latency inflation, not wasted work

The Kafka output rate plateaued at ~102-120 msgs/s between 500 and 1000 concurrency.
Adding more workers just makes each worker slower — the total work done per second
stays constant because Spotify's servers limit aggregate request throughput.

#### Effective Throughput (observed ceiling)

| Concurrency | Kafka msgs/s | Avg Lambda Duration | Per-fan time |
|-------------|-------------|---------------------|-------------|
| 15 | ~19 | ~28s | ~0.28s |
| 500 | ~102 | — | — |
| 1,000 (dirty tokens) | ~102 | ~211s | ~2.73s |
| 1,000 (clean tokens) | ~102 | ~273s | ~2.73s |

**~102 successful fans/sec is the observed ceiling for the SMF app** at any
concurrency level beyond 500. This is ~14x Songwhip's rate-limited throughput
of ~7.5 fans/sec.

### Production Scale Projections (70M fans)

#### Observed Ceiling

| Scenario | Fans/sec | Time for 70M fans |
|----------|----------|--------------------|
| Songwhip (15 workers, 11% 429s) | 7.5 | **108 days** |
| SMF (500+ workers, 0% 429s, observed ceiling) | **~102** | **~6.7-7.9 days** |

The SMF app's soft throttle at ~102-120 fans/sec means **70M fans would take ~7.9 days**
with a single app. Clean vs dirty tokens does not change aggregate throughput — it
only changes per-Lambda duration and efficiency.

#### Paths to Faster Collection

1. **Multiple Spotify apps in parallel** — each app has its own ~102-120 fans/sec budget.
   2 apps = ~204 fans/sec (~4 days), 4 apps = ~408 fans/sec (~2 days).
   Tokens must be partitioned by issuing app.
2. **Reduce HTTP calls per fan** — skip one endpoint (e.g. only collect top_artists),
   reducing from 3 calls to 2 per fan. Could increase throughput by ~50%.
3. **Request elevated API access** from Spotify — if SMF qualifies for a higher tier,
   the soft throttle may be lifted or raised.
4. **Accept 8-day collection window** — with `LAST_COLLECTED_AT ASC` ordering
   (stalest fans first) and weekly collection, this may be acceptable.

#### Infrastructure Constraints (secondary — not currently the bottleneck)

1. **Lambda concurrency**: 500 is sufficient (no gain beyond that). AWS default
   account limit is 1,000; can be increased.
2. **MSK/Kafka**: 6 partitions handle ~102-120 msgs/s easily. Would need scaling at
   500+ msgs/s (multi-app scenario).
3. **Snowflake sink connector**: Single task handles ~5,000 rec/s — not a concern
   until multi-app scale.