# Heavy Rotation Streamlit App

A Snowflake Streamlit app for the Heavy Rotation POC that analyzes Spotify listening behavior. Processes multiple fans from DynamoDB presave data and fetches their top artists from the Spotify API.

## Project Status

**Phase 1: API Integration POC** - ✅ **COMPLETE** (November 2025)

Phase 1 delivers a functional POC with:
- DynamoDB integration for production tokens
- Spotify API integration with automatic token refresh
- **Wave-based parallel processing** for up to 2,000 fans (~2.3x speedup)
- Progress tracking and error handling
- CSV/JSON export functionality
- Modern Spotify-inspired UI with dark theme (config.toml theming)

**Phase 2: Database Integration & Production Features** - 🚧 **IN PROGRESS**

Track current tasks and progress on the [Notion Board](https://www.notion.so/2fe87274f6d74b4ca587fc45e9be2398).

See [plan.md](plan.md) for Phase 1 architecture reference and implementation patterns.

---

## Features

### Core Functionality
- 📊 **Presave Campaign Integration** - Select from active Songwhip presave campaigns
- 🎯 **Artist Matching** - Identify fans with specific artists in their Heavy Rotation
- 📦 **Batch Processing** - Analyze up to 2,000 fans per run from DynamoDB presave data
- ⚡ **Wave-Based Parallel Processing** - ~2.3x faster with configurable concurrency
- ⏱️ **Time Range Selection** - Analyze listening behavior over 4 weeks, 6 months, or 1 year
- 🔄 **Automatic Token Refresh** - Seamless OAuth token management with rate limit handling
- 📈 **Real-time Progress** - Live progress tracking with wave-by-wave updates
- 💾 **Data Export** - Export results to CSV or JSON

### User Interface
- 🎨 **Modern Dark Theme** - Spotify-inspired design with signature green accents
- 📱 **Responsive Layout** - Card-based design with hover effects and smooth animations
- 🎯 **Clear Workflow** - Numbered steps guide users through the analysis process
- 📊 **Rich Metrics Display** - Custom styled metrics cards with visual indicators
- 🔒 **Secure Credentials** - Uses Snowflake Secrets for API keys and AWS credentials

## Setup

1. **Install dependencies:**
   ```bash
   pip install -r requirements.txt
   ```

2. **Configure Spotify credentials using Snowflake Secrets:**

   The application retrieves Spotify API credentials from Snowflake Secrets for secure credential management.

   - Get your Spotify API credentials:
     - Go to https://developer.spotify.com/dashboard
     - Create an app
     - Copy your Client ID and Client Secret

   - Store credentials as Snowflake Secrets (requires ACCOUNTADMIN role):
     ```sql
     -- Create secrets in your schema
     CREATE SECRET <SCHEMA>.spotify_client_id
       TYPE = GENERIC_STRING
       SECRET_STRING = '<your_client_id>'
       COMMENT = 'Spotify API Client ID for Heavy Rotation app';

     CREATE SECRET <SCHEMA>.spotify_client_secret
       TYPE = GENERIC_STRING
       SECRET_STRING = '<your_client_secret>'
       COMMENT = 'Spotify API Client Secret for Heavy Rotation app';

     -- Grant access to your role
     GRANT READ ON SECRET <SCHEMA>.spotify_client_id TO ROLE <YOUR_ROLE>;
     GRANT READ ON SECRET <SCHEMA>.spotify_client_secret TO ROLE <YOUR_ROLE>;
     ```

   See [CLAUDE.md](../CLAUDE.md#snowflake-secrets-management) for more details on secrets management.

3. **Configure AWS credentials (for DynamoDB batch processing):**

   Create a `secrets.toml` file in the `streamlit-test/` directory with your AWS credentials:

   ```bash
   # Get AWS credentials using awsume
   awsume prod              # Assume prod profile with MFA
   awsume songwhip -o default   # Assume songwhip role

   # Create secrets.toml file with your credentials
   cat > secrets.toml << 'EOF'
   AWS_ACCESS_KEY_ID = "YOUR_ACCESS_KEY"
   AWS_SECRET_ACCESS_KEY = "YOUR_SECRET_KEY"
   AWS_SESSION_TOKEN = "YOUR_SESSION_TOKEN"
   AWS_REGION = "us-east-1"
   EOF

   # Get your actual credential values from AWS CLI
   echo "AWS_ACCESS_KEY_ID = \"$(aws configure get aws_access_key_id)\""
   echo "AWS_SECRET_ACCESS_KEY = \"$(aws configure get aws_secret_access_key)\""
   echo "AWS_SESSION_TOKEN = \"$(aws configure get aws_session_token)\""
   echo "AWS_REGION = \"us-east-1\""
   ```

   **Manual creation:** Alternatively, create `secrets.toml` manually:
   ```toml
   # AWS credentials for DynamoDB access
   AWS_ACCESS_KEY_ID = "AKIA..."
   AWS_SECRET_ACCESS_KEY = "wJalr..."
   AWS_SESSION_TOKEN = "FwoGZ..."  # Required for temporary credentials
   AWS_REGION = "us-east-1"
   ```

   **Note:** `secrets.toml` is in `.gitignore` and will not be committed.

   See [CLAUDE.md](../CLAUDE.md#aws-integration-dynamo-test) for details on AWS access.

## Running in Snowflake

This application runs in Snowflake's Streamlit Container Runtime environment.

### Prerequisites

1. **secrets.toml file** with Spotify credentials (see [CLAUDE.md](../CLAUDE.md#using-secrets-in-streamlit-container-runtime) for configuration details)
2. **External Access Integration** attached to your Streamlit app

### Deployment Steps

1. Create a Streamlit app in Snowflake (via UI or Git integration)
2. Upload your code including `secrets.toml` file
3. **Attach External Access Integration** (CRITICAL):
   ```sql
   -- Find your app name
   SHOW STREAMLITS IN SCHEMA FANSIFTER_APP_REPORTING.DEV_MMACHADO;

   -- Attach the integration
   ALTER STREAMLIT FANSIFTER_APP_REPORTING.DEV_MMACHADO.<YOUR_APP_NAME>
   SET EXTERNAL_ACCESS_INTEGRATIONS = (HEAVY_ROTATION_EXTERNAL_ACCESS);
   ```
4. Restart the app (refresh browser)

### Troubleshooting Network Errors

**Error:** `Failed to resolve 'accounts.spotify.com'`

**Cause:** External access integration not attached

**Fix:** Run the `ALTER STREAMLIT ... SET EXTERNAL_ACCESS_INTEGRATIONS` command above

See [CLAUDE.md](../CLAUDE.md#external-access-integration-for-container-runtime) for detailed external access configuration.

## Usage

### Step-by-Step Workflow

1. **Select Presave Campaign** (Step 1️⃣)
   - Choose from the dropdown of active presave campaigns
   - Refresh list if needed to see latest campaigns
   - Artist name and Songwhip URL are auto-populated

2. **Configure Artist Matching** (Step 2️⃣)
   - Artist is auto-filled from presave campaign
   - Or enter a Spotify artist URL or ID manually
   - App validates and displays artist details with image

3. **Set Batch Parameters** (Step 3️⃣)
   - Choose number of fans to analyze (up to 2,000)
   - Select time range: 4 weeks, 6 months, or 1 year
   - Configure advanced settings (wave size, concurrency, delay)
   - Click "🚀 Start Analysis"

4. **Monitor Progress**
   - Real-time progress bar shows processing status
   - See wave-by-wave updates with fan count and completion percentage
   - Processing happens in main content area with parallel execution

5. **Review Results**
   - Summary metrics: total fans, heavy rotation matches, success rate
   - Detailed table with fan names, emails, artist rankings
   - Error categorization for failed processing
   - ✅ Success: Top artists fetched successfully
   - ❌ Error: Token issues or API errors

6. **Export Results**
   - Download results as CSV or JSON
   - Includes all fan data, artist rankings, and analysis outcomes

## Architecture

Following the patterns from `api-test/` and `dynamo-test/`:

### Spotify Integration
- **lib/spotify_auth.py** - OAuth token refresh using client credentials
- **lib/spotify_client.py** - HTTP client with automatic token refresh on 401
  - `SpotifyRateLimitError` - Custom exception for 429 errors with `retry_after` attribute
- **lib/heavy_rotation_fetcher.py** - Fetch top artists from Spotify API

### Wave Processing
- **lib/wave_processor.py** - Parallel batch processing using `ThreadPoolExecutor`
  - `WaveConfig` - Configuration (wave_size, concurrency, inter_wave_delay_ms)
  - `WaveResult` - Results with timing and error tracking
  - `WaveProcessor` - Processes fans in waves with configurable parallelism
  - Rate limit (429) handling with automatic retry using `Retry-After` header

### DynamoDB Integration
- **lib/dynamodb_client.py** - AWS DynamoDB client for fetching presave tokens
  - `load_aws_credentials()` - Load credentials from secrets.toml
  - `create_dynamodb_client()` - Initialize boto3 client with retry logic
  - `fetch_refresh_tokens()` - Query production presave tasks table

### Theming
- **.streamlit/config.toml** - Spotify-inspired dark theme configuration
  - Native Streamlit theming (no CSS hacks)
  - Works correctly in Snowflake Container Runtime

### Application
- **heavy_rotation_app.py** - Streamlit UI with wave-based batch processing

## Project Tracking

Development tasks are tracked in Notion: https://www.notion.so/2fe87274f6d74b4ca587fc45e9be2398

## Completed Features (Phase 1)

- ✅ DynamoDB integration for production tokens
- ✅ Multi-user batch processing support (up to 2,000 fans)
- ✅ Wave-based parallel processing (~2.3x speedup)
- ✅ Configurable concurrency settings (wave size, workers, delay)
- ✅ Error handling and progress tracking with wave-level reporting
- ✅ Rate limit (429) handling with automatic retry
- ✅ AWS credentials management with temporary tokens
- ✅ Export functionality for batch results (CSV/JSON)
- ✅ Spotify OAuth token refresh
- ✅ External Access Integration for secure API calls
- ✅ Native config.toml theming (Spotify-inspired dark theme)

## Current Limitations

- ⚠️ In-memory data storage (session state) - lost on page refresh
- ⚠️ No job history or historical data viewing
- ⚠️ No automatic artist matching (manual inspection required)
- ⚠️ Limited error analysis capabilities

## Next Steps (Phase 2)

See [next_steps.md](next_steps.md) for complete roadmap. Immediate priorities:

1. **Phase 2A: Database Schema** (2 hours)
   - Create SQL setup scripts for Snowflake tables
   - POC_JOB_STATUS, POC_RAW_LISTENING_DATA, POC_AGG_HEAVY_LISTENERS, POC_ERROR_LOG

2. **Phase 2B: Artist Matching** (3 hours)
   - Add artist ID input
   - Implement automatic matching logic
   - Track artist rank (1-50)

3. **Phase 2C: Data Persistence** (4 hours)
   - Replace session state with Snowflake database
   - Store all data in VARIANT columns
   - Job tracking and status updates

4. **Phase 2D: Job History UI** (3 hours)
   - View past analysis runs
   - Resume failed jobs
   - Export from history

5. **Phase 2E: Deployment Docs** (2 hours)
   - Complete deployment guide
   - Dependencies documentation
   - Production setup guide

**Future Phases:**
- Phase 3: Production readiness (unit tests, monitoring, performance optimization)
- Phase 4: Scale to production architecture (Python UDFs, Serverless Tasks)