# MetaMuLate v10.0

**Music Metadata Aggregation & Reconciliation Tool**

A browser-based application that searches and reconciles artist/track metadata from multiple sources: Apple Music, Spotify, MusicBrainz, Discogs, Genius, Snowflake, Bandcamp, Wikidata, Lyrics.ovh, and custom web scraping.

---

## Hosted Deployment (Recommended)

**MetaMuLate is deployed to a development environment at:**

### https://metamulate.dev.theorchard.io/

Using the hosted version provides several advantages:

| Benefit | Description |
|---------|-------------|
| **No CORS Issues** | Server-side proxying eliminates all CORS restrictions |
| **No Local Setup** | No need to run the proxy binary locally |
| **Always Updated** | Latest version deployed automatically |
| **Shared Sessions** | Team members can share session export files |

### When to Use Local Development

The proxy code is still provided for:
- **Offline development** - Working without network access
- **Local testing** - Testing changes before deployment
- **Legacy support** - Environments that can't access the hosted version

### Snowflake Security Note

Even with the hosted deployment, the **Snowflake proxy remains the most secure method** for Single Page Applications (SPAs). The hosted version uses server-side proxying for CORS, but Snowflake credentials are still handled via the local proxy with SSO support to prevent credential exposure in the browser.

---

## What's New in v10.0

### 🌟 Persistent Golden Records System

- **Golden records now persist across ALL searches and sessions**
- Tracks confirmed and resolved are permanently stored in the `goldenRecords` array
- Golden records survive:
  - New discovery searches
  - Track switching
  - Session history dumps
  - Browser refreshes (when session is exported/imported)
- **Only cleared by:** Full system reset or targeted golden record reset
- View all golden records in dedicated modal with search and filtering

### 📚 Session History Management

- **All scraped metadata preserved across searches**
- `sessionHistory` array stores complete track data including:
  - All scraped metadata from every source
  - Discovery data and platform IDs
  - Multi-result pagination state
- Tracks with scraped data no longer lost when performing new searches
- Session history accumulates until explicitly cleared via DUMP function

### 🗄️ DUMP Session History Feature

- New "DUMP & Clear Session History" function in Data & Config menu
- **Exports all session data** (tracks + history + golden records) to JSON
- **Clears non-golden data** (tracks queue + session history)
- **Preserves golden records** - they remain intact after DUMP
- Use for: cleaning up large sessions while keeping confirmed metadata

### 📊 Session Statistics Dashboard

- Real-time stats displayed in console panel footer:
  - Total tracks in session history
  - Number of golden records
  - Total data size (KB/MB)
- "View All Golden Records (X)" button shows current count
- Automatic size tracking with formatted display

### 🎯 Enhanced Export System

**All exports now include complete golden records and session history:**

- **Session JSON Export:** Includes `goldenRecords`, `sessionHistory`, and `sessionStats`
- **Session XLSX Export:** "Golden Records" sheet now pulls from persistent `goldenRecords` array
- **Primary Export Button:** Exports all golden records (not just current queue)
- **Platform IDs included:** Apple Music ID, Spotify ID, MusicBrainz ID, Discogs ID, Wikidata ID

### 🔍 Golden Records Modal

- New modal to view and manage all confirmed tracks
- Features:
  - Sortable list of all golden records
  - Track details: name, artist, album, ISRC, resolved date
  - View/Edit button to jump to track (restores from history if needed)
  - Remove button for individual record deletion
  - Clear All button with confirmation
  - Total count and size display

### 🔄 Smart Session Restoration

- Discovery searches now preserve existing data:
  - All current tracks moved to `sessionHistory` before new search
  - Golden records automatically restored if track re-appears
  - Tracks marked as resolved when matching golden record found
- Reconcile (scrape) updates preserve all previous scrapes
- No data loss when switching between tracks

---

## What's New in v9.95!

### Dynamic Multi-Source Scraping

- **8+ data sources** queried dynamically based on your matrix configuration
- Sources include: Apple Music, Spotify, MusicBrainz, Discogs, Genius, Snowflake, Bandcamp, Wikidata
- Matrix columns automatically determine which sources are queried

### Smart Default Column Configuration

- **Conditional source inclusion**: Default matrix only includes sources that are properly configured
- Spotify appears in default matrix only when Client ID & Secret are configured
- Snowflake appears only when local proxy is running OR credentials are configured
- Sources without required configuration are automatically omitted from defaults

### Swappable Discovery Source

- **Choose Apple or Spotify** as your primary artist search source
- Configure in Connections modal with visual toggle
- The "other" source automatically becomes first second-tier source
- Matrix column order dynamically adjusts (Primary → Secondary → Others)
- Search bar placeholder updates to show active source
- Apple always available (falls back to iTunes Public API)
- Spotify requires OAuth credentials

### Flexible Column Configuration with Swapping

- **First column locked** to discovery source (configurable in Connections)
- **All other columns** can be changed to any available source
- **Column position swapping**: If you select a source already in the matrix, the two columns swap positions
- Data integrity preserved - only display order changes, scraped data remains intact

### Spotify Integration

- **OAuth 2.0 Client Credentials** flow with automatic token refresh
- Provides: ISRC, Label, P-Line, UPC, Popularity score, Duration, Content Rating, Spotify ID
- **Artist genre fallback**: When track/album genres are unavailable (Spotify deprecated these), artist profile genres are used with a visual indicator
- Tokens cached for 1 hour with automatic refresh before expiry
- CORS-aware: No proxy needed when served, auto-proxies token endpoint in file:// mode
- **Multi-result support**: Up to 10 search results, 5 with full album enrichment

### Multi-Result Pagination

Sources with configurable result limits (Search / Detail):

| Source | Search Limit | Detail Limit | Background Loading |
| ------ | ------------ | ------------ | ------------------ |
| MusicBrainz | 10 | 5 | ✓ |
| Discogs | 5 | 3 | ✓ |
| Spotify | 10 | 5 | ✓ |
| Wikidata | 5 | 3 | ✓ |
| Genius | 5 | 3 | ✓ |

### Enhanced Matrix View

- **Resizable columns** (250px - 600px) with drag handles
- **Paginated results** - navigate multiple matches per source with colorful pill-shaped controls
- **Scroll position preserved** when navigating between results
- **Release-level fields** now displayed: Catalog #, UPC/Barcode, Country, Format, Styles

### Comprehensive Export/Import System

Three distinct data management categories:

| Category | Export | Import | Contains |
| -------- | ------ | ------ | -------- |
| **Session Data** | JSON + Excel (multi-sheet) | JSON | Tracks, golden records, scraped metadata |
| **Configuration** | JSON | JSON | Matrix settings, timeouts, preferences (no keys) |
| **API Keys** | JSON | JSON | Sensitive credentials only |

**Golden Records Export**: All **confirmed and resolved** tracks are exported from the persistent `goldenRecords` array. Golden records persist across all searches and sessions.

**Session History**: All scraped metadata is preserved in `sessionHistory` and exported with session data.

**DUMP Function**: Export and clear session history while preserving golden records. Use the "DUMP & Clear Session History" button to clean up large sessions.

**Platform IDs Captured**: Exports now include Apple Music ID, Spotify ID, MusicBrainz ID, Discogs ID, and Wikidata ID for each track.

### Interactive Onboarding Tour

- **7-step guided tour** with demo data (works offline)
- **Keyboard navigation**: Arrow keys, Enter, Escape
- **Previous/Next buttons** with progress indicator
- **Demo data** automatically loaded for demonstration
- **State preservation** - previous session restored after tour

### Web Scraping
- Extract metadata from **any URL** or **pasted text**
- AI-powered field detection and mapping
- Add as additional column in matrix

---

## Local Development (Legacy/Offline)

**For local development without the hosted environment:**

### macOS / Linux
```bash
./start.sh
```

### Windows
Double-click `start.bat` or run from command prompt:
```cmd
start.bat
```

**That's it!** The start script handles everything automatically:

1. Detects your operating system and selects the correct proxy binary
2. Creates `proxy/proxy-config.json` from the example if it doesn't exist
3. Starts the CORS proxy server on port 8080
4. Opens MetaMuLate in your default browser
5. Keeps the proxy running until you press `Ctrl+C`

> **Note:** If you just open the HTML file directly without running the start script, API calls to Discogs, Genius, and Snowflake will fail due to CORS restrictions.

---

## Configuration Files

MetaMuLate uses proxy configuration stored in `proxy/proxy-config.json`.

### Full Example Configuration

```json
{
  "log_level": 3,
  "cors_enabled": true,
  "cors_port": 8080,
  "snowflake_enabled": false,
  "snowflake_port": 8081,
  "snowflake_account": "your-account.us-east-1",
  "snowflake_user": "your_username",
  "snowflake_password": "your_password",
  "snowflake_database": "YOUR_DATABASE",
  "snowflake_schema": "PUBLIC",
  "snowflake_warehouse": "COMPUTE_WH",
  "snowflake_role": "YOUR_ROLE",
  "snowflake_external_browser": false,
  "external_browser_timeout": 120,
  "query_timeout": 60,
  "login_timeout": 30
}
```

### Snowflake Authentication Methods

1. **Password Authentication** (simplest)
2. **Browser SSO Authentication** (recommended for corporate accounts)
3. **JWT Key-Pair Authentication** (most secure)
4. **OAuth Token Authentication**

### Sony Music / The Orchard Configuration Example

```json
{
  "log_level": 3,
  "cors_enabled": true,
  "cors_port": 8080,
  "snowflake_enabled": true,
  "snowflake_port": 8081,
  "snowflake_account": "SME-ORCHARD",
  "snowflake_user": "YOUR_EMAIL@SONYMUSIC-PDE.COM",
  "snowflake_external_browser": true,
  "external_browser_timeout": 120,
  "snowflake_database": "ORCHARD_APP_REPORTING_V2",
  "snowflake_schema": "ART_RELATIONS_PROD_ART_RELATIONS",
  "snowflake_warehouse": "DEV_OWS_WAREHOUSE",
  "snowflake_role": "DEV_ENGINEERING",
  "query_timeout": 60,
  "login_timeout": 30
}
```

---

## First-Time Setup

### 1. Configure API Keys (Optional but Recommended)

Open MetaMuLate, click **Connections** in the header, and add your API tokens:

| Service | Required? | How to Get |
|---------|-----------|------------|
| Apple Music | Optional | Apple Developer Portal |
| Spotify | Recommended | [Spotify Developer Dashboard](https://developer.spotify.com/dashboard) - Create app, get Client ID & Secret |
| Discogs | Recommended | Discogs Developer Settings |
| Genius | Recommended | Genius API Clients |
| Snowflake | Optional | Your organization's credentials |

### 2. Take the Interactive Tour

First-time users see a **Welcome Modal** with options to:
- **Start Interactive Tour** - 7-step guided walkthrough with demo data
- **Skip Tour** - Jump straight to the application

The tour can be restarted anytime via the **Tour** button in the header.

---

## Folder Structure

```
MetaMuLate/
├── start.sh                    # START HERE (macOS/Linux)
├── start.bat                   # START HERE (Windows)
├── metamulate_v9.95.html       # Main application
├── README.md                   # This file
├── proxy/                      # Proxy server files (legacy/local dev)
│   ├── proxy-config.json       # YOUR PROXY CONFIGURATION
│   ├── bin/                    # Pre-compiled binaries
│   └── code/                   # Go source code
├── logs/                       # Exported session files
└── docs/                       # Technical documentation
```

---

## How to Use

### Basic Workflow

1. **Search** - Enter an artist name in the search box
2. **Select Track** - Click any track to trigger multi-source scraping
3. **Review Matrix** - Compare metadata from all sources side-by-side
4. **Navigate Results** - Use pagination controls for multiple matches
5. **Build Golden Record** - Click cells to select the best value
6. **Mark Resolved** - Confirm the track is complete
7. **Export** - Download resolved tracks as XLSX or JSON

### Data Sources

| Source | Provides | Needs Proxy? | Multi-Result |
|--------|----------|--------------|--------------|
| Apple Music/iTunes | Track, Artist, Album, ISRC, Apple Music ID | No | Yes (5)* |
| Spotify | ISRC, Label, P-Line, UPC, Popularity, Genre†, Spotify ID | Auto‡ | Yes (5)* |
| MusicBrainz | ISRC, Label, Release info, Country | No | Yes (5) |
| Discogs | Label, P-Line, Year, Genre, Styles, UPC | Yes | Yes (3) |
| Genius | Composer, Writer credits | Yes | Yes (3) |
| Snowflake | Custom database queries | Yes | No |
| Bandcamp | Tags, Description, Credits | Yes | No |
| Wikidata | ISRC, ISWC, External IDs, MusicBrainz/Discogs IDs | No | Yes (3) |
| Lyrics.ovh | Lyrics snippet | No | No |

*Apple and Spotify can serve as both discovery AND reconciliation sources. 25 tracks per discovery search, 5 detailed results for reconciliation.  
†Genre from artist profile (track/album genres deprecated by Spotify)  
‡Spotify API supports CORS when served; uses proxy automatically in file:// mode

### Matrix Fields

The matrix displays both **track-level** and **release-level** metadata:

| Track-Level | Release-Level | Extended |
|-------------|---------------|----------|
| Track Name | Catalog # | Genre |
| ISRC | UPC/Barcode | Styles |
| Artist | Country | Composer |
| Album | Year | Lyrics |
| Track # | Format | |
| Duration | | |
| P-Line | | |
| Label | | |

---

## Understanding Scores

### Track Display Indicators

| Indicator | Meaning |
|-----------|---------|
| **ISRC Badge** | Official recording identifier present |
| **Score** | Authority score (0-150+) |
| **XX%** | Metadata completeness percentage |
| **(YYYY)** | Release year |

### Score Calculation

| Factor | Points |
|--------|--------|
| ISRC present | +50 points |
| Each filled field | +2 points |
| Pre-2000 release | +30 points |
| P-Line present | +10 points |
| Label present | +10 points |
| Composer present | +5 points |
| MusicBrainz score | +native score |

### Score Color Key

| Color | Range | Quality |
|-------|-------|---------|
| **Green** | 80+ | High authority, complete metadata |
| **Blue** | 50-79 | Medium authority, good metadata |
| **Amber** | <50 | Lower authority, incomplete metadata |

---

## Data Management

### Export Options

Access via **Data & Config** menu in the header:

#### Session Data (Scraped Metadata)
| Format | Use Case |
|--------|----------|
| **JSON** | Re-import later, backup, share with team |
| **Excel (XLSX)** | Analysis, reporting, multi-sheet format |

The Excel export creates multiple sheets:
- **Golden Records** - Resolved/confirmed tracks only (matches dedicated export button)
- **All Tracks** - Complete discovery data with all fields and platform IDs
- **Apple Music, Spotify, MusicBrainz, Discogs, Genius, etc.** - Per-source scraped data with URLs
- **Raw Data Dump** - Normalized format with all fields from all sources

**Note**: Only tracks marked as "Resolved" appear in the Golden Records sheet. Unresolved tracks are still exported in the All Tracks and per-source sheets.

#### Configuration (No Keys)
- Matrix column settings
- Timeout values
- Custom fields
- Debug mode preference

#### API Keys (Sensitive)
- Contains credentials in plain text
- Keep secure, don't share publicly

### Import Options

| Type | Restores |
|------|----------|
| Session JSON | All tracks and golden records |
| Config JSON | Settings without API keys |
| Keys JSON | API credentials only |

---

## Interactive Tour

### Starting the Tour

- **First run**: Welcome modal appears automatically
- **Restart**: Click **Tour** button in header

### Tour Controls

| Action | Method |
|--------|--------|
| Next step | Click **Next**, press Right Arrow or Enter |
| Previous step | Click Back button or press Left Arrow |
| End tour | Click X, press Escape |

### Tour Steps

1. **Discovery Engine** - Search for artists
2. **Track List & Selection** - View results with quality scores
3. **Source Columns** - Resize columns, pagination controls
4. **Reconciliation Matrix** - Compare data field by field
5. **Golden Record** - Build your curated metadata
6. **Quick Export** - Export resolved tracks
7. **Connections & Settings** - Configure API keys and timeouts

### Demo Data

The tour loads fictional demo data that works **offline**:
- 3 demo tracks with complete metadata
- Simulated MusicBrainz, Discogs, Genius, and Lyrics data
- Pre-built golden record example

After the tour, your previous session is automatically restored.

---

## Debug Mode

Debug Mode provides detailed logging for troubleshooting API connection issues.

**Enable in:** Console Panel (bottom of screen) - Toggle "Debug Mode"

### Debug Mode Shows:
- Full API request URLs
- Request headers (tokens redacted for security)
- Request bodies
- Response status codes
- Complete response payloads

---

## CORS Proxy Options

MetaMuLate needs a CORS proxy to access APIs that don't allow direct browser requests.

### Option 1: Local Proxy (Recommended)

- Fastest and most reliable
- Included in the start scripts
- No external dependencies
- URL: `http://localhost:8080/?url=`

### Option 2: Public Fallbacks (Automatic)

If the local proxy isn't running, MetaMuLate can fall back to public CORS proxies.

### Option 3: Self-Hosted (Teams)

For team deployments, you can host your own CORS proxy.

---

## Security Considerations

### API Tokens
- Stored locally in your browser only
- Never transmitted except to their respective APIs
- Export Keys files contain secrets - keep secure

### Snowflake Connection Methods

| Method | Security | Best For |
|--------|----------|----------|
| **Local Proxy (SSO)** | Most Secure | Production - credentials never in browser |
| **Direct SQL API** | Credentials in browser | Hosted environments only (https://) |
| **Auto-Detect** | Varies | Let MetaMuLate choose automatically |

### Best Practices

1. Use **Local Proxy with SSO** for production and enterprise use
2. Use **Direct** only for hosted internal deployments
3. Don't share Export Keys files - they contain API credentials
4. Use private/incognito mode on shared computers
5. Clear browser data when done on shared machines

---

## Data Lifecycle & Reset Options

### Golden Records Persistence

Golden records are designed to persist through normal operations:

- ✅ **Survives:** New searches, track switching, session dumps, browser refreshes (with export/import)
- ❌ **Cleared by:** Full system reset or targeted golden record reset only

### Reset Functions

| Function | Location | What's Cleared | What's Preserved |
|----------|----------|----------------|------------------|
| **DUMP Session History** | Data & Config menu | Current queue + session history | Golden records + API keys |
| **Clear All Golden Records** | Golden Records modal | All golden records only | Session history + current queue |
| **Factory Reset** | Settings | Everything (keys, config, data) | Nothing - complete wipe |

### Data Management Workflow

**Recommended workflow for large projects:**

1. Search and scrape tracks throughout the day
2. Confirm and resolve important tracks → they become golden records
3. Periodically run **DUMP Session History** to clear accumulated data
4. Golden records remain for final export
5. Export golden records to Excel when project complete

---

## Tips & Tricks

- **Sort smartly:** Results auto-sort by authority score - highest quality first
- **Verify sources:** Click external link icons to check source data directly
- **Monitor size:** Watch session stats in console footer to track data growth
- **Regular dumps:** Use DUMP to keep sessions manageable (recommended every 50-100 tracks)
- **Golden record first:** Always resolve tracks as you go - don't wait until the end
- **Track progress:** Use "Mark Resolved" to track completed tracks
- **Save often:** Export session frequently to avoid losing work
- **Debug issues:** Enable Debug Mode when troubleshooting
- **Check completeness:** The % indicator shows how complete the metadata is
- **Trust ISRC:** Tracks with ISRC badges have verified recording identifiers
- **Prefer originals:** Pre-2000 releases are usually originals, not reissues
- **Use keyboard:** Navigate tour with arrow keys, exit with Escape
- **Resize columns:** Drag column edges to make data more readable

---

## Troubleshooting

### "Proxy not running" error

Make sure you started with `./start.sh` (not just opening the HTML file).

### CORS errors in browser console

1. Verify proxy is running (check terminal)
2. Check proxy URL in Connections matches `http://localhost:8080/?url=`
3. Enable proxy toggle for the failing API

### API timeouts

Increase the timeout in Connections modal for that specific API.

### Snowflake SSO not working

1. Ensure `snowflake_external_browser` is set to `true`
2. Increase `external_browser_timeout` if login takes longer
3. Check that your browser allows popups from localhost

### Tour not showing demo data

1. Clear browser localStorage: `localStorage.clear()`
2. Refresh the page
3. Click "Tour" button to restart

### Matrix columns not resizing

- Minimum width: 250px
- Maximum width: 600px
- Drag the right edge of column headers

### Building from source (if binaries don't work)

Requires Go 1.21+:

```bash
cd proxy/code
go build -ldflags="-s -w" -o ../bin/metamulate-proxy metamulate-proxy.go
```

---

## Version History

| Version | Changes |
|---------|---------|
| **v9.95!** | Smart default columns (conditional on config), column position swapping, resolved-only Golden Records export, platform ID capture (Apple/Spotify/MusicBrainz/Discogs/Wikidata), locked discovery column with configurable other columns, comprehensive Raw Data Dump export sheet, swappable discovery source (Apple ↔ Spotify), Spotify multi-result with background loading, Wikidata multi-result support, dynamic matrix column ordering, Spotify OAuth 2.0 with artist genre fallback, hosted deployment at metamulate.dev.theorchard.io, configurable search/detail limits per source |
| v9.0 | Dynamic multi-source scraping, resizable matrix columns (250-600px), paginated results with colorful controls, release-level fields (Catalog #, UPC, Country, Format), comprehensive export/import (Session JSON/XLSX, Config, Keys), interactive 7-step tour with demo data, keyboard navigation, Bandcamp & Wikidata sources |
| v8.1 | Multi-method Snowflake connections (Auto, Proxy, Direct, CORS), improved connection UI, hosted environment support |
| v8.0 | Unified proxy, per-API timeouts/proxy toggles, improved UI |
| v6.2 | Snowflake integration, enhanced reconciliation |
| v6.0 | Matrix comparison view, golden record builder |

---

## License

Internal tool - Sony Music / The Orchard
Copyright 2026 Sony Music Entertainment
For internal use only. Do not distribute.