# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

# Songwhip API Architecture Guide

## Overview

Songwhip API is the backend service that powers the Songwhip music discovery platform. It provides a RESTful API interface for managing music releases, artist profiles, album pages, and user accounts. The service integrates with The Orchard's infrastructure and external services like Kafka, AWS (S3, SQS, KMS), and various monitoring/analytics tools.

**Tech Stack:**

- Node.js 16+ with TypeScript
- Express.js 5.x for HTTP API
- PostgreSQL (AWS RDS) for persistent storage
- Knex.js for database migrations
- Zod for schema validation
- JWT-based authentication

## Application Startup Flow

### Entry Point: `index.ts`

1. Initializes Datadog APT tracing (`dd-trace/init`)
2. Imports Express app configuration from `app/index.ts`
3. Reads environment variables: `PORT`, `DEBUG`, `SONGWHIP_ENV`, `NODE_ENV`
4. Starts HTTP server with: `app.listen(PORT, ...)`

### App Initialization: `app/index.ts`

Middleware chain (in order):

1. **Trust Proxy**: Sets `trust proxy = true` for X-Forwarded-\* headers behind load balancers
2. **Disable X-Powered-By Header**: Removes `X-Powered-By: Express` for security
3. **Sentry Middleware**: Error tracking and performance monitoring (must be first)
4. **Trace Context Middleware**: Captures trace parameters for debug logging
5. **Request Context Middleware**: Stores request metadata (user, IP, user agent, referer)
6. **Request Logger Middleware**: Console logging for development
7. **Sub-Request Mock Middleware**: For Cypress testing in test environment only
8. **CORS Middleware**: Cross-origin request handling
9. **Router**: Routes requests to v3 API endpoints
10. **Error Handler Middleware**: Centralized error handling (must be last)

Routes are mounted at:

- `/v3/*` - Primary API routes
- `/` - Default routes (legacy)
- `/health` - Health checks

## Directory Structure

### `/app/` - Core Application Logic

#### `/app/database/` - Data Persistence Layer

- **Schema & Models**: TypeScript types and database table definitions
  - `users/`, `albums/`, `tracks/`, `artists/`, `accounts/`, `customPages/`
  - Each domain has types, queries, and mutations
- **Migrations**: Knex.js migration scripts
  - `/migrations/schema/` - Database schema changes (CREATE TABLE, ALTER COLUMN, etc.)
  - `/migrations/data/` - Data updates and transformations
  - `/migrations/scripts/` - Helper scripts for creating/running migrations
- **Utilities**: Database helper functions and transaction management
  - `knex.ts` - Knex instance initialization with error handling
  - `knexfile.ts` - Connection configuration (PostgreSQL RDS)
  - `transaction.ts` - Transaction helpers for multi-step operations
- **Types**: `sharedTypes.ts` contains domain enums and interfaces (AddonTypes, ItemTypes, OrchardBrands, ItemConfig, etc.)

#### `/app/entities/` - Domain Models (Active Record-like Pattern)

Each entity (Album, Artist, Track, User, Account) is a domain model with:

- Database query methods (`getById()`, `getAll()`)
- Instance methods for mutations (`patch()`, `delete()`, `serialize()`)
- Business logic (state transitions, validations)
- Related sub-operations (localize, upgrade, refresh, etc.)

Key entities:

- **Album**: Manages music release pages (prerelease → live upgrade flow)
- **Artist**: Manages artist profiles linked to The Orchard
- **Track**: Individual songs
- **User**: Account holders with roles and scopes
- **Account**: Business accounts (labels, distributors)
- **CustomPage**: User-created custom landing pages

#### `/app/lib/` - Shared Utilities & Services

**Authentication & Authorization:**

- `auth/accessToken.ts` - JWT encoding/decoding with Zod validation
- `auth/constants.ts` - Access scopes (permissions) definitions
- `token/index.ts` - Token creation/parsing (user tokens, M2M tokens)

**Error Handling:**

- `ApiError/` - Centralized error class
  - Supports error codes, HTTP status codes, custom data
  - Serialization for API responses
  - Error reporting control (by status level)

**Logging & Monitoring:**

- `logger/` - Pino-based logging with debug namespace support
- `reporter/` - Multi-transport error/event reporting
  - Sentry transport (error tracking)
  - Datadog transport (informational logging)

**External Services:**

- `request/context.ts` - AsyncLocalStorage for request-scoped context (userId, requestId, IP, etc.)
- `email/` - Email template generation
- `config/aws.ts` - AWS configuration
- `events/` - Songwhip event publishing
- Various utility helpers: `getPublicEndpoint`, `getGitCommit`, `getSongwhipEnv`, etc.

#### `/app/services/` - Microservice Integrations

**Kafka** (`services/kafka/`)

- Asynchronous event publishing
- Used for notifying other services of state changes
- Configured with service name "songwhip-api"

**AWS Services** (`services/aws/`)

- **S3**: File storage (upload bucket configured via `AWS_UPLOAD_BUCKET`)
- **SQS**: Async task queue for high-load operations (enqueueMessage pattern)
- **KMS**: Key management for encryption
- **Fargate**: Task execution coordination

**External APIs:**

- **Songwhip Email Service**: Email sending (staging vs production endpoints)
- **Songwhip Lookup Service**: Music metadata resolution
- **Songwhip Analytics**: Album page analytics
- **Cloudflare**: DNS/CDN integration for custom domains
- **Orchard GraphQL API**: Music catalog data (artists, labels, releases)

**Feature Flags** (`services/split/`)

- Split.io integration for feature flag management
- Features kept in memory with background polling
- Available flags: embedded_videos, custom_domain, affiliate_tokens, themes, presaves, etc.

#### `/app/middlewares/` - Express Middleware

**Request Handling:**

- `requestHandler/` - Async route handler wrappers
  - `privateRoute()` - Requires authentication + optional scopes
  - `publicRoute()` - No authentication required
  - `adminRoute()` - Admin-only access
  - `webhookRoute()` - Webhook request validation
  - Built-in Zod validation for request/response bodies

**Authentication:**

- `authorize/` - Scope-based authorization middleware
- `requestContext/` - Request metadata storage

**Cross-Cutting Concerns:**

- `errorHandler/` - Centralized error handling with Sentry/Datadog reporting
- `sentry/` - Sentry SDK initialization
- `traceContext/` - AWS X-Ray and Datadog trace ID capture
- `cors/` - CORS policy configuration
- `requestLogger/` - Development request logging

#### `/app/routes/v3/` - API Endpoints

Organized by resource domain (RESTful):

- `/accounts` - User account management
- `/albums` - Album CRUD and operations (prerelease creation, upgrade, config)
- `/artists` - Artist profiles
- `/tracks` - Track management
- `/users` - User accounts and authentication
- `/custom-pages` - Custom landing page builder
- `/orchard` - Integration with The Orchard backend
- `/resolve` - URL/metadata resolution
- `/lookup` - Music lookup service
- `/auth` - Authentication endpoints
- `/appreciation-engine` - Fan engagement features
- `/cache` - Cache management (admin only)
- `/prereleases` - Prerelease workflow
- `/resources` - Resource utilities

Each route module follows the same pattern:

1. Define Zod schemas for request/response validation
2. Implement route handlers using `privateRoute()` or `publicRoute()`
3. Call business logic from `app/logic/`
4. Return serialized response

**Example Flow** (Albums):

```
GET /v3/albums/:albumId
  → privateRoute middleware extracts user
  → getAlbum() queries database
  → Check authorization (assertUserCanWriteAlbum)
  → album.serialize() returns response
```

#### `/app/logic/` - Business Logic & Use Cases

Implements domain-specific operations:

- `albums/` - Album creation, upgrade, validation
- `users/` - User authentication, loading
- `artists/` - Artist operations
- `tracks/` - Track operations
- `customPages/` - Custom page logic
- `cache/` - Cache invalidation patterns
- `resolve/` - URL/metadata resolution
- `notifications/` - Email/notification sending
- `appreciation-engine/` - Engagement features
- `paths/` - URL path generation and validation

### `/test/` - Testing Infrastructure

**Unit Tests** (`test/unit/`)

- Vitest configuration
- Tests in `tests/` directory matching `*.test.ts` pattern
- Mock setup files: `setup.ts`, `setupAfterEnv.ts`
- Path aliases configured: `~/*` maps to root

**E2E Tests** (`test/e2e/`)

- Supertest for HTTP assertions
- Database mocking utilities
- GraphQL query mocking (fetchMock)
- Test setup in `setupAfterEnv.ts`

**Test Objects** (`test/objects/`)

- Factory functions for creating test data
- Mock entities for various domains

### `/http/` - API Documentation

Bruno API client collections for manual testing:

- `.bru` files documenting endpoints
- Not actual code, just documentation

### `/packages/` - Shared NPM Package

`@theorchard/songwhip-api` - Published to GitHub Packages

- Exports TypeScript types for client libraries
- Version in `packages/songwhip-api/package.json`
- Published via CI workflow

### `/scheduler/` - Scheduled Jobs

CRON-triggered endpoints:

- `upgrade_pending_prereleases` - Daily album upgrade
- `emails_for_incomplete_presaves` - Notification emails
- `create_test_prereleases` - Test data generation

### `/lambda/` - AWS Lambda Functions

Serverless handlers for specific operations

## Key Architectural Patterns

### 1. Request Handler Pattern

All route handlers use a wrapper pattern for consistency:

```typescript
// Async handler with error catching
asyncRequestHandler<InputSchema, OutputSchema>(
  async ({ data, tokenData, req, res, next }) => {
    // Business logic here
    return result;
  },
  {
    inputSchema: SomeSchema, // Zod validation
    outputSchema: SomeResponseSchema,
    authenticationRequired: true,
  }
);
```

Benefits:

- Automatic error handling (passes to errorHandler middleware)
- Automatic response serialization
- Optional request/response validation via Zod
- Consistent authentication flow

### 2. Entity/Domain Model Pattern

Each major domain (Album, Artist, etc.) is implemented as:

```typescript
class Album {
  static async getById(id: number): Promise<Album>;
  static async getAll(): Promise<Album[]>;

  async patch(updates): Promise<void>;
  async delete(): Promise<void>;
  serialize(): SerializedAlbum;
}
```

Benefits:

- Encapsulates business logic with data
- Single source of truth for entity behavior
- Chainable operations

### 3. Middleware-Based Request Processing

1. **Sentry** - Error tracking
2. **Trace Context** - Trace ID capture
3. **Request Context** - AsyncLocalStorage for request scope
4. **Request Logger** - Debug logging
5. **Routes** - Business logic
6. **Error Handler** - Centralized error response

### 4. Service Layer Pattern

External integrations are abstracted:

- AWS services via `@aws-sdk/*`
- Email via `songwhipEmailApi()`
- Kafka via `KafkaProducer`
- GraphQL via code-generated types

### 5. Database Transaction Pattern

Multi-step operations use transactions:

```typescript
await db.transaction(async (trx) => {
  await album.update(trx);
  await album.createLinks(trx);
  await publishEvent(trx);
});
```

### 6. Feature Flags Pattern

Split.io provides runtime feature toggles:

```typescript
const flags = getFeatureFlags({ userId: 123 });
if (flags.songwhip_custom_layouts) {
  // New feature path
}
```

### 7. Cache Management Pattern

Strategic cache invalidation:

```typescript
deleteAlbumCache(albumId); // Invalidate CDN cache via webhooks
getAlbumCache(albumId); // Retrieve from cache layer
```

## Authentication & Authorization

### Access Token Structure (JWT)

```typescript
{
  iat: number,                          // Issued at
  exp: number,                          // Expiration
  userId: number,
  userBrand?: 'orchard' | 'awal' | 'sme',
  type?: 'user' | 'm2m',               // Machine-to-machine
  scopes?: AccessScope[],              // Permissions
  featureFlagOverrides?: Record<string, boolean>
}
```

### Token Sources

1. Authorization header: `Bearer <token>`
2. Cookie (legacy)

### Scope-Based Authorization

Access scopes provide fine-grained permissions:

- Defined in `app/lib/auth/constants.ts`
- Validated via `authorize(scope1, scope2)` middleware
- Checked in `privateRoute()` handlers

### Authentication Flow

```
HTTP Request
  → getTokenFromRequest()
  → verifyAccessToken() (JWT signature validation)
  → setRequestUser() (AsyncLocalStorage)
  → Route handler access via req.user or tokenData
```

## Error Handling Strategy

### Error Classification

- **HTTP 4xx**: Client errors (don't report by default)
- **HTTP 5xx**: Server errors (report to Sentry/Datadog)
- **Configurable**: `report: true/false` flag in ApiError

### Error Response Format

```json
{
  "status": "error",
  "error": {
    "name": "ApiError",
    "status": 400,
    "code": "CUSTOM_ERROR_CODE",
    "message": "Human readable message",
    "data": {
      /* custom context */
    },
    "stack": "..." /* dev only */
  }
}
```

### Error Handling Middleware

1. Catches all thrown errors and async rejections
2. Converts to ApiError format
3. Sends to Sentry/Datadog if `report: true`
4. Returns formatted JSON response
5. Sets Cache-Control: no-store headers

## Database Layer

### Connection Management

- Knex.js with PostgreSQL driver
- Connection pool: min=0, max=30
- SSL support for RDS with CA certificate
- Error logging via reporter

### Migration Strategy

**Schema Migrations** (versioned, reversible):

```bash
pnpm db:migrate:schema:create name
# Creates: app/database/migrations/schema/<timestamp>_<name>.ts
```

Files contain `up()` and `down()` functions following Knex conventions.

**Data Migrations** (one-way, ordered):

```bash
pnpm db:migrate:data:create name
# Creates: app/database/migrations/data/<timestamp>_<name>.ts
```

**Zod Schema Generation**:

```bash
pnpm db:migrate:schema:zod
# Generates TypeScript types from actual database schema
# Uses temp local PostgreSQL instance
# Outputs to: app/database/schema/generated/
```

### Query Patterns

All database queries go through typed Knex queries:

```typescript
await knex('albums').where({ id }).first();
await knex('users').insert(userData).returning('*');
```

Entity methods provide abstraction:

```typescript
const album = await Album.getById(123);
```

## External Service Integrations

### Kafka

- Asynchronous event publishing to message queue
- Service name: "songwhip-api"
- Brokers configured via `ORCHARD_KAFKA_BROKERS` env var
- Used for event-driven architecture

### AWS

- **S3**: Upload bucket for media files
- **SQS**: Queue for async high-load tasks (enqueueMessage pattern)
- **KMS**: Encryption key management
- Region: us-east-1
- Credentials from environment: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`

### Songwhip Email Service

- Staging endpoint: `https://email-staging.songwhip.com`
- Production endpoint: `https://email.songwhip.com`
- API key: `SONGWHIP_EMAIL_API_KEY`

### The Orchard GraphQL API

- GraphQL endpoint: `https://qa-graphql-router.theorchard.io/graphql`
- Code generation via `@graphql-codegen/cli`
- Query types in: `app/services/orchard/api/queries/`
- Schema types in: `app/services/orchard/api/schemaTypes.ts`
- Supports fragments for reusable query parts

### Sentry & Datadog

- Error/performance monitoring
- Trace ID propagation for distributed tracing
- Headers: `x-amzn-trace-id`, `x-datadog-trace-id`, `x-datadog-parent-id`

## Configuration & Environment

### Environment Detection

```typescript
getSongwhipEnv(); // Returns: 'production' | 'staging' | 'test'
isProductionEnv(); // Boolean check
isTestEnv(); // Boolean check
```

## Logging & Monitoring

### Pino Logger

- Default log level: INFO
- Development: Pretty-printed console output
- Production: JSON format with structured fields
- All logs include trace IDs

### Trace IDs

Captured from these headers (in order):

1. `x-amzn-trace-id` - AWS ALB/Lambda trace ID
2. `x-datadog-trace-id` - Datadog APM trace ID
3. `x-datadog-parent-id` - Parent span ID

### Error Reporting

- **Sentry**: Captures exceptions, performance issues
- **Datadog**: Logs and APM metrics
- Reporter interface allows multiple transports

## Testing

### Unit Tests

- Framework: Vitest
- Run: `pnpm test:unit`
- Watch mode: `pnpm test:unit:watch`

### E2E Tests

- Framework: Vitest + Supertest
- Database mocking support
- GraphQL query mocking
- Run: `pnpm test:e2e`
- Watch mode: `pnpm test:e2e:watch`

### Test Utilities

- Factory functions for entity creation
- Mock objects for services
- Snapshot testing support
- Request/response assertion helpers

## Build & Deployment

### Build Process

```bash
pnpm build
# Runs: tsc -p ./tsconfig.build.json
# Then: tsc-alias -p ./tsconfig.build.json (for path alias resolution)
# Copies SQL files to .built/ directory
```

Output: `.built/` directory with compiled JavaScript

### Startup

```bash
npm start
# Runs compiled index.js with increased max-http-header-size
```

### Local Development

```bash
pnpm dev          # With .env file
pnpm dev:staging  # With .env.staging
pnpm dev:test     # With .env.test
```

### Deployment

- Triggered automatically on commits to `master` or `staging` branches
- Uses Jenkins for orchestration
- Deployed to AWS ECS/Fargate
- Canary deployments to staging before production

## GraphQL Integration

### Code Generation Workflow

1. Queries defined in: `app/services/orchard/api/queries/`
2. Fragments defined in: `app/services/orchard/api/fragments/`
3. Include `/* GraphQL */` comment for code detection
4. Run: `pnpm graphql:codegen`
5. Generates: `generatedTypes.ts` files with TypeScript types

### Query Naming Convention

- Folder names: camelCase (e.g., `artist`, `album`)
- Query names: PascalCase without verbs (e.g., `OrchardArtist`)
- Results in types: `OrchardArtistQuery`, `OrchardArtistQueryVariables`

### Fragment Pattern

Reusable query fragments for common fields:

```typescript
export const TRACK_FRAGMENT = `/* GraphQL */
  fragment OrchardTrack on GlobalSoundRecording {
    id
    name
  }
`;
```

## Localization

### i18n System

- Tool: `@theorchard/frontend-cli-i18n`
- Scans for: `*.i18n.json` and `*.i18n.md` files
- Generates: `locales/` folder with type-safe translations
- Initialize: `pnpm i18n:init`
- Watch changes: `pnpm i18n:watch`

### Email Localization

Email templates in `app/lib/email/<emailName>/`:

- `index.ts` - Email construction logic
- `i18n.json` - Subject line and fields
- `body.i18n.md` - HTML body in Markdown
- `textBody.i18n.md` - Plain text version

Usage:

```typescript
import locales from '~/locales/fragments/myFeature';

const t = getFormatter(locales, 'fr');
const value = t('myKey', { args });
```

### Translation Sync with POEditor

```bash
POEDITOR_API_TOKEN=<token> pnpm i18n:sync
# Uploads new terms, downloads all translations
```

## Performance Considerations

### Connection Pooling

- Knex pool: min=0 (idle connections close), max=30
- Prevents connection exhaustion at scale

### Caching Strategy

- Cloudflare Edge Cache integration via webhooks
- Cache invalidation on write operations
- `RECORD_CHANGE_WEBHOOK_URLS` triggers cache purges

### Async Task Queue

- High-load operations via SQS
- Prevents blocking HTTP responses
- Long-running tasks execute asynchronously

### Feature Flags

- In-memory cache with background polling
- No per-request network calls
- Immediate flag evaluation

## Security

### CORS

- Configured in middleware
- Restricts cross-origin requests

### Proxy Trust

- `trust proxy = true` enables X-Forwarded-\* headers
- Safe behind load balancers

### Error Response Sanitization

- Stack traces hidden in production
- Error details limited to development
- Sensitive data excluded from API responses

### JWT Validation

- Signature verification required
- Expiration checked
- Schema validation with Zod

### Secret Management

- `JWT_SECRET` loaded at startup
- AWS credentials from environment
- API keys for external services in env vars

## Common Development Tasks

### Adding a New Endpoint

1. Create route file or add to existing: `app/routes/v3/<resource>/<action>.ts`
2. Define Zod schemas: `app/routes/v3/<resource>/schema/`
3. Implement handler using `privateRoute()` or `publicRoute()`
4. Call business logic from `app/logic/<resource>/`
5. Return serialized entity

### Adding a Database Migration

1. Create schema migration: `pnpm db:migrate:schema:create add_field`
2. Add SQL to `up()` and reversible SQL to `down()`
3. Generate Zod schemas: `pnpm db:migrate:schema:zod`
4. Commit and push to trigger CI deployment

### Adding a New Service Integration

1. Create folder: `app/services/<serviceName>/`
2. Initialize client/connection
3. Implement API methods
4. Export from `app/services/index.ts`
5. Use in route handlers or business logic

### Writing a Test

1. Create file: `test/unit/tests/.../<feature>.test.ts`
2. Import test utilities and factories
3. Use Vitest assertions and mocks
4. Run: `pnpm test:unit`

## Relevant File Locations

Quick reference for common modifications:

| Task               | File                                |
| ------------------ | ----------------------------------- |
| Add endpoint       | `app/routes/v3/<resource>/index.ts` |
| Define error code  | `app/<feature>/errors.ts`           |
| Add feature flag   | `app/services/split/index.ts`       |
| Change auth scope  | `app/lib/auth/constants.ts`         |
| Add email template | `app/lib/email/<name>/`             |
| Modify entity      | `app/entities/<Entity>/<Entity>.ts` |
| Add business logic | `app/logic/<domain>/`               |
| Database migration | `app/database/migrations/schema/`   |
| Update types       | `packages/songwhip-api/types.ts`    |
| Configure AWS      | `app/lib/config/aws.ts`             |

## Contact & Resources

- **API Documentation**: See `.bru` files in `/http/` directory
- **README**: `/README.md` for development setup
- **GitHub**: https://github.com/theorchard/songwhip-api
- **Notion**: Architecture documentation and guides
- **Staging**: https://api-staging.songwhip.com
- **Production**: https://api.songwhip.com