# ows-product-staging - GitHub Copilot Instructions

## Project Overview

**ows-product-staging** is an Orchard web service that serves as a staging area for content before ingest. It provides APIs for managing bulk content sessions, metadata uploads, asset management, and ingestion workflows for music products, tracks, and projects.

**Tech Stack:**
- **Language:** Python 3.13
- **Framework:** FastAPI (async web framework)
- **ORM:** SQLAlchemy 2.0 (async)
- **Database:** MySQL (prod/qa), SQLite (test)
- **Cache:** Redis
- **Cloud:** AWS (S3 via aioboto3)
- **Deployment:** Docker, Jenkins CI/CD
- **Dependencies:** uv for package management

## Core Architecture

### Layered Architecture Pattern
The application follows a clean architecture with clear separation of concerns:

```
API Layer (routers) → Logic Layer → Connector Layer → External Services/DB
                   ↓
              Schemas (Pydantic models for validation)
                   ↓
              Models (SQLAlchemy ORM models)
```

### Key Components

1. **API Layer** (`product_staging/api/`)
   - FastAPI application with middleware for auth, logging, context
   - JWT authentication (environment-based)
   - Correlation ID tracking
   - Sentry integration for error monitoring

2. **Logic Layer** (`product_staging/logic/`)
   - Business logic for bulk sessions, ingestion, uploads
   - Stateless service layer functions
   - Handles workflows and orchestration

3. **Connector Layer** (`product_staging/connectors/`)
   - Database connections (MySQL/SQLite)
   - Redis cache
   - External OWS services (account, permissions, pricing, product-digital)
   - S3 operations via aioboto3

4. **Models** (`product_staging/models/`)
   - SQLAlchemy ORM models for database tables
   - Represents bulk sessions, ingestion entities, assets, metadata files

5. **Schemas** (`product_staging/api/schemas/`)
   - Pydantic models for request/response validation
   - API contracts and data transfer objects

## Directory Structure

```
.
├── product_staging/              # Main application package
│   ├── __init__.py
│   ├── config.py                 # Application configuration (env, DB, Redis, secrets)
│   ├── exceptions.py             # Custom exceptions
│   │
│   ├── allowed_value_sets/       # Sub-module for validation enums
│   │   ├── api/routers/          # Endpoints for allowed values
│   │   ├── connectors/           # Fetch enums from external services
│   │   └── logic/                # Enum validation business logic
│   │
│   ├── api/                      # FastAPI application
│   │   ├── main.py               # FastAPI app initialization
│   │   ├── auth.py               # Authentication utilities
│   │   ├── datasources.py        # Lifespan management (DB, Redis)
│   │   ├── error_handlers.py     # Global error handling
│   │   ├── routers/              # API endpoint routers
│   │   │   ├── bulk_session.py           # Bulk session CRUD
│   │   │   ├── bulk_session_assets.py    # Asset management
│   │   │   ├── bulk_session_ingestion.py # Ingestion workflows
│   │   │   ├── bulk_session_metadata.py  # Metadata uploads
│   │   │   └── infra.py                  # Health checks
│   │   └── schemas/              # Pydantic models for API
│   │
│   ├── cli/                      # Command-line utilities
│   │   └── export/               # Data export scripts
│   │
│   ├── connectors/               # External service integrations
│   │   ├── db.py                 # Database session management
│   │   ├── redis.py              # Redis cache client
│   │   ├── ows_account.py        # Account service client
│   │   ├── ows_permissions.py    # Permissions service client
│   │   └── features/             # Feature flag management
│   │
│   ├── constants/                # Application constants
│   │   ├── auth.py
│   │   ├── bulk_session_ingestion.py
│   │   ├── error.py
│   │   ├── features.py           # Feature flags
│   │   ├── header.py             # HTTP headers
│   │   ├── metadata.py
│   │   └── services.py
│   │
│   ├── logic/                    # Business logic layer
│   │   ├── bulk_session*.py      # Bulk session management
│   │   ├── asset*.py             # Asset handling
│   │   ├── *_upload.py           # Upload workflows
│   │   ├── *_ingestion*.py       # Ingestion processes
│   │   ├── jwt.py                # JWT utilities
│   │   ├── metadata_json/        # Metadata JSON parsing
│   │   └── utils/                # Shared utilities
│   │
│   ├── models/                   # SQLAlchemy ORM models
│   │   ├── bulk_session.py               # Main session entity
│   │   ├── bulk_session_asset*.py        # Asset file tracking
│   │   ├── bulk_session_ingestion*.py    # Ingestion entities
│   │   ├── bulk_session_metadata_file.py # Metadata file tracking
│   │   └── task.py                       # Background tasks
│   │
│   └── schema/                   # Data validation schemas
│       ├── catalog.py            # Catalog data structures
│       └── metadata_json.py      # Metadata JSON schemas
│
├── tests/                        # Test suite
│   ├── unit/                     # Unit tests (mirrors src structure)
│   ├── integration/              # Integration tests
│   │   ├── client.py             # Test client utilities
│   │   ├── conftest.py           # Pytest fixtures
│   │   └── seed.py               # Test data seeding
│   └── data/                     # Test fixtures and sample data
│
├── scripts/                      # Build and test scripts
│   ├── integration.sh            # Run integration tests
│   └── unit-lint.sh              # Run unit tests and linting
│
├── dev.py                        # Local development server
├── uvicorn-start.sh              # Production server startup
├── docker-compose.yml            # Docker compose for deploy
├── docker-compose-dev.yml        # Docker compose for development
├── Dockerfile                    # Container image definition
├── Jenkinsfile                   # CI/CD pipeline
├── Makefile                      # Build automation commands
├── pyproject.toml                # uv dependencies and config
└── software-catalog.yaml         # Service catalog metadata
```

## Key Domain Concepts

### Bulk Sessions
A **bulk session** represents a batch ingestion workflow for uploading and processing music content (products, tracks, projects) along with their metadata and assets.

**Workflow:**
1. Create bulk session
2. Upload metadata files (Excel/JSON)
3. Upload asset files (audio, images)
4. Create ingestion records (products, tracks, projects)
5. Execute ingestion to downstream systems
6. Track status and generate reports

**Related Models:**
- `BulkSession` - Main session entity
- `BulkSessionMetadataFile` - Uploaded metadata files
- `BulkSessionAssetFile` - Uploaded asset files
- `BulkSessionIngestion` - Ingestion jobs
- `BulkSessionIngestionProduct/Track/Project` - Individual entities to ingest

### Allowed Value Sets
Enumeration and validation of allowed values for metadata fields (languages, territories, genres, etc.) fetched from external services (OWS Product Digital, OWS Pricing, OWS Carveouts).

## Development Patterns

### Async/Await
- All database operations use SQLAlchemy async
- Use `async with` for database sessions
- HTTP clients use `httpx` or `aioboto3` (both async)
- FastAPI endpoints are `async def`

### Dependency Injection
FastAPI dependencies are used for:
- Database sessions: `get_db_session()`
- Authentication: JWT middleware
- Request context: Correlation IDs

### Error Handling
- Custom exception hierarchy inherits from `OwsProductStagingException`
- Global error handlers in `error_handlers.py`
- Sentry integration for production error tracking

### Configuration
- Environment-based config via `environs`
- Secrets managed via `secrets-manager` library
- `.env` file for local development
- Environment variable: `Environment` (dev/qa/prod/test)

### Testing
- **Unit tests:** `tests/unit/` - Mock external dependencies
- **Integration tests:** `tests/integration/` - Dockerized, test against real services
- Pytest fixtures in `conftest.py`
- Run with: `make test_unit` or `make docker_integration_test`

## Common Commands

```bash
# Development
make dev                # Run local dev server with hot reload
make trace-dev          # Run with DataDog tracing enabled

# Testing
make test_unit          # Run unit tests
make watch_unit         # Run tests in watch mode
make docker_integration_test  # Run integration tests in Docker

# Linting & Formatting
make lint               # Run ty and ruff
make fmt                # Format code with ruff

# Docker
make up_dev             # Run dev container (port 8888)
make up_deploy          # Run deploy container (port 8889)
make docker_unit_lint   # Run tests and lint in Docker
```

## Coding Conventions

### Naming
- **Files:** snake_case (e.g., `bulk_session_ingestion.py`)
- **Classes:** PascalCase (e.g., `BulkSessionIngestion`)
- **Functions/Variables:** snake_case (e.g., `get_bulk_session_by_id`)
- **Constants:** UPPER_SNAKE_CASE (e.g., `SERVICE_NAME`)

### Imports
- Standard library imports first
- Third-party imports second
- Local application imports last
- Use absolute imports from `product_staging.*`

### Type Hints
- Use type hints on all function signatures
- Models use Pydantic BaseModel or SQLAlchemy declarative
- ty is enforced in CI

### Documentation
- Docstrings for modules, classes, and public functions
- Use `"""Triple quotes"""` for docstrings
- Inline comments for complex logic

## Database

### SQLAlchemy Models
- Inherit from SQLAlchemy declarative base
- Async engine and sessions
- Models in `product_staging/models/`
- Use `relationship()` for foreign keys
- Timestamps: `created_at`, `updated_at` (auto-managed)

### Migrations
- Database schema changes handled outside this repo
- Coordinate with DBA team for schema updates

## External Service Integration

### OWS Services
- **ows-account:** User account management
- **ows-permissions:** Authorization checks
- **ows-product-digital:** Product metadata and enums
- **ows-pricing:** Pricing-related enums

### AWS Services
- **S3:** Asset and metadata file storage (via `aioboto3`)
- **Secrets Manager:** Credential management (via `secrets-manager`)

## Authentication & Authorization

- JWT-based authentication via `jwtauth` library
- Environment-based: disabled in dev/test, enabled in qa/prod
- Auth middleware validates tokens on protected routes
- Permissions checked via `ows-permissions` service

## Monitoring & Observability

- **Logging:** Structured JSON logs via `audience-common.logger`
- **Tracing:** DataDog APM integration (`ddtrace`)
- **Errors:** Sentry SDK for error tracking
- **Metrics:** Correlation IDs for request tracking

## Important Notes for Copilot

1. **Always use async/await** for database and HTTP operations
2. **Follow the layered architecture:** API → Logic → Connector
3. **Use dependency injection** for database sessions and auth
4. **Environment awareness:** Code must support dev/test/qa/prod environments
5. **Error handling:** Use custom exceptions and global error handlers
6. **Type safety:** Add type hints and ensure ty compliance
7. **Testing:** Write unit tests for logic, integration tests for endpoints
8. **Dependencies:** Use uv for package management (`uv add <package>`)
9. **Feature flags:** Check Split.io feature flags in logic layer
10. **Security:** Never log sensitive data (tokens, secrets, PII)

## Related Services

This service interacts with:
- **ows-account** - User and account management
- **ows-permissions** - Authorization service
- **ows-product-digital** - Product catalog and enums
- **ows-pricing** - Pricing data and enums
- **ows-carveouts** - Carveout restrictions

## Contact & Documentation

- **Repository:** git@github.com:theorchard/ows-product-staging.git
- **Authors:** Sony Music PDE
- **Service Catalog:** See `software-catalog.yaml`