# CLAUDE.md — ows-analytics

This file provides guidance for AI assistants working in the ows-analytics repository.

## Project Overview

**ows-analytics** is a Python Flask REST API that serves as the Data Access Layer (DAL) to Snowflake for streaming analytics data. It is the largest OWS service by endpoint count (80+), providing analytics for accounts, participants (artists), products, sound recordings, and TikTok data across 51+ streaming platforms. Consumed by `graphql-analytics` and other internal services.

## Essential Commands

```bash
# Development
python dev.py               # Dev server with hot-reload

# Testing
make test_unit              # pytest unit tests with coverage
make test_integration       # Integration tests (requires Snowflake)
make test                   # All tests

# Linting
make lint                   # flake8 analytics/ tests/

# Full CI job
make unit_lint_job          # pip_dev + lint + test_unit
make integration_job        # Integration tests against QA
```

To run a single test file:
```bash
pytest tests/unit/test_handlers.py -v
pytest tests/unit/test_handlers.py::test_specific_function -v
```

## Setup

Copy `.env.shadow` to `.env` before running locally: `cp .env.shadow .env`

Snowflake access requires:
- Private key in `~/.ssh/snowflake/` (local dev)
- AWS Secrets Manager credentials (QA/prod)

## Architecture

### Tech Stack

| Layer | Technology |
|-------|-----------|
| Framework | Flask (with flask-compress, flask-executor) |
| Language | Python 3.14 |
| Database | Snowflake (via snowflake-connector-sqlalchemy 1.7.3) |
| ORM | SQLAlchemy 1.4+ |
| Validation | Marshmallow |
| Caching | Redis (with FakeRedis fallback) |
| Package Manager | Pipenv (Pipfile + Pipfile.lock) |
| APM | Datadog (ddtrace) |
| Error Tracking | Sentry SDK |
| Authorization | owsrequest + access_rules.yml |
| Feature Flags | pythonfeatures |
| API Docs | flasgger (Swagger) |

### Directory Structure

```
analytics/
├── api.py                       # Flask app setup + flask-executor
├── config.py                    # Extensive config (~23KB: stores, feeds, thresholds)
├── application.py               # Entry point (ddtrace, sentry, compress)
├── handlers.py                  # Main handlers (~73KB, largest file)
├── account_handlers.py          # Account-specific endpoints
├── participant_handlers.py      # Artist/participant endpoints
├── product_handlers.py          # Product endpoints
├── sound_recording_handlers.py  # Sound recording endpoints
├── tiktok_handlers.py           # TikTok-specific endpoints
├── handler_utils.py             # Shared handler utilities
├── features.py                  # Feature flag logic
├── access_rules.yml             # YAML-based access control rules
├── connectors/                  # External service connectors
│   ├── snowflake.py             # Snowflake connection (QueuePool, PKI auth)
│   ├── dynamo.py                # DynamoDB metadata
│   └── redis.py                 # Redis cache (FakeRedis fallback)
├── constants/                   # ~20 constant modules (feeds, stores, dates, queries)
├── models/                      # SQLAlchemy ORM models (~26 files)
├── schemas/                     # Marshmallow response schemas (~30 files)
├── queries/                     # Query classes per domain (~16 directories)
│   ├── account/
│   ├── sound_recording/
│   ├── participant/
│   ├── product/
│   ├── tiktok/
│   └── common/
├── logic/                       # Business logic (~45 modules)
│   ├── dataloaders/             # DataLoader pattern for N+1 prevention
│   ├── metadata_loader.py
│   ├── permissions_logic.py
│   └── cache_management.py
├── services/                    # Service layer
│   ├── account_service.py
│   └── permissions/
├── validation/                  # Input validation
├── utils/                       # Utilities (DB, caching, SQL formatting)
└── permissions/                 # Permission modules

tests/
├── unit/                        # Unit tests (mock Snowflake, Redis, features)
│   ├── conftest.py              # Autouse fixtures
│   └── fixtures/
└── integration/                 # Integration tests (real Snowflake)

warmup_cache/                    # Cache warming scripts
spec/                            # OpenAPI specifications
```

### Data Flow

```
HTTP Request → owsrequest (auth + headers) → Handler → Logic → Query → Snowflake
                                                  │        ↑
                                                  │    Redis cache
                                                  ↓
                                          Marshmallow schema → JSON Response
```

## Key Patterns

### Store & Feed Definitions

`config.py` contains extensive dictionaries mapping streaming platforms to store IDs, feed IDs, and update thresholds. 51+ feeds covering Spotify, Apple Music, Amazon, YouTube, TikTok, Deezer, etc.

### Handler Architecture

5 handler modules organize 80+ endpoints by domain:
- `handlers.py` — Main/shared endpoints
- `account_handlers.py` — `/account/<id>/...`
- `participant_handlers.py` — `/participant/<id>/...`
- `product_handlers.py` — `/product/<id>/...`
- `sound_recording_handlers.py` — `/sound-recording/<isrc>/...`
- `tiktok_handlers.py` — TikTok-specific endpoints

### Authorization (access_rules.yml)

Fine-grained access control defined in YAML. The `owsrequest` library validates Grass headers (account type, ID, profile ID) against these rules per-endpoint.

### DataLoader Pattern

`logic/dataloaders/` prevents N+1 queries by batching related data fetches.

### DynamoDB Metadata

Analytical metadata (feed status, high water marks, store outages) is stored in DynamoDB, separate from the Snowflake analytics data.

### Cache Warming

Separate scripts in `warmup_cache/` pre-populate Redis cache after deployments.

### Snowflake Connection

- **Auth**: Private key pair (PKI) — not password-based
- **Pool**: QueuePool with 15 connections, 4hr recycle, max 5 overflow
- **Two databases**: APOLLO_DB (metadata) + main analytics database
- **Session**: `keep_alive=True`, DDL passthrough enabled

## Environment Variables

| Variable | Purpose |
|----------|---------|
| `ENV` | `qa` / `prod` |
| `SNOWFLAKE_ACCOUNT` | Snowflake account identifier |
| `SNOWFLAKE_USER` | Snowflake username |
| `SNOWFLAKE_ROLE` | Snowflake role |
| `SNOWFLAKE_WAREHOUSE` | Snowflake warehouse |
| `SNOWFLAKE_DATABASE` | Snowflake database |
| `SNOWFLAKE_SCHEMA` | Snowflake schema |
| `REDIS_HOST` | Redis host |
| `SPLIT_API_KEY` | Split.io feature flags |
| `SENTRY_DSN` | Sentry error tracking |
| `DD_*` | Datadog configuration |

## Testing

- **Unit tests**: Mock Snowflake, Redis, and feature flags via autouse fixtures in `conftest.py`
- **Markers**: `disable_mock_cache`, `disable_mock_execute` to override default mocks
- **Coverage target**: 80%+ (enforced in CI)
- **Integration tests**: Run against real Snowflake (separate environment/venv)

## Docker

Multi-stage: `base` (python:3.11-slim-bullseye) → `deploy` (uWSGI) | `pr_tests` (CI).

Virtual environment created inside container at `/var/app/env`.