# CLAUDE.md — Orchard Insights Platform

This file provides system-wide guidance for AI assistants working across the Orchard Insights platform — a music industry analytics ecosystem built by TheOrchard (Sony Music).

## Where insights-ai Lives

The `insights-ai` repository must be a **sibling directory** alongside all service repositories. It is located under a common parent directory (e.g., `Sony/`):

```
Sony/
├── insights-ai/           # This repo (AI agents & context)
├── frontend-insights/
├── orchard-suite/
├── graphql-router/
├── graphql-analytics/
└── ... (all other services as siblings)
```

This sibling layout is **required** because:
- `setup.sh` creates symlinks using `$(dirname "$SCRIPT_DIR")` to find the parent
- Cross-service scripts read service repos from the parent directory
- All relative path references depend on this consistent layout

If `insights-ai` is placed elsewhere (e.g., in a separate location), symlinks and cross-service coordination will fail.

## System Architecture

```
                    ┌─────────────────────┐
                    │  frontend-insights   │  React 18 / TS 4.9 / Apollo Client 3
                    │   orchard-suite      │  Monorepo: 71 shared packages (React/TS)
                    └─────────┬───────────┘
                              │ GraphQL
                    ┌─────────▼───────────┐
                    │   graphql-router     │  Apollo Router 2.12 (Rust 1.94)
                    │  (17 federated       │  Custom plugins: auth, client-name
                    │   subgraphs)         │
                    └─────────┬───────────┘
           ┌─────────┬───────┼───────┬──────────┐
           ▼         ▼       ▼       ▼          ▼
     graphql-   graphql-  graphql- graphql-  graphql-
     analytics  knowledge knowledge product   user
                          -search
           │         │       │       │          │
           ▼         ▼       ▼       ▼          ▼
     OWS services  Neo4j  OpenSearch 25+ OWS  Neo4j
     (REST)       Snowflake         services  (Cypher)
                   Kafka
           │
     ┌─────┼─────┐
     ▼     ▼     ▼
   ows-  ows-  ows-
   analytics charts playlist
     │     │     │
     ▼     ▼     ▼
   Aggregated Tables (Snowflake)
   STREAMS_BY_*_DAILY, METRICS_BY_*_ROLLUP, V_STREAMS_BY_*
              ▲
              │ transforms (606+ models)
     ┌────────┴────────┐
     │  dbt-analytics   │  dbt-core 1.9 / Python 3.11 / 14 packages
     └────────┬────────┘
              │ reads
              ▼
   Raw FACT_* Tables (Snowflake Data Warehouse)
```

## Service Registry

### Frontend (React / TypeScript)

| Service | Purpose | Key Tech | Package Manager |
|---------|---------|----------|-----------------|
| `frontend-insights` | Main analytics app (Orchard Insights) | React 18, TS 4.9, Apollo Client 3, Webpack+SWC, Jest 29 | Yarn |
| `orchard-suite` | Shared component library monorepo (71 packages) | React 18, TS 5.6, Vite, Vitest, 70+ components | pnpm 9 |

### GraphQL Backend (Node.js / TypeScript)

| Service | Purpose | Key Tech | Databases | Package Manager |
|---------|---------|----------|-----------|-----------------|
| `graphql-analytics` | Analytics data aggregation | Apollo Server (wrapper 4.3), Node 22, TS 4.9 | Redis cache, 5 OWS connectors | Yarn |
| `graphql-knowledge-search` | Search across knowledge graph | Apollo Server 5, Node 22, TS 5.9 | OpenSearch, Redis, 4 OWS connectors | Yarn |
| `graphql-knowledge` | Knowledge graph operations | Apollo Server (wrapper), Node 22, TS 5.9 | Neo4j, Snowflake, Kafka, Redis | Yarn |
| `graphql-product` | Product catalog data | Apollo Server 5, Node 24, TS 5.5 | 25+ OWS connectors, OpenSearch, Redis | Yarn |
| `graphql-user` | User management | Apollo Server 5, Node 20, TS 5.9 | Neo4j, 5 OWS connectors, Redis | Yarn |

### GraphQL Router (Rust)

| Service | Purpose | Key Tech |
|---------|---------|----------|
| `graphql-router` | Apollo Federation gateway | Apollo Router 2.12, Rust 1.94, 17 subgraphs |

### Python OWS — Data Access Layer (Snowflake)

| Service | Purpose | Key Tech | Package Manager |
|---------|---------|----------|-----------------|
| `ows-analytics` | Analytics DAL (80+ endpoints) | Flask, Python 3.14, Marshmallow, Pipenv | Pipenv |
| `ows-charts` | Charts DAL | Flask 2.2, Python 3.11, Flask-SQLAlchemy, Pydantic | pip |
| `ows-playlist` | Playlist DAL | Flask 3.1, Python 3.11, JinjaSQL, Marshmallow | pip |

### Data Transformation (dbt / Snowflake)

| Service | Purpose | Key Tech | Package Manager |
|---------|---------|----------|-----------------|
| `dbt-analytics` | Snowflake data transformations (606+ models, 14 packages) | dbt-core 1.9.1, dbt-snowflake 1.9.0, Python 3.11 | Pipenv |

## Shared Infrastructure

| Concern | Technology |
|---------|-----------|
| **Authentication** | Auth0 (multi-tenant: Orchard, AWAL, SME) |
| **Feature Flags** | Split.io (`@theorchard/connector-splitio`, `pythonfeatures`) |
| **Observability** | Datadog APM (dd-trace), Sentry |
| **CI/CD** | Jenkins (declarative pipelines) |
| **Container Registry** | AWS ECR (account: 086679231553) |
| **Deployment** | AWS ECS Fargate |
| **NPM Registry** | GitHub Packages (@theorchard scope) |
| **PyPI Registry** | Internal The Orchard PyPI index |
| **Secrets** | AWS Secrets Manager |
| **Caching** | Redis (via Keyv in Node.js, redis-py in Python) |

## Shared @theorchard Packages (Node.js)

| Package | Purpose |
|---------|---------|
| `@theorchard/graphql-server` | Apollo Server wrapper (standardized config, middleware) |
| `@theorchard/datasource-ows` | REST DataSource for OWS services |
| `@theorchard/datasource-neo4j` | Neo4j DataSource |
| `@theorchard/dataloader-zod` | Zod-validated DataLoader with batching |
| `@theorchard/dataloader-redis-zod` | Redis-backed Zod DataLoader |
| `@theorchard/connector-neo4j` | Neo4j driver wrapper |
| `@theorchard/connector-splitio` | Split.io feature flag client |
| `@theorchard/connector-spotify` | Spotify API client |
| `@theorchard/jwt-service-handler` | JWT validation middleware |
| `@theorchard/suite-components` | Shared React component library (70+) |
| `@theorchard/suite-frontend` | Frontend utilities, hooks, styles |
| `@theorchard/frontend-cli` | Build CLI (Webpack/Vite) |

## Agent Routing

When working in this ecosystem, use these agents based on the service:

| Service Pattern | Language Agent | Build Agent | Review Agent |
|----------------|---------------|-------------|--------------|
| `frontend-*`, `orchard-suite` | `typescript-reviewer` | `build-error-resolver` | `code-reviewer` |
| `graphql-*` (except router) | `typescript-reviewer` | `build-error-resolver` | `code-reviewer` |
| `graphql-router` | `rust-reviewer` | `rust-build-resolver` | `code-reviewer` |
| `ows-*` | `python-reviewer` | — | `code-reviewer` |
| `dbt-analytics` | `python-reviewer` | — | `code-reviewer` |

For cross-service work, use the custom agents in `agents/`:
- `agents/cross-service.md` — End-to-end feature implementation
- `agents/dbt-service.md` — dbt model development, testing, optimization
- `agents/schema-reviewer.md` — GraphQL schema changes across federation

## Cross-Service Development Patterns

### Adding a New dbt Model

See `templates/new-dbt-model.md` for the full checklist. Summary:

1. **Identify package** — Choose the correct dbt package (streams, metrics, playlists, etc.)
2. **Implement model** — Write SQL with correct materialization (incremental/table/view)
3. **Add tests** — Schema tests (not_null, relationships) + data quality tests
4. **Verify downstream** — Check which OWS services query the output table

### Adding a New GraphQL Field (End-to-End)

See `templates/new-graphql-field.md` for the full checklist. Summary:

1. **dbt model** (if needed) — Create/modify aggregation in dbt-analytics
2. **OWS service** (Python) — Add/modify Snowflake query + REST endpoint
2. **GraphQL subgraph** (Node.js) — Add connector method, DataLoader, schema field, resolver
3. **graphql-router** — Recompose supergraph (automatic if federation schema is valid)
4. **frontend-insights** — Add `.gql` query, regenerate types, build UI

### Adding a New OWS Endpoint

See `templates/new-ows-endpoint.md`. Summary:

1. **SQL** — Write Snowflake query (JinjaSQL template or SQLAlchemy model)
2. **Handler** — Add Flask route + Marshmallow schema
3. **Tests** — Unit tests with mocked Snowflake + integration tests
4. **Connector** — Add corresponding connector method in the consuming graphql-* service

### Adding a New React Component

See `templates/new-component.md`. Summary:

1. **orchard-suite** — Create component following established patterns (see AGENTS.md in suite-components)
2. **Publish** — Build ESM+CJS, version bump, publish to GitHub Packages
3. **Consume** — Update dependency in `frontend-insights`, use in pages

## Common Commands Quick Reference

### Frontend
```bash
# frontend-insights
yarn start              # Dev server (requires VPN)
yarn test:unit          # Jest unit tests
yarn generate:types     # Regenerate GraphQL types
yarn lint               # ESLint + Stylelint

# orchard-suite
pnpm install            # Install all workspace deps
pnpm test               # Run all tests
pnpm -F @theorchard/suite-components test:unit  # Test specific package
```

### GraphQL Services (all follow same pattern)
```bash
yarn start              # Dev server with hot-reload
yarn build              # Production build (tsc)
yarn test:unit          # Jest unit tests
yarn test:integration   # Integration tests (requires running server)
yarn generate:types     # Regenerate from GraphQL schema
yarn lint               # ESLint + schema linting
```

### graphql-router
```bash
make dev                # Dev mode with hot-reload
make test               # Cargo tests
make lint               # Clippy
make supergraph         # Recompose federation supergraph
make fmt                # rustfmt
```

### OWS Services (Python)
```bash
python dev.py           # Dev server
make test_unit          # pytest unit tests
make lint               # flake8
make test_integration   # Integration tests (requires Snowflake access)
```

### dbt-analytics
```bash
make run                    # Run all models (incremental)
make run_full_refresh       # Full refresh all models
make test_unit              # dbt unit tests
make test_integration       # dbt integration tests (requires Snowflake)
dbt compile --select model  # Preview compiled SQL
dbt ls --select +model      # Show upstream dependencies
```