# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Overview

Python data pipeline that pulls Claude Code usage/cost/activity data from the Anthropic Admin APIs, processes it into JSON, and hands it to the React frontend (`../frontend`). Pure stdlib — **no runtime dependencies** (see `pyproject.toml`); dev tooling is `ruff`, `mypy`, `vulture` via `uv`.

Three data-source apps plus debug tools, all requiring `CLAUDE_ADMIN_KEY`:

1. **Usage** (`src/usage/`) — fetches usage/message data, **deduces** costs from token counts × `PRICING`. Output cost unit: **dollars**.
2. **Cost Report** (`src/cost_report/`) — fetches **direct** cost data from the Cost Report API, grouped by workspace. Output cost unit: **cents** (e.g. `177730.05` = $1777.30).
3. **Analytics** (`src/analytics/`) — fetches per-user-per-day Claude Code activity (sessions, lines of code, commits, PRs, tool actions, model token usage).
4. **Debug** (`src/debug/`) — reconciliation and diagnostic scripts.

All share `src/shared/`: `api_client.py` (`fetch_api_data()`, base URL `https://api.anthropic.com`) and `date_utils.py`.

> The data this pipeline produces is the contract with the frontend. See the root `../CLAUDE.md` for the end-to-end flow and the cents-vs-dollars trap. The frontend's Zod schemas in `../frontend/src/types/` define the expected JSON shape — changing a field here without updating those will break the frontend at load.

## Output layout

Everything writes to `output/` (relative to this dir). Per source: a `raw/` dir of untouched daily API responses, and a single processed JSON. `output/` is gitignored.

```
output/
├── usage/
│   ├── raw/{messages_YYYY-MM-DD.json, users.json, api_keys.json, workspaces.json}
│   └── costs.json            # deduced costs (dollars)
├── cost_report/
│   ├── raw/YYYY-MM-DD.json
│   └── cost_report.json      # direct costs (cents)
├── analytics/
│   ├── raw/YYYY-MM-DD.json
│   └── analytics.json
└── debug/differences.json    # from `make compare-costs`
```

The raw/processed split means HTML/JSON can be regenerated from `raw/` **without re-hitting the API** (`make regenerate-*`).

> Note: an `output/dashboard/` dir with old `index.html`/`leaderboard.html`/`view_*.js` may still exist on disk. These are stale artifacts from a removed HTML-dashboard app — no current code generates them, and the React frontend has replaced them. Ignore them.

## Commands

> **Do not run `make` targets that hit the API or upload yourself** (`update`, `fetch-*`, `upload-s3`) — they cost money and need `CLAUDE_ADMIN_KEY`. Provide the exact command for the user to run. Read-only/offline targets (`regenerate-*`, `compare-costs`, `lint`, `type-check`) are fine to run.

### Setup & code quality
```bash
make install-dev     # uv sync (installs ruff, mypy, vulture)
make format          # ruff format  (single quotes, 200-char lines — see ruff.toml)
make lint            # ruff check
make lint-fix        # ruff check --fix
make type-check      # mypy --strict, Python 3.13 (see pyproject.toml)
make check           # lint + type-check
```
`make test` is a placeholder (no test framework wired up). Always format before committing.

### Full refresh (the main workflow)
Run from `api/`. Interactive — prompts for how many days back, then fetches all three sources, regenerates, compares, syncs to frontend, and uploads to S3:
```bash
make update
```

### Per-source targets
```bash
# Usage (deduced costs)
make fetch-usage-full          # Aug 28 → today, daily
make fetch-usage-day           # prompts for a single date
make fetch-metadata            # only users/api_keys/workspaces metadata
make regenerate-usage          # rebuild costs.json from raw/ (no API call)

# Analytics
make fetch-analytics-full      # Aug 28 → today, daily
make fetch-analytics-day       # prompts for a single date
make regenerate-analytics      # rebuild analytics.json from raw/ (no API call)

# Cost Report
make fetch-cost-report-full    # Aug 28 → today, workspace-level
make regenerate-cost-report    # rebuild cost_report.json from raw/ (no API call)

# Reconciliation + sync
make compare-costs             # diff the three sources → output/debug/differences.json
make sync-frontend             # cp processed JSON into ../frontend/src/data/
```

### Direct script invocation
Each analyzer is a module entry point. Common flags: `--admin-key` (falls back to `CLAUDE_ADMIN_KEY`), `--days N` or `--start/--end`, `--time-grouping {day,month}`, `--format`, `--output`, `--debug`.
```bash
uv run python -m src.usage.cost_analyzer --start 2025-08-28 --end 2025-10-25 \
  --time-grouping day --format json --output output/usage/costs.json --debug
uv run python -m src.usage.rebuild_costs_from_raw --time-grouping day --output output/usage/costs.json

uv run python -m src.analytics.analytics_analyzer --days 30 --time-grouping day \
  --format json --output output/analytics/analytics.json
uv run python -m src.analytics.regenerate_from_raw --output output/analytics/analytics.json

uv run python -m src.cost_report.cost_report_analyzer --start 2025-08-28 --end 2025-10-27 \
  --output output/cost_report/cost_report.json --debug
uv run python -m src.cost_report.regenerate_from_raw --output output/cost_report/cost_report.json
```

## Architecture

### Usage app (`src/usage/`)
- `cost_analyzer.py` — main analyzer; fetches usage and **deduces cost from tokens** using the `PRICING` dict (top of the file, ~line 27). Time grouping builds `api_key → time_period → model → usage`.
- `cost_fetcher.py` — simpler usage fetcher. `fetch_metadata.py` — users/api_keys/workspaces only.
- `rebuild_costs_from_raw.py` — regenerate `costs.json` from `output/usage/raw/`.
- `csv_to_json.py`, `add_totals_to_csv.py` — CSV helpers.
- Cost data flows as plain dicts (no dataclasses here).

### Cost Report app (`src/cost_report/`)
- Endpoint `/v1/organizations/cost_report`; params `starting_at`, `ending_at`, `bucket_width`, `group_by`, `limit`. Returns **pre-calculated costs in cents**, grouped by workspace/model/token-type.
- `cost_report_types.py` — dataclasses (`CostBucket`, `CostReportResponse`, `WorkspaceCosts`, `ModelCosts`, `TimePeriod`).
- `cost_report_fetcher.py`, `cost_report_analyzer.py`, `regenerate_from_raw.py`.

### Analytics app (`src/analytics/`)
- Endpoint `/v1/organizations/usage_report/claude_code`; single `starting_at` date per call, so the fetcher loops one call per day. Structure: `actor → time_period → metrics`.
- `analytics_types.py` — dataclasses (`ClaudeCodeUsageRecord`, `ActorInfo`, `CoreMetrics`, `ToolActions`, `ModelBreakdown`), Python 3.13 `X | None` union syntax.
- `analytics_analyzer.py`, `regenerate_from_raw.py`.

### Debug tools (`src/debug/`)
- `compare_costs.py` — aggregate the three sources by date, flag days with large gaps → `output/debug/differences.json` (run via `make compare-costs`).
- `verify_differences.py` — compares Usage and Analytics against Cost Report (baseline); uses `src.usage.cost_analyzer.PRICING`.
- `debug_cost_api.py` — investigate one user/date, e.g. `python -m src.debug.debug_cost_api 2025-10-18 --email user@example.com` (reads `output/usage/raw/`).
- `diagnose_missing_data.py` — find sync gaps between sources. `anonymize_costs.py` — strip PII for sharing.

## Common tasks

### Add a model to pricing
Edit the `PRICING` dict in `src/usage/cost_analyzer.py`. Values are **USD per million tokens**; cache-write costs default to the input price if omitted.
```python
PRICING = {
    'claude-new-model-20251231': {
        'input': 3.00, 'output': 15.00,
        'cache_creation': 3.75, 'cache_read': 0.30,
        'cache_write_5m': 3.75, 'cache_write_1h': 3.75,  # optional
    },
}
```

## Patterns

- **Type safety**: mypy `--strict`, Python 3.13. Dataclasses for the cost-report and analytics response shapes; use `Path`, not string paths.
- **Defensive I/O**: file reads wrapped with fallbacks; warnings printed rather than raised so a partial dataset still produces output.
- **Incremental updates**: always fetch raw → `regenerate-*` to rebuild the full processed JSON from all raw files. No merge logic; the raw dir is the source of truth.

## History

The repo was split (Oct 2025) out of a larger project; conversation-rating tooling moved elsewhere. A standalone HTML-dashboard app (`src/dashboard/`, `src/shared/html_*`, `highcharts_config.py`, `view_*.py` generators) was later **removed** — the React app in `../frontend` replaced it. If you find references to those modules in older docs, they no longer exist.