# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this repo is

A monorepo of independently-deployed AWS workloads (Lambdas and Fargate tasks) plus CLI scripts that automate RDS database operations for Sony Music / The Orchard: refreshing non-prod databases from prod snapshots, sanitising PII, managing backups/snapshots, RDS user creation, and integrations (Snowflake refresh, Fivetran re-sync, Kafka/Debezium connectors, Datadog monitoring).

Each subproject is its own self-contained unit with its own dependencies, tests, and Docker image. There is no shared top-level build — you operate inside one subproject directory at a time.

## Layout

- `lambda/<name>/` — Lambda functions (e.g. `sanitise_rds_data`, `snowflake_refresh`, `fivetran_sync`, `extract_native_configuration`).
- `fargate/<name>/` — Fargate tasks (`restore`, `create_shareable_snapshot`, `manage_backup_snapshots`).
- `scripts/<name>/` — interactive CLI tooling (`rds-user-creation`).
- `common/` — shared Python package injected into other images as a Docker build context named `common` (see `additionalContexts` in `Jenkinsfile`). It is NOT pip-installed from a registry; the build wires it in.

Most subprojects follow: `src/app.py` (Lambda `handler(event, context)` entry point), `src/logic/` (DB-engine-specific logic, often `mysql.py` / `postgresql.py`), `src/connectors/boto_clients.py`, `config.py` (env-var config), and `tests/unit/` + `tests/integration/`.

## Two toolchains — check which one a subproject uses before running anything

The repo is mid-migration. A subproject uses **uv + ruff + mypy** if it has a `pyproject.toml` with a `[tool.uv.index]`/`[tool.ruff]` section (`common`, `lambda/fivetran_sync`, `lambda/extract_native_configuration`). Everything else uses **pip + flake8** (presence of a `.flake8` file).

This determines what `lint-and-test.sh` runs:
- **flake8 projects**: `flake8 src/ tests/ config.py`
- **uv projects**: `uv run ruff check`, `uv run ruff format --check`, `uv run mypy src/` — line length 82, ruff lint rules `E,F,I,Q,D`.

Don't introduce `uv`/`ruff` into a flake8 project or vice versa; match the subproject.

## Commands (run inside a subproject directory)

All dev work runs in Docker via the per-project `Makefile` / `docker-compose.yaml`. Don't run pytest/flake8 on the host.

```bash
cd lambda/<name>          # or fargate/<name>, scripts/<name>

make lint_and_test        # build_test image + run lint + unit tests (the standard check)
make build                # build the function/service image
make build_test           # build only the test image
```

`make lint_and_test` ≈ `docker compose run --rm lint-and-test`, which executes `lint-and-test.sh`. Integration tests are excluded there (`--ignore=tests/integration/`); unit tests under `tests/` run with coverage over `src/`.

### Integration tests

Integration tests (`tests/integration/`) are a separate, local-only gate — they're excluded from `lint_and_test` and not run in CI. Only some subprojects have them. `lambda/sanitise_rds_data` has the fleshed-out example: `make integration_test` builds the test image and runs the suite against throwaway `postgres-source` / `postgres-target` containers (defined in its `docker-compose.yaml`, data dirs on `tmpfs`), exercising the real role-copy/script-running logic instead of mocks. The target always runs `docker compose down` afterward (even on failure) and re-exits with the test status.

### Tuning a test run

`lint-and-test.sh` reads env vars set in `docker-compose.yaml` under the `lint-and-test` service. To run a subset or skip lint locally, edit those values (do **not** commit the change — it alters the CI build):

- `TEST_ARGS` — passed verbatim to pytest, no quotes. E.g. `TEST_ARGS=-v -k test_name` to run a single test.
- `SKIP_LINT=1` — skip linting.
- `COV_REPORT=html` — serve an HTML coverage report on `http://localhost:8000/` (default `term`; `off` disables).

For a CI-accurate exit code:
```bash
docker compose up --exit-code-from lint-and-test --abort-on-container-exit --build lint-and-test
```

## CI / deploy

Two Jenkins pipelines:
- `Jenkinsfile` — build/test/release/deploy pipeline. The `PROJECTS` map at the top is the registry of every deployable subproject: its image name, `serviceType` (`LAMBDA` or `FARGATE`), target AWS accounts/roles per environment, and whether it pulls in the `common` build context. **Adding a new subproject means adding an entry here.**
- `Jenkinsfile.refresh` — orchestrates the actual RDS refresh by triggering the `shared-rds-refresh-state-machine` Step Function; the `SERVICES` map defines per-database source/target environments, restore roles, sanitisation function, and Debezium/Kafka connectors.

Images build `FROM` the internal parent image `086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:lambda-python314`. The internal package index is `https://pypi.theorchard.io/pypi/` (e.g. the `lambdacommon`, `owslogger` deps).

## RDS refresh domain model

The subprojects here are the *steps*; the *orchestration* lives in a separate repo: `terraform-infra/shared/prod/rds-refresh/`. That Terraform deploys the Step Function `shared-rds-refresh-state-machine` (the same ARN `Jenkinsfile.refresh` triggers) from `sfn.json`, and wires each step's deployed Lambda/Fargate name into the state-machine template via `sfn.tf` (`templatefile(...)`). To change orchestration logic (ordering, branching, retries) edit `sfn.json` there; to change a step's behaviour edit the subproject here.

### Step Function flow (`sfn.json`)

Input fields drive the branching — key ones: `source_account_id`/`target_account_id`, `source_db_name`/`target_db_name`, `target_account_role`, `force_snapshot`, `kafka_connector` (+ `kafka_bootstrap_servers`/`kafka_topics`/`connector_name`/`connector_account_id`), `requires_snowflake_refresh`, `requires_fivetran_sync` (+ `fivetran_connector_id`/`fivetran_historical_sync`).

1. **GetSourceDbInfo → GetTargetDbInfo** (`lambda/get_db_info`) — fetch DB type + config for both ends.
2. **ValidateConfig** — fail fast if source/target DB types differ.
3. **Restore-mode decision** (a chain of Choice states): same account, source encrypted with the default KMS key, `force_snapshot`, or target-is-a-cross-account-clone all influence whether `clone_restore` is true (fast clone, skips snapshot) or false (full snapshot path → **CreateShareableSnapshot**, `fargate/create_shareable_snapshot`).
4. **ScaleInKafkaConnector** (`lambda/manage_kafka_connector`) — only if `kafka_connector` is present; pause CDC before restore, scaled back out afterward.
5. **Restore** (`fargate/restore`) — restore into the target DB; it invokes the sanitise function (`sanitise_data_function_name` → `lambda/sanitise_rds_data`, which copies users then runs per-DB SQL scripts). The RDS-level steps in `common/logic/database.py` are engine-agnostic — they branch only on `cluster`/`standalone` and pass the source's `Engine` through. The **engine-specific** part is the sanitise step: `fargate/restore` passes the engine (from `get_config`'s `Engine`) into the sanitise payload, and `sanitise_rds_data/src/app.py` dispatches to `src/logic/mysql.py` (pymysql, `mysql.user`/grants) or `src/logic/postgresql.py` (psycopg, `pg_authid` roles/memberships) on it. Both logic modules share the `copy_users(source, target)` / `run_scripts(creds, dir)` interface. Per-DB sanitisation SQL must be written in the matching dialect.
6. **Cleanup** — ScaleOutKafkaConnector (restore CDC), then DeleteIntermediateSnapshots in source + target accounts (`lambda/delete_intermediate_snapshots`), skipped on the clone path.
7. **RefreshSnowflakeDatabase** (`lambda/snowflake_refresh`) and **FivetranSync** (`lambda/fivetran_sync`) — conditional post-refresh integrations.
8. **CheckForError** — Fargate/Lambda steps `Catch` to `Cleanup` and stash `$.error`; a present `$.error` routes to a terminal `Fail`.

The Fargate steps use `ecs:runTask.waitForTaskToken` (the task calls back with `TASK_TOKEN`); long timeouts (CreateShareableSnapshot 7200s, Restore 10800s).

### Sanitisation scripts

`lambda/sanitise_rds_data/scripts/<db-identifier>/` holds the SQL run during sanitisation. Rules (`scripts/README.md`):
- The folder name must exactly match the RDS database identifier (e.g. `qa-art-relations`).
- Multiple `.sql` files run in **lexicographical order** — hence the numeric prefixes (`001_…`, `002_…`).
- A database that needs no sanitisation must still have a folder with a README explaining why.

Many commits to this repo are data/permission changes: adding sanitisation SQL, RDS users (`rds-user-creation`), or grants — not application-code changes.