# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

---

## Architecture

Three EventBridge/SQS-triggered Lambda functions that automate Shopify store onboarding through Fivetran into Snowflake.

```
EventBridge (hourly) → Lambda 1: connect-shopify-stores
                         Scans ARTIST_STORES, excludes duplicates/CRM/Fansifter/already-synced
                         Pushes each new store to SQS → Lambda 2
                         Writes initial ECOMMERCE_STORE_REGISTRY row (owns IS_ACTIVE)

SQS → Lambda 2: onboard-shopify-stores
       Creates Fivetran connector, issues Connect Card URL
       Writes pending_oauth to D2C_ONBOARDING_STATE

EventBridge (every 15 min) → Lambda 3: sync-shopify-data-connectors
                               Sweeps in-flight connectors; advances state machine
```

### State machine

```
pending_oauth → pending_configure → pending_sync → sync_complete
             ↘ oauth_timeout (14 days)
                               ↘ broken (connector BROKEN)
```

- `pending_configure`: OAuth completed; next sweep applies table schema config and triggers sync
- `pending_sync`: waiting for the `SHOP` table to appear in `SHOPIFY_STORES_GLOBAL.INFORMATION_SCHEMA`
- `sync_complete`: SHOP present; registry enriched with vendor ID and Global Participant

### Lambda layout

Each Lambda is fully self-contained under `lambda/<name>/`:

| File | Role |
|---|---|
| `src/app.py` | Lambda handler entry point |
| `src/fivetran_client.py` | Fivetran REST API wrapper (pydantic models, retry/backoff) |
| `src/snowflake_client.py` | Snowflake connection + query helpers |
| `src/store_utils.py` | `normalise_url`, `derive_schema_name` |
| `config.py` | Env-var config; loads secrets from Secrets Manager in qa/prod |
| `tables_config.json` | Shopify tables to enable (Lambda 2 and 3 only) |
| `Dockerfile`, `pyproject.toml`, `uv.lock` | Self-contained build |

---

## Local development

### Prerequisites

- Colima or Docker Desktop
- `uv` — `brew install uv`
- Snowflake private key at `<path-to-snowflake-key>` (e.g. `~/.ssh/snowflake/rsa_key.p8`)
- AWS SSO credentials exported in shell

### Environment setup

```bash
cp lambda/connect-shopify-stores/.env.shadow lambda/connect-shopify-stores/.env
# Fill in: SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER, SNOWFLAKE_ROLE, SNOWFLAKE_KEY_PASSPHRASE
# For Lambda 2 and 3 also fill in: FIVETRAN_API_KEY, FIVETRAN_API_SECRET
```

`ENVIRONMENT=local` (the default) bypasses Secrets Manager; credentials are read directly from env.

### Build and run

```bash
colima start

docker build -t lambda-d2c-connect-shopify-stores:local lambda/connect-shopify-stores/
docker run -p 9000:8080 \
  --env-file lambda/connect-shopify-stores/.env \
  -v <path-to-snowflake-key>:/var/task/rsa_key.p8 \
  lambda-d2c-connect-shopify-stores:local

# Second terminal — invoke the handler:
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
```

Use the same pattern for the other two Lambdas (different image name and `.env` file).

### Linting

Ruff config is in `pyproject.toml` at the repo root (target Python 3.13, line length 120):

```bash
ruff check .
ruff format .
```

### Updating dependencies

```bash
cd lambda/<name>
uv add <package>   # updates pyproject.toml and uv.lock; rebuild the image after
```

---

## CI / deployment

Jenkins (`Jenkinsfile`) detects which Lambdas were modified per-commit and builds only those. On `master`:
1. Builds Docker images and pushes to ECR (`lambda-d2c-<name>`)
2. Scans for vulnerabilities
3. Deploys to QA Lambda functions
4. Publishes Datadog software catalog definitions

Terraform manages the Lambda functions, SQS queue, EventBridge rules, and IAM in `terraform-infra/ecommerce/`.

---

## Critical invariants

- **`FIVETRAN_GROUP_ID` is selected by environment** — Lambda 2's `config.py` uses `glade_scone` for local/QA and `stimuli_armrest` for prod.
- **Always read schema name from the connector, not state.** Use `conn.name` (the connector's actual destination schema) for any SHOP-table lookup or registry enrichment. The derived `schema_name` stored in `D2C_ONBOARDING_STATE` can drift.
- **Lambda 1 owns `IS_ACTIVE`.** The sync Lambda's `upsert_registry` never writes this field. Do not add it to sync-lambda registry writes.
- **`tables_config.json` must reflect actual dbt model dependencies.** Enabling a table that has no dbt model, or disabling one that does, breaks downstream reporting. Check `shopify-global-schema` before modifying.
- **Enqueue to SQS before writing the registry row** in Lambda 1. If the SQS send fails, nothing is written and the store is rediscovered next run. If the registry write fails after a successful send, Lambda 2 still onboards it and the registry row is completed at `sync_complete`.
- **`pending_configure` can regress to `pending_oauth`** if the connector's setup state is not CONNECTED on the configure sweep (OAuth revoked between sweeps). This is intentional — the pending_oauth handler re-issues a card.
- **Lambda 3 respects invocation budget.** It stops processing new stores when fewer than `INVOCATION_RESERVE_SECONDS` (120s) remain, logging deferred counts. Deferred stores advance on the next sweep.