# neo4j-backup

Takes a `neo4j-admin` backup of a Neo4j cluster, uploads it to S3 (KMS-encrypted, via `s5cmd`), and can optionally refresh the QA cluster from the fresh backup.

## Deployment topology

This is **not** run by hand in production. The container is deployed as follows:

- **Image**: built from `Dockerfile` and pushed to ECR (`086679231553.dkr.ecr.us-east-1.amazonaws.com/neo4j-cluster-backup`) by `backup/Jenkinsfile` — its `Build and Push Image` stage runs `dockerToEcr` on the `master` branch, publishing both `:${GIT_COMMIT}` and `:latest`. The image bakes `neo4j-enterprise` (v5) and `s5cmd`, and runs `uv run /app/index.py`. **Deploy = pushing `:latest`**: the EC2 cron (below) pulls `neo4j-cluster-backup:latest` at run time, so a merge to `master` reaches prod on the next 04:15 run once the pipeline has published the new `:latest`.
- **Host**: an EC2 instance provisioned by Terraform in `terraform-infra/prod/neo4j_4/backup.tf` (an ASG of size 1 + launch template + Chef bootstrap). Despite the `-fargate` suffix on `backup_service_name`, this runs on **EC2**, not Fargate — the Fargate task variables in that Terraform are stale leftovers.
- **Configuration**: the instance is configured by the Chef cookbook `chef-repo/cookbooks/irrigate-neo4j`. The recipe `recipes/backup.rb` writes `/usr/local/sbin/neo4j-backup.sh` and registers a **cron at 04:15** that runs the container detached:
  ```
  docker run -d --rm \
    -v /var/lib/neo4j/v5:/mnt \
    -e NEO4J_BACKUP_FROM_FOLLOWER=false \
    -e NEO4J_SERVER_MAJOR_VERSION=5 \
    -e NEO4J_BACKUP_DATABASE='graph.db' \
    -e NEO4J_BACKUP_DIRECTORY=/mnt/ \
    -e NEO4J_BACKUP_DIRECTORY_CLEAN=true \
    -e NEO4J_BACKUP_REFRESH_QA=true \
    ... neo4j-cluster-backup:latest
  ```
  > `neo4j_backup_wrapper.sh.erb` in the cookbook is a **stale, unused** invocation path (deprecated `ecr get-login`, sets env vars the code no longer reads). The live path is `backup.rb`.

## Local development

Tooling is `uv` + `ruff` (there is no Makefile; the root README's `make env`/`python3.6` instructions are wrong for this module).

```shell
cp .env.shadow .env          # then fill in NEO4J_HOST, S3 bucket, KMS key id, etc.
uv sync --locked

# Lint (matches CI):
./scripts/unit-lint.sh       # ruff check + ruff format --check + typos

# Run the backup entrypoint:
uv run index.py

# Refresh QA from the latest S3 backup (typer CLI; --apply performs the drop/recreate):
uv run refresh_neo4j.py <s3-prefix> --apply
```

Set `config.DRY_RUN = True` (in `config.py`) to log the intended `neo4j-admin` / `s5cmd` commands without executing them.

## How it works

`index.py` (`main()`):
1. Emits DataDog `backup_attempt`.
2. Resolves the backup host — from the cluster's follower node (`NEO4J_BACKUP_FROM_FOLLOWER=true`) or `NEO4J_HOST` directly.
3. `run_backup_process` — builds a v4 or v5 `neo4j-admin` command (selected by `NEO4J_SERVER_MAJOR_VERSION`) and runs it. `clean_backup_dir` first removes stale `*.backup` files when `NEO4J_BACKUP_DIRECTORY_CLEAN=true`.
4. `inspect_backup_metadata` — **best-effort, v5 only.** Runs `neo4j-admin backup inspect --latest-backup --show-metadata --format=JSON` over the backup directory and captures `lastTxId` (the `highestTransaction` of the latest backup) plus the rest of the inspect record. `--show-metadata` is required — without it inspect only reports the file uri, not the transaction ids. Any failure (v4, subprocess error, unparseable output) logs and returns `None` so it never fails the backup.
5. `upload_backup_to_s3` — `s5cmd sync` the `.backup` files to `s3://{bucket}/{prefix}/{date}/` with KMS SSE, then writes a `backup-complete.txt` marker. When metadata was collected, it also writes and uploads `backup-metadata.json` (the full inspect record, incl. `lastTxId`) alongside the backup for the Neo4j Aura import tooling. The metadata sidecar upload is best-effort (a failure logs but does not fail the run).
6. If `NEO4J_BACKUP_REFRESH_QA=true`, calls `refresh_qa_backup` (see below).
7. Emits DataDog `backup_success`.

`refresh_neo4j.py` — finds the latest `.backup` in S3, then on the QA server drops non-protected `graph.*` databases and recreates one via `CREATE DATABASE ... TOPOLOGY 3 PRIMARIES ... seedURI`. Cypher identifiers are backtick-escaped and URLs single-quote-escaped. `graph.db` (the alias) and the current alias target are never dropped.

### Neo4j 4 vs 5

`NEO4J_SERVER_MAJOR_VERSION` (4 or 5) selects entirely different `neo4j-admin` command shapes **and** S3 upload globs. Any change to backup commands must update **both** branches in `run_backup_process` and `upload_backup_to_s3`. v5 also requires `--additional-config=/app/additional.conf` (SSL policy for the backup port; mounted from `/etc/neo4j/default` on the host).

The `inspect_backup_metadata` step is **v5 only** — the v4 `neo4j-admin` inspect surface differs and is unverified here, so it is skipped on v4 (logged, no-op). Because prod runs `NEO4J_SERVER_MAJOR_VERSION=5`, the `backup-metadata.json` sidecar is produced in prod.

## Configuration

All config is read at **import time** in `config.py` from env vars, with secrets pulled from AWS Secrets Manager (`${env}/${service_name}/SECRET_NAME`) — importing the module (including in tests) therefore requires valid AWS credentials or mocking, and invalid config raises immediately.

Key env vars (see `config.py` for defaults and the full list): `Environment`, `NEO4J_HOST`, `NEO4J_SERVER_MAJOR_VERSION`, `NEO4J_BACKUP_DATABASE`, `NEO4J_BACKUP_DIRECTORY`, `NEO4J_BACKUP_DIRECTORY_CLEAN`, `NEO4J_BACKUP_FROM_FOLLOWER`, `NEO4J_BACKUP_S3_BUCKET`, `NEO4J_BACKUP_KMS_KEY_ID`, `NEO4J_BACKUP_S3_PREFIX`, `NEO4J_BACKUP_UPLOAD_TO_S3`, `NEO4J_BACKUP_REFRESH_QA`, `NEO4J_BACKUP_TIMEOUT`, `NEO4J_BACKUP_INSPECT_TIMEOUT`, `NEO4J_BACKUP_COMPRESS_BACKUP`, `LOGGER_DSN`.

## Observability & alerting

- **DataDog** count metrics: `backup_attempt`, `backup_success`, `backup_failure`, `backup_upload_failure` — emitted under metric namespace `{SERVICE_NAME}` = `neo4j-cluster-backup-fargate`.
- **Alerting** is a `datadog_monitor` in Terraform. Note the monitor currently defined lives in `terraform-infra/prod/neo4j/` (the **v5** cluster) and queries the `neo4j_cluster_backup.*` namespace — see the "Known issues" below regarding the namespace mismatch and the absence of a monitor in `neo4j_4/`.

## Known issues / gotchas

These are current, unresolved problems found during review — read before changing this module:

1. **Sentry is a no-op.** The code calls `sentry_sdk.capture_exception()` on failures, but `config.py` never defines `SENTRY_DSN` and `index.py` never calls `sentry_sdk.init()` (unlike the other modules in this repo). The Chef cookbook also never sets `SENTRY_DSN`. Exceptions are **not** reported to Sentry. Fix: add `SENTRY_DSN = os.environ.get('SENTRY_DSN')` to `config.py` and `sentry_sdk.init(config.SENTRY_DSN)` at the top of `index.py`, and set the DSN in the cookbook.
2. **Failed backups can be invisible.** The cron runs `docker run -d ... >/dev/null 2>&1`, discarding all container output. With Sentry dead (above), the only failure signal is the DataDog metric — see #3.
3. **Metric/monitor namespace mismatch.** Code emits `neo4j-cluster-backup-fargate.backup_success`; the only backup monitor (in `terraform-infra/prod/neo4j/`) queries `neo4j_cluster_backup.backup_success`. Verify in DataDog which name actually lands. There is **no** backup monitor in `terraform-infra/prod/neo4j_4/` at all, even though this host backs up the v4 cluster.
4. **QA refresh is destructive and runs automatically.** Because the cron sets `NEO4J_BACKUP_REFRESH_QA=true`, every successful prod backup calls `refresh_neo4j_backup(apply=True)`, which **drops and recreates** `graph.*` databases on QA. Note the asymmetry: the automated path defaults `apply=True`, while the human-facing `typer` CLI defaults `apply=False`.
5. **QA refresh failure masks backup success.** If `refresh_qa_backup` raises, `main()` never reaches the `backup_success` metric — so a fully successful backup+upload gets reported as a failure. Consider decoupling the refresh from the backup success signal.
6. **Success metric emitted even when upload is skipped.** If `NEO4J_BACKUP_UPLOAD_TO_S3=false`, `main()` still emits `backup_success` — hiding the fact that nothing durable was produced.
7. **Dead code / minor:** the `result.returncode != 0` checks after `subprocess.run(..., check=True)` can never fire (a non-zero exit already raised), so stderr is never surfaced on the success path. `NEO4J_BACKUP_PORT` defaults to an `int` but env override yields a `str` (works only because it's f-string interpolated). `get_follower_node_host` indexes `follower_nodes[0]` with no guard (not exercised in prod, where `FROM_FOLLOWER=false`).