# sanitise-rds-data

Lambda for sanitising RDS database data.

## PostgreSQL role passwords

When sanitising a PostgreSQL database the lambda copies the source roles into
the restored target. RDS will not expose the source password hashes (the master
user is `rds_superuser`, which cannot read `pg_authid`/`pg_shadow`), so login
roles would otherwise be created without a usable password. To give them their
intended dev/QA passwords, the lambda reads them from AWS Secrets Manager.

### Secret naming convention

The secret name is built as:

```
{ROLE_PASSWORDS_SECRET_NAME_PREFIX}{target_db_name}/postgres-user-passwords
```

for example `qa/sanitise-rds-data/qa-songwhip/postgres-user-passwords`.

* `ROLE_PASSWORDS_SECRET_NAME_PREFIX` is an environment variable. Include the
  target database name in the path (as above) so one deployment that sanitises
  several databases in the same account can keep a separate secret per database.
* Set the prefix to the lambda's own `{env}/{service_name}/` namespace so the
  `terraform-lambda`/`terraform-fargate` module's auto-attached IAM policy
  (which grants access to `{env}/{service_name}/*`) covers the secret without an
  extra statement.
* The prefix is **optional**. When `ROLE_PASSWORDS_SECRET_NAME_PREFIX` is unset
  (e.g. MySQL-only accounts, which never need it) no secret is read and roles
  are left passwordless. The same passwordless fallback applies if the named
  secret does not exist.

### Secret contents

The `SecretString` is a JSON object mapping role name to password:

```json
{
  "app_login": "...",
  "app_owner": "..."
}
```

A password is applied to a role only when it has a matching key, on both new
roles (`CREATE ROLE`) and existing ones (`ALTER ROLE`); applying it on `ALTER`
intentionally overwrites the prod password carried over in the restored
snapshot. A role with no entry keeps the existing behaviour: a new role is left
passwordless and an existing role keeps its current password. Roles still
needing a password can be handled by a per-database `users.sql` sanitisation
script (see `scripts/README.md`).

## Setup
* Install [docker](https://docs.docker.com/get-docker/)
* Clone this repository
* Navigate to the lambda you want to work on in `lambda/`

## Running

### Start Container
```
$ docker-compose up --build -d function
```

### Execute Function
Use the HTTP client of your choice. The body of the request is the `event` passed into the function.

```
curl --request POST \
  --url http://localhost:9000/2015-03-31/functions/function/invocations \
  --header 'Content-Type: application/json' \
  --data @tests/sample_event.json
```

### Load Changes
The `src` directory and `config.py` are mounted into the container via the `docker-compose.yaml` configuration. This means you don't need to re-build the entire container to see changes. **However** the container will need to be restarted. This process should be as quick as restarting uwsgi when making changes to a web application.
```
$ docker-compose restart function
```
:zap: You should add this command to a post-write hook in your preferred editor to automatically restart.

## Linting and Testing

### Run Container
```
$ docker-compose up --build lint-and-test
```

### Set pytest Options
Sometimes you may want to customize how tests run using [pytest options](https://docs.pytest.org/en/stable/usage.html) when doing development locally, only running a subset of tests using `-k` for example. Simply edit `TEST_ARGS` value in `docker-compose.yaml`, **without** quotes, to set the options you need.

:warning: Commited changes will modify how the build pipeline runs tests

### Set pytest-cov Options
By default test coverage is calculated for files in `./src` and displayed in a table after tests run.
* If you want to see a detailed output, edit `COV_REPORT` value in `docker-compose.yaml` to be `html` and visit `http://localhost:8000/`
* If you want to turn off coverage report generation, edit `COV_REPORT` value in `docker-compose.yaml` to be `off`

:warning: Commited changes will modify how the build pipeline runs tests, **do not** commit `COV_REPORT=html`

### Skip Linting
When developing locally, you might not want to worry about linting when working to make tests pass. Simple edit the `SKIP_LINT` value in `docker-compose.yaml` to be `1`.

:warning: Careful not to commit changes unless you **really** want to not run linting on the build pipeline

### Capture Exit Code
Unless the container itself crashes, or is killed, the exit code will be `0`. When the exit code of `lint-and-test.sh` script matters, a pull-request builder for example, run tests with the following flags.
```
$ docker-compose up --exit-code-from lint-and-test --abort-on-container-exit --build lint-and-test
```
### Datadog Configuration

In order to get traces and metrics (in addition to logs) from the [Datadog Lambda Library](https://docs.datadoghq.com/serverless/installation/python/?tab=containerimage), you'll need to have the following environment variables:
ENVIRONMENT
DD_LAMBDA_HANDLER (your lambda's original handler, e.g. "src.app.handler")
These can be set via terraform, for example [here](https://github.com/theorchard/terraform-infra/blob/master/qa/lambda-sound-recordings/lambda_sr_add_version.tf#L84-L91).
Notice, the linked example in terraform-infra is utilizing our terraform-lambda module, which does a lot of the setup work under the hood.