# ows-payee Payee management service for The Orchard (owner: `tax-and-payments`). ## Requirements - Python 3.13 - Docker - uv ## Getting Started Install `uv` if missing. See [Installing uv](https://docs.astral.sh/uv/getting-started/installation/) for details. - Homebrew: ```brew install uv``` - Curl: ```curl -LsSf https://astral.sh/uv/install.sh | sh``` Running locally pulls images from two AWS ECR registries, so you'll need AWS session credentials for the QA/PROD account: - `make local_start_db` pulls the `royalties-liquibase-runner` image from account `437795906767`. - Docker builds (`make local_integration_stack`, the CI targets) pull the `docker-parent-images` base from account `086679231553`. Generate AWS session creds (example using `awsume`): ```sh awsume prod ``` If you have the [amazon-ecr-credential-helper](https://github.com/awslabs/amazon-ecr-credential-helper) set up (see the [AWS Access](https://app.notion.com/p/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f) doc), `awsume prod` is all you need: docker authenticates to both registries automatically and you can skip the manual logins below. Otherwise, log docker in manually. `make docker_login` covers the build registry `086679231553`: ```sh make docker_login ``` If the `royalties-liquibase-runner` image isn't already cached locally, also log into the registry that hosts it (`437795906767`): ```sh aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin 437795906767.dkr.ecr.us-east-1.amazonaws.com ``` For the PDF generation logic to work locally you need to have "wkhtmltopdf" installed. If "wkhtmltopdf" is missing you will see this error: > OSError: No wkhtmltopdf executable found: "b''" To install "wkhtmltopdf" follow the instructions here: https://github.com/JazzCore/python-pdfkit/wiki/Installing-wkhtmltopdf ### Run the service locally There are two ways to run ows-payee locally: **1. Dev server on the host (recommended for development).** Runs `dev.py` via uv on port `5000` with debugging enabled, while MySQL and DynamoDB run in containers. ```bash # Copy the env template and fill in values to match the local containers cp .env.shadow .env ``` Set the MySQL connection vars in `.env` to match the local `payee-mysql` container (see [docker-compose.yml](./docker-compose.yml) and [docker-compose.local.yml](./docker-compose.local.yml)): ```sh MYSQL_DB_HOST=localhost MYSQL_DB_PORT=6155 MYSQL_DB_USER=royalties MYSQL_DB_PASS=1234 MYSQL_DB_NAME=royalty_accounting DYNAMODB_URL=http://localhost:6157 ``` ```bash # Install the local virtualenv (also run automatically by dev/lint/test) make dev_env # Start local mysql and dynamodb, then run liquibase migrations make local_start_db # Start the dev server at http://localhost:5000 make dev ``` **2. Full service in Docker.** Builds and runs the `payee-ows-payee` container (reads `.env`, exposed on port `6151`): ```bash make local_start_db make local_integration_stack ``` ### Local ports | Service | Host port | Container port | |------------------------|-----------|----------------| | dev server (`make dev`)| 5000 | 5000 | | ows-payee (Docker) | 6151 | 8080 | | MySQL | 6155 | 3306 | | DynamoDB | 6157 | 8000 | --- ## `make` commands Common dev tasks are automated via `make` commands. For example, to fully reinstall your local environment and then run tests, run: `make clean test` ### Common tasks | Command | Description | |--------------------------|----------------------------------------------------------------------------------------| | `make dev` | Start a development server at [http://localhost:5000](http://localhost:5000) | | `make test` | Run unit & functional tests and print a coverage report | | `make format` | Format code | | `make lint` | Lint code | | `make dev_env` | Create a local virtualenv (with dev deps). Automatically called by `dev`, `lint`, & `test` | | `make clean` | Delete local dependencies and generated files | | `make local_start_db` | Start local mysql and dynamodb containers and run liquibase migrations | | `make local_integration_stack` | Build and run the full ows-payee service container locally (port 6151) | | `make docker_down` | Stop local containers | > Attention Windows Users > > `make` is a linux cli tool. You can attempt to use the [WSL](https://docs.microsoft.com/en-us/windows/wsl/about), or install [GNUWin](http://gnuwin32.sourceforge.net/packages/make.htm) to get this functionality. --- ## Secure Document Manager The secure document manager wraps the [AWS dynamodb encryption SDK](https://aws-dynamodb-encryption-python.readthedocs.io/en/latest/). It consists of a [SecureDocumentManager](./payee/connectors/secure_data/manager.py) class and [SecureDocument](./payee/connectors/secure_data/document.py) class that can be used to define the schema of a secure document and persist encrypted versioned copies of those documents in dynamodb. Extend the SecureDocument class to define a document schema to store. Example ```python import boto3 from payee.connectors.secure_data.fields.text_field import TextField from payee.connectors.secure_data.document import SecureDocument from payee.connectors.secure_data.manager import get_kms_cmp from payee.connectors.secure_data.manager import get_local_cmp from payee.connectors.secure_data.manager import SecureDocumentManager class DiaryEntry(SecureDocument): """A private diary entry.""" _document_type = 'diary_entry' _fields = { 'body': TextField('body', encrypt=True), 'name': TextField('date', encrypt=False, required=True) } def main(): """Entrypoint.""" diary_entry = DiaryEntry( { 'body': 'it was a good day', 'name': 'Ice Cube' } ) sdm = _get_sdm_() sdm.save_item("Person", 2, diary_entry) def _get_sdm_(): """Get SDM.""" # Get a local cryptographic materials provider cmp = get_local_cmp() # or get a kms based cmp cmp = get_kms_cmp('some-key-id') # Get a dynamodb table resource table = boto3.resource('dynamodb').Table('table-name') return SecureDocumentManager(table, cmp) ``` ### Cryptographic Materials Providers (CMPs) Encrypting and decrypting uses keys managed by a CMP. The secure document manager supports a local CMP and an AWS KMS managed CMP. The **local CMP** uses temporary keys that are discarded along with the secure document manager instance. This means that dynamodb items saved using the dev server cannot be decrypted after restarting the server. You can initialize a local CMP using the helper function `get_local_cmp()`. The **KMS CMP** uses the supplied KMS Key id or alias to encrypt and decrypt items. As long as that key is accessible, items can be encrypted or decrypted. To use this option locally, first ensure you have access to the key, set the `KMS_KEY_ID` env var, and generate AWS session credentials. You can initialize a KMS CMP using the helper function `get_kms_cmp('key id')` > When using a key alias, prefix the alias with `alias/`. Example for key alias "foo": `alias/foo` When the flask server starts, it will store a CMP in the SDM configuration which can be reused for all subsequent request handling. ### Mocking DynamoDB Locally The DynamoDB table can either be an AWS resource or a local container. By default, local development will use the containerized table. To use an AWS DynamoDB table, set the `DYNAMODB_TABLE_NAME` env var. Ensure you have access to put & get items and generate AWS session creds before running. > Note that the DynamoDB table isn't created when the container comes up. Tests > automatically create and clear the table as needed. When running the local server, > you'll need to manually create the table using the definition in [conftest.py](./tests/conftest.py) --- ## Dependencies ### Adding new packages - Regular packages: ```uv add ``` - Dev packages: ```uv add --dev``` - Integration packages: ```uv add --group integration``` ### Upgrading packages - All packages: ```uv sync --upgrade``` - A specific package: ```uv sync --upgrade-package ``` --- ## Other make commands | Command | Description | |--------------------------------------------|-----------------------------------------------------------------------------| | `env` | install the runtime virtualenv (no dev/integration groups) | | `start_db` | start mysql/dynamodb and run liquibase using the base compose file only | | `integration_stack` | build and run the ows-payee service container (base compose file) | | `local_integration_stack` | build and run the ows-payee service container with the local compose overlay | | `drop_unit_db` & `drop_integration_db` | reset mysql state between jenkins runs | | `test_unit` | run unit tests and output xml coverage + junit report files | | `test_integration` | run the integration pytest suite | | `ci_unit_lint` & `ci_unit_lint_clean` | run / tear down the dockerized unit + lint job (also used for the PR job) | | `ci_test_integration` & `ci_test_integration_clean` | run / tear down the dockerized integration test job | | `docker_test` | run the full dockerized unit-lint and integration jobs end to end | | `docker_login` | log docker into the AWS ECR build registry (`086679231553`) | | `openapi` | regenerate the OpenAPI spec; include the generated file in the pull request | ### Linting Ruff with a custom config is being used for formatting and linting the code.