## direct_delivery ### Running Workers Locally #### Setup ##### 1. Github Token To install some private packages, you need to be able to authenticate with GitHub. * Create GitHub access token [here](https://github.com/settings/tokens) with all **repo** scope only. * Set auth token as environment variable ```bash export GITHUB_AUTH='{"github-oauth": {"github.com": ""}}' ``` ##### 2. Database Credentials * Create `.env` file ```bash cp .env.shadow .env ``` * Enter **qa** values in `.env` for * DD_USER * DD_PASS * AR_DB_USERNAME * AR_DB_PASSWORD :bangbang: Make sure your user has access to run stored procedures on the QA direct delivery database, if not, add that `GRANT` [here](https://github.com/theorchard/python-rds-utils/blob/master/lambda/sanitise_rds_data/scripts/qa-direct-delivery-cluster/users.sql). ##### 3. AWS Credentials * Setup [awsume](https://www.notion.so/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f#5618a9119254428e9baeada414767e85) and activate a profile that has access to the prod account using MFA. * Install [ecr-credentials-helper](https://www.notion.so/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f#ad5d3a529a7d41b299daa604c517ebf2) to automatically use your configured AWS credentials when pulling down ECR images. #### Execute Use a helper script to set up a job and automatically orchestrate infrastructure. Workers will automatically restart when `*.php` files change. ```bash docker compose run --rm --build cli {command} {args} ``` ```bash usage: cli.py encode [-h] [-a] [-d] [--priority {1}] [--dms_id DMS_ID] upc positional arguments: upc product UPC to create job with => https://www.notion.so/Encoding-Workers-4614a8e498034b4984eba275c3709c21?pvs=4#4a9f2a8ff4d04661a8dd5f37fdc67229 options: -h, --help show this help message and exit -a package assets in addition to metadata (default: False) -d deliver package after encoding (default: False) --priority {1} dpm priority of job (default: 1) --dms_id DMS_ID dms store id to create job with (default: 1896) ``` ```bash usage: cli.py deliver [-h] eqd_id positional arguments: eqd_id encoding_queue_detail_id to deliver options: -h, --help show this help message and exit ``` #### Logging :mag: Tail worker logs to see what they're doing. ```bash docker compose logs {encode|deliver} ``` ### PHP Debugger 1. Attach to already running container to start debugging sessoion ```bash docker compose exec -it {service_name} bash -c './dbgpClient' ``` 2. Set a breakpoint in php code, app with restart automatically - [reference](https://xdebug.org/docs/all_functions#xdebug_break) ```php xdebug_break(); ``` :bangbang: debugging client must be active or else breakpoints will be skipped 3. Hit breakpoint then enter commands - [reference](https://xdebug.org/docs/dbgpClient#usage) ```bash Xdebug Simple DBGp client (0.6.1) Copyright 2019-2024 by Derick Rethans Waiting for debug server to connect on port 9003. Connect from [::1]:33628 DBGp/1.0: Xdebug 3.4.2 — For PHP 8.2.25 (cmd) context_get 4 | context_get 4 | uninitialized $aws_params 4 | uninitialized $dd_db_conn 4 | uninitialized $dd_job 4 | uninitialized $dd_jobs 4 | uninitialized $encodeQueueUrl 4 | uninitialized $encoding_queue_detail_id 4 | uninitialized $job_info 4 | uninitialized $message 4 | uninitialized $net_watcher 4 | int $pid: 27 4 | uninitialized $queue_name_pattern 4 | uninitialized $response 4 | uninitialized $result 4 | uninitialized $row 4 | uninitialized $sql 4 | bool $x: true 4 | object $this(Orchard\Vector\Encoding\Manager) { 4 | object webserviceClient(Orchard\Vector\Utils\WebserviceClient) { 4 | } 4 | } ``` ### Binaries Debugging 1. Startup the base image without the application installed ```bash docker compose run --rm --build base ``` 2. Add files you want to test with into `./tmp` on the host machine, they will be accessible in the container at `/tmp` 3. Run commands on the container's prompt ### Update composer.lock After adding or updating php packages, you need to re-generate `composer.lock`. ```bash docker compose run --rm --build update_composer_lockfile ``` ### Unit Tests + Linting :warning: If your build hangs after the entrypoint, `ant` is probably taking up all of your CPU resources and stalling. This can happen more often when `$TEST_FILTER` is set and phpunit tries to filter. Increase the CPU your host machine allows docker to use (on a Mac, configure "Resources" in Docker Desktop app). Try at least `6` CPUs for a balance of reliability and speed. * Build and run ```bash docker compose run --rm --build unit-lint ``` :zap: Optionally run tests faster when developing locally * Set `TEST_FILTER` environment variable in `docker-compose.yaml` to apply `--filter` to `phpunit` * Set `SKIP_LINT` environment variable in `docker-compose.yaml` to `true` * Set `SKIP_COVERAGE` environment variable in `docker-compose.yaml` to `true` :warning: The PHP Code Sniffer (phpcs) is currently disabled. It can take a long time to run, especially on large codebases. If you want to skip it, set `SKIP_LINT` environment variable in `docker-compose.yaml` to `true`. To run PHP Code Sniffer (phpcs) manually. This will check all files for style issues. ```bash php ./vendor/bin/phpcs --standard=PSR12 --extensions=php src tests/unit encode_tracks.php deliver_tracks.php ``` #### Fix (fixable) Style Issues Automatically To automatically fix style issues, you can run the `format` service. This will apply fixes to all files that can be fixed automatically. ```bash docker compose run --rm --build format ``` :warning: The PHP Code Sniffer fixer (phpcbf) cannot fix all issues. You should review what has been fixed before committing the changes. You should run phpcs again after running phpcbf to see what is left to fix. ### Local Stack Run encode and deliver against local infrastructure (local MySQL, Moto S3/SQS, local SFTP) without touching the QA database. #### Available test UPCs | UPC | Title | Tracks | DMS stores | |---|---|---|---| | 191018253790 | Do Not Delete: Xml Offer Type Test | 8 | 1223, 1810, 1896 | | 196292947182 | Perros Veganos | 10 | 1223, 1810, 1896 | | 886788019584 | The Voice Of Freedom | 15 | 1223, 1810, 1896 | | 886788731288 | Super Mario Bros. Theme (Instrumental Version) | 1 | 1223, 1810, 1896 | | 5054526973572 | Duelling Yodellers | 19 | 1223, 1810, 1896 | > :warning: The UPC must exist in QA. Even though the local DB is seeded with the above UPCs, XML generation (ows-metadata) and raw asset fetching still hit QA infrastructure. If the UPC is missing from QA, encoding will fail. > :information_source: **Future improvement:** replace the QA art_relations product lookup with an OWS API call so we can validate QA product existence without requiring QA DB credentials. #### Adding a new UPC 1. Confirm the UPC has complete assets in QA S3 2. Run the TrackInfo query from `localdev/data_seeds/art_relations/` against QA art_relations and append the result rows to `seed.sql` 3. Export the relevant `dms_delivery_spec` and `dms_encoding_profile` rows for the target DMS stores and append to `localdev/data_seeds/direct_delivery/seed.sql` 4. Add the UPC to the table above with its title, track count, and supported DMS stores 5. Run `export LOCAL_STACK=true && docker compose run --rm --build cli encode {upc} --local-stack` to verify #### Running ```bash export LOCAL_STACK=true docker compose run --rm --build cli encode {upc} --local-stack docker compose run --rm --build cli encode {upc} -d --local-stack docker compose run --rm --build cli deliver {eqd_id} --local-stack ``` `LOCAL_STACK=true` must be exported in your shell before running — Docker Compose resolves it for the encode/deliver worker containers at startup, so an inline prefix on `docker compose run` is not sufficient. The `--local-stack` flag automatically seeds the local databases on each run, swaps all DB connections to local infrastructure, and keeps QA untouched. > :information_source: To start with a clean database between runs, use `docker compose down -v` before bringing the stack up again. The test DB container uses an anonymous volume that persists across normal `docker compose down`. ### Integration Tests * Ensure `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optionally `AWS_SESSION_TOKEN` are exported in your shell (e.g. via `awsume`). * Build and run ``` $ docker compose run --rm --build integration ``` > **Note:** Tests that call OWS services (e.g. `WebserviceClientIntegrationTest`) require the assumed role to have `secretsmanager:GetSecretValue` access to the M2M JWT secrets. If your credentials lack this permission the tests will fail with a credentials error — these are typically only runnable in CI where the correct role is available. `VectorOrderFlowTest` is skipped when `RUN_E2E_TESTS=true` (set automatically by Jenkins when the E2E stage is enabled). To simulate that behaviour locally: ``` $ RUN_E2E_TESTS=true docker compose run --rm --build integration ```