# Running Scripts ## Getting Started ### 1. Secrets Access Before running any of the scripts locally, you will need to ensure you have permission to get the following secrets via secrets manager via CLI: - `dev/lambda-jwt-refresh/jwt_token` - `dev/lambda-jwt-refresh/jwt_token_expiration` You can request access via DevOps [here](https://www.notion.so/DevOps-End-User-Support-Process-fae6dcaca43e48e28bf9f601bdaa7722). Otherwise, you won't be able to generate a JWT token needed as part of the request's `Authorization` header to connect to `ows-moneyhub` in either `QA` or `PROD`. ### 2. AWS Token Assuming you have access to this secret, generate an aws token using your `Dev` profile and set your `default` profile to use your `Dev` credentials in your `~/. aws/credentials` file. Run the following command to generate an aws token session using [awsume](https://www.notion.so/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f#5618a9119254428e9baeada414767e85) (recommended) or aws cli. `awsume` Afterwards, you can now attempt to run the script in the terminal. ## Running Locally The scripts are run using a Docker container and the script-runner.sh shell script. The docker image can be run using the following make command: ```shell make script_runner_docker ``` Note that the docker image takes the environment variables from the local system so the ones used by the various scripts need to be defined. See the descriptions of the scripts below to see which environment variables they use. ## Running in Production If you want to run the scripts in QA/prod you need to go to the following Jenkins job: [ows-moneyhub-script](https://pipeline.theorchard.io/job/ows-moneyhub-script/) Select "Build with Parameters" and from there select the environment, script, and enter any parameters. Different parameters are used in differnet scripts, so when running a script it may be useful to check the script description to see what parameters it supports (see below). ## Scripts ### backfill_vat_summary Loads data in from a CSV file and attempts to backfill the `ledger_vat_summary` table with the data. It will filter out all the entries which aren't in the provided statement period. It requires you to pass in the path to the CSV file as a CLI parameter. It uses the following environment variables: * `Environment` Environment to run the script for (qa/prod/dev). * `STATEMENT_PERIOD_ID` Which statement period to backfill for. ### create_statement_attachments Creates the invoice statement attachments and is typically done when a statement period is closed. Note that this will not trigger the lambda to actually generate the documents, but will just create the database entries. This was done to allow us to validate whether the expected reports were created before triggering the (potentially hundreds of) lambdas. To trigger the actual document generation you need to run the `trigger_attachment_generation` script. It uses the following environment variables: * `Environment` Environment to run the script for (qa/prod/dev). * `STATEMENT_PERIOD_ID` ID of the statement period to create attachments for. ### regenerate_custom_reports Regenerates the custom reports for a given statement period. It updates every report that is associated with the period and sets their status to `"in_progress"` and then triggers their lambdas to genereate the documents. It uses the following environment variables: * `Environment` Environment to run the script for (qa/prod/dev). * `STATEMENT_PERIOD_ID` ID of the statement period to regenerate reports for. ### regenerate_statement_attachments Regenerates statement attachments for a given statement period, optionally filtered by account and type. It updates every appropriate attachment and sets its status to `"in_progress"` and then triggers the lambdas that genereates the documents. It uses the following environment variables: * `Environment` Environment to run the script for (qa/prod/dev). * `STATEMENT_PERIOD_ID` ID of the statement period to regenerate attachments for. * `ACCOUNT_ID` Optional ID of the account to regenerate attachments for. * `STATEMENT_ATTACHMENT_TYPE` Optional type of attachment to regenerate. Note that `ACCOUNT_ID` and `STATEMENT_ATTACHMENT_TYPE` are filters, so if they aren't provided it will regenerate all the attachments for a given statement period. ### trigger_attachment_generation This will trigger the lambdas that generate the statement attachment documents. It will look for any attachment whose status **is not** `"complete"` and trigger the appropriate lambda for it. This can be used to generate new documents created by the `create_statement_attachments` script, or to retry attachments that previously had an error. It uses the following environment variables: * `Environment` Environment to run the script for (qa/prod/dev). * `STATEMENT_PERIOD_ID` ID of the statement period to generate attachments for. ### check_attachment_statuses This will get and post to slack all stuck custom reports and statement attachment documents. It will look for any attachment whose status **is** `"in_progress"` and `"created_by"` is over one hour old. Script logs stuck documents as well as post a message to the slack channel. It uses the following environment variables: * `Environment` Environment to run the script for (qa/prod/dev). * `SLACK_WEBHOOK_URL` Link to post the message (provided by slack webhooks). * `OWSREQUEST_SERVICE_MAP` Should be used if developer wants to connect to the local server for developing and testing. ### backfill_internal_attachments This script scans the dynamodb `{env}_statement_attachments_success` table for attachments within the given statement period range. Afterwards, it transforms the internal attachments then uploads them to the `statement_attachment` rds table. Existing backfilled internal attachments are skipped to prevent duplicates. It uses the following environment variables: * `Environment` Environment to run the script for (qa/prod). * `STATEMENT_PERIOD_ID` A single or range of statement period IDs of internal attachments to backfill. For e.g. `270-273` or just `270`. ## Specific Scenarios ### Backfilling Internal Attachments When running locally, you will need to add the following to your `.env` file in order to connect to dynamo db. ``` Environment=qa STATEMENT_PERIOD_ID=whatever-range-or-id-you-want AWS_ACCESS_KEY_ID=your-actual-tid AWS_SECRET_ACCESS_KEY=your-actual-key AWS_SESSION_TOKEN=your-actual-token ``` In your terminal, you can run `awsume prod` then run `env`. Afterwards, you can simply copy and paste your aws environment variables to your `.env` file. Then, point your MySQL environment variables to point to your local MySQL docker container. ``` MYSQL_DB_HOST=localhost MYSQL_DB_USER=royalties MYSQL_DB_PASS=1234 MYSQL_DB_NAME=royalty_accounting MYSQL_DB_PORT=6500 ``` ### Generating VAT docs When a statement period is closed, or accounting data has been set as visible to some accounts, then we will need to generate VAT docs for them. This is done in three steps: 1. Run the `create_statement_attachments` script as this will create invoices for any accounts that have data available (and don't already have invoices). 2. Verify in the DB that the attachments that you were expecting were created. 3. Run the `trigger_attachment_generation` which will trigger lambdas for the newly created attachments. Note that the `create_statement_attachments` script is idempotent which means that you can run it as many times as you want without it creating the same attachments every time. It will only create attachments where ones don't already exist, so if it has been run once for KNR it can then be run later for AWAL and once again when the statement period has been closed. #### Potential Issues * If you're running the `create_statement_attachments` job and it's not creating any attachments, it may be because the data has only been set as "visible to accounts" in production and not QA. * Some invoices can fail to be generated because of missing tax information (we get a `404` response from ows-payee). This is to be expected, especially in QA where a lot of the data hasn't been synced over from production. If an account doesn't have tax information available when the initial generation is run, but they are updated later, it is possible to re-try their failed attachment by just running the `trigger_attachment_generation` script again.