# bulk-asset-downloader

This Dockerized script partially automates a manual dev process to download a label's catalog of assets from S3 to S3 using the bulk asset download Step Function.

## References
The Bulk Asset Download process was created to relieve the Operations Team of copying label's catalogs of assets in a very time-consuming, manual process.

* Manual Dev Process: [Notion doc](https://www.notion.so/Bulk-Asset-Download-Dev-Process-20e97177520f803199bdf764110453ab)
* [prod-workflow-tools-bulk-asset-download-sfn](https://us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn%3Aaws%3Astates%3Aus-east-1%3A437795906767%3AstateMachine%3Aprod-workflow-tools-bulk-asset-download-sfn?type=standard)
* [prod-bulk-asset-download-output](https://us-east-1.console.aws.amazon.com/s3/buckets/prod-bulk-asset-download-output?region=us-east-1&tab=objects&bucketType=general)

## Setup

1. Add environment variable 

```bash
$ cp .env.shadow .env
```
* `ENVIRONMENT` - should be set to `prod` for actual request

2. ensure [ECR access](https://www.notion.so/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f?pvs=4#7a6624c58c6642abaecd804e5f25c820) is setup
3. enable AWS access to the **prod** account using [awsume](https://www.notion.so/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f?pvs=4#beaec95488ad4473899a26a2b8cf603f)


## Linting & Formatting

### Run linting
```sh
$ docker compose run --build --rm lint-and-test
```

### Format code

```sh
$ docker compose run --build --rm format
```

## Running

```bash
$ docker compose run --build --rm download {account_id} {account_type} {asset_types(no spaces)} {username} {user_email} {jira_ticket}
```

```
usage: main.py [-h] account_id {vendor,subaccount} asset_types username user_email jira_ticket                                                                                       

positional arguments:
  account_id           OA Label ID.
  {vendor,subaccount}  Type of account: vendor or subaccount.
  asset_types          Comma-separated list of asset types to download: (e.g. wav,video_master,tif).
  username             Your username (e.g. mclark).
  user_email           Your user email (e.g. mikayla.clark@sonymusic-pde.com).
  jira_ticket          JIRA Ticket associated with the asset request (e.g. cdam-1234).

options:
  -h, --help           show this help message and exit
```

IMPORTANT: If you're re-running the script for the same request, be sure to enter `n` and enter the export ID UUID. This will ensure that the same S3 output folder is used and that files are not duplicated.
```bash
$ Do you want to generate a new export ID for this request? (y/n)
```

The script will kickoff the SFN execution. You will not be able to proceed until you confirm all details have been filled in the spreadsheet (entering `y`).
```bash
$ Please fill in the export spreadsheet with the following details while the execution runs...

EXPORT_ID: 65bbe723-603a-45eb-9f27-b70844517ca7
ACCOUNT_ID: 4427
ACCOUNT_TYPE: subaccount
ASSET_TYPES: wav, tif
EXECUTION_NAME: c66c7122-0c97-47b1-b272-3fb871cffa87
EXPORT_START_DATEe: 2025-10-02T15:19:51.557Z
USERNAME: japple
USER_EMAIL: japple@sonymusic.com
JIRA_TICKET: CDAM-1234
```

The script will poll the SFN execution status for about 15 minutes.
```bash
$ Checking execution status...
```

If the task is still running after 15 minutes, you'll have to check the status of the execution in the AWS Console.
```bash
$ This appears to be a long running task, check the AWS console later...
$ Don't forget to enter the END_DATE into the export spreadsheet when it completes.
$ Goodbye! 👋
```

If the execution succeeds, you'll be prompted to add the `ECPORT_END_DATE` to the spreadsheet.
```bash
$ Execution succeeded! 🎉 🪩
Please enter the end_date into the export spreadsheet...
EXPORT_END_DATE: 2025-10-02T15:22:11.828Z
```

If the execution fails, you'll be prompted to check the AWS Console for details.
```bash
$ Status: TIMED_OUT

$ Execution arn:aws:states:us-east-1:437795906767:execution:prod-workflow-tools-bulk-asset-download-sfn:c66c7122-0c97-47b1-b272-3fb871cffa87 did not complete successfully.
$ Please check the AWS console for more details.
```