# bulk-download-s3-assets


Download large numbers of assets from S3 and save them with human readable metadata.

### Setup

1. create `.env` file that will pass variables to container
```
$ cp .env.shadow .env
```
2. ensure [ECR access](https://www.notion.so/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f?pvs=4#7a6624c58c6642abaecd804e5f25c820) is setup
3. enable AWS access to the **prod** account using [awsume](https://www.notion.so/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f?pvs=4#beaec95488ad4473899a26a2b8cf603f)


#### Auth Using Browser

Shorter setup, slow to run script many times.

* In `.env`
  * set `SNOWFLAKE_AUTH_TYPE` to `externalbrowser`
  * set `SNOWFLAKE_USER` to your username (should be email)
* Log into Snowflake using your browser
* Follow prompts after running script

#### Auth Using Key Pair

Longer setup, fast to run script many times.

* Setup key pairs as described [here](https://www.notion.so/Snowflake-7c88cc17b0034e7db669a88fd2962bab?pvs=4#3aa649311d19425f8a3dd2e84eda8348)
* In `.env`
  * set `SNOWFLAKE_AUTH_TYPE` to `snowflake`
  * set `SNOWFLAKE_HOST_KEY_DIR` to directory on host containing private key
  * set `SNOWFLAKE_PRIVATE_KEY_FILE_PWD` to your private key's passphrase if configured else empty
  * set `SNOWFLAKE_USER` to your username (should be email)

### Running

UPC can either be a single string value or a filename in `./input` containing one UPC per line.

```
$ docker compose run --build --rm download ...
```

```
usage: -c [-h] [-w] [-f] [-t] [-m] [-j] [--workers WORKERS] upc_input

positional arguments:
  upc_input          UPC comma separated string or filename containing UPCs (one per line) in ./input

options:
  -h, --help         show this help message and exit
  -w                 Download mezzanine WAV assets (default: None)
  -f                 Download mezzanine FLAC assets (default: None)
  -t                 Download mezzanine TIF assets (default: None)
  -m                 Download mezzanine MP3 assets (default: None)
  -j                 Download mezzanine JPG assets (default: None)
  --workers WORKERS  Number of parallel workers to download with (default: 20)
```

Output is organized by UPC into `./output`