# bulk-upload-feeder
This tool is a software version of an automatic document feeder for the 
Bulk Upload tool.

This tool allows for the rapid parallel ingestion fully-populated Bulk 
Upload Templates. Any Bulk Upload Template file which could be processed 
manually can similarly be queued for processing (along with others). 



There is currently a bug in the parallel ingestion method which leads to
a race condition, causing multiple artist rows to be created when the
artist has not been previously created. This issue must be fixed in the 
associated lambda, 
[lambda-artist-info-sync](https://github.com/theorchard/lambda-artist-info-sync).

## Environment:
### Bare Minimum (S3-based):
```
export ENVIRONMENT=dev (prod, dev, qa, test)

export AR_MYSQL_USER=
export AR_MYSQL_PASSWORD=
export AR_MYSQL_HOST=<squad>.db.devorch.com
export AR_MYSQL_DATABASE=art_relations

export BULK_UPLOAD_API_URL=vapi.<user>.<squad>.devorch.com

export SNOWFLAKE_ACCOUNT=orchard
export SNOWFLAKE_USER=
export SNOWFLAKE_ROLE=DEV_ENGINEERING
export SNOWFLAKE_PASSWORD=
export SNOWFLAKE_WAREHOUSE=DEV_OWS_WAREHOUSE
export SNOWFLAKE_DATABASE=
export SNOWFLAKE_SCHEMA=(e.g. DEV, PROD, etc.)
export SNOWFLAKE_KEY_PASSPHRASE= (password to decrypt rsa key for access without 2FA)

export S3_BUCKET=<bucket_name>
export AWS_ACCESS_KEY_ID_DEV=<key>
export AWS_SECRET_ACCESS_KEY_DEV=<secret>
```

### Table-Based Mandatory Fields:
If S3 operation should be disabled, 

```
export S3_INGEST=False
export S3_OUTPUT=False

export SNOWFLAKE_SOURCE_TABLE=<table_name>
```

### Optional fields:
```
export RESUME_SESSION_ID=<optional if resuming>
export NUM_THREADS=< (1 <= n <= SANE_VALUE LIKE 20) >

# If the file formats change. The defaults work for standard CSV.
export SNOWFLAKE_FILE_FORMAT=<input snowflake file format name>
export SNOWFLAKE_FILE_FORMAT_OUTPUT=<output snowflake format name>
```

### MYSQL RDS vs Snowflake Logging
All logging was originally written to Snowflake to allow for joining output to
```
# Output reporting must be disabled at the present time, as the RDS and Snowflake
#  tables are not federated in any fashion, and cannot be joined.
export S3_OUTPUT=False
# Set to true to use an RDS
export USE_RDS_FOR_LOGGING = False

# Required if true
export RDS_MYSQL_USER=
export RDS_MYSQL_PASSWORD=
export RDS_MYSQL_HOST=
export RDS_MYSQL_DB=bulk_upload

```

The full list of config vars (with expanded definitions) is available in 
`config.py`, in the comments.

### String Literal Constants
Certain strings are abstracted as constants, and certain string literal
constants are used to process logic.

For example, the strings stored in `constants/artist_fields.py` are used 
to control the pre-processing and embargoing of artists to prevent race 
conditions. More information about artist embargoing is illustrated 
below.

## Artist Embargoing
Due to an extant bug in 
`https://github.com/theorchard/lambda-artist-info-sync`
there is a race condition that happens under certain conditions:

 * When executing multiple bulk-upload requests containing releases from
   a new artist both asynchronously and in close proximity.
  
 * When the number of close-to-concurrent requests containing new 
   artists modulo (%) 9 is equal to 1
   
These issues are mitigated by the pre-processing of all input datasets 
to ensure that a sole (random) release for each new artist is 
prioritized ahead of all remaining releases for each new artist.

The list of fields to check for new artists is defined by two constants 
contained in `constants/artist_fields.py.` One general list for all 
tracks/releases, and a second which extends the general list with 
additional fields to search for classical genre tracks/releases. (i.e. 
composer, conductor, etc.)

## How to Run
<https://scheduler.theorchard.io/job/bulk-upload-feeder>

## How to Demo
Example metadata set (Snowflake table):
- `PHONOFILE.DEV.PHONOFILE_METADATA_INGESTOR_DEV_TEST`

Setup/connect dev hosts for all endpoints:
- art_relations
- vapi

## Outputs
- Ingestion Logs:
  - These tables contain logs of the per-project / per-product
  insertion state. Separated by Environment (dev, prod, qa)
  - DEV
    - `INGESTION_RELEASE_LOG_DEV`
    - `INGESTION_TRACK_LOG_DEV`
  - QA
    - `INGESTION_RELEASE_LOG_QA`
    - `INGESTION_TRACK_LOG_QA`
  - PROD
    - `INGESTION_RELEASE_LOG_PROD`
    - `INGESTION_TRACK_LOG_PROD`
- Output Files (S3 only)
  - Reports that reflect the result of an ingestion operation are stored on S3 
  in the same bucket as the input bucket ``in a folder determined by the 
  `output_key` value present in `constants/s3.py`

- New DB Rows are inserted into `art_relations`.