# SWITCH-3509 Ingest Performers (POC)

Made to run on scheduler.theorchard.io, this script will accept a structured Excel file (`.xlsx`) of track-level dataand process it through a series of track performer GRaphQL mutations.

This is an very quick and dirty tool to perform ad-hoc Performers' field ingestion using GraphQL queries.

This code is extracted from lambda-ddex-ingester.

## Process
1. Enter the needed information in the `.env` file
2. Place the data file on **S3** in the location you provided in the `.env` file
3. Run the script.

## .env
```
# Envirnoment
Environment=DEV  # QA/Prod on scheduler

# GraphQL
GRAPHQL_GATEWAY_URL=  # The URL of the graph instance to which to send queries
OA_USER=  # Must be a user with proper permissions. Recommend using OA service-user.

# AWS 
# These must be provided by Devops for scheduler deploy. (Use your own creds for local ad-hoc.)
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

# Processing keys
BUCKET=""  # The bucket name on S3 which contains the .xlsx file
KEY=""  # The full key path to the .xlsx file record on S3.
VENDOR_ID=0  # The Vendor ID with which to create the relationships 

# SUBACCOUNT_ID=0  # Subaccount NOT TESTED, NOT IMPLEMENTED. subaccount_id is simply not passed beyond initial handler.
 
DEV_CORRELATION_ID=""  # Force correlation ID for id/testing purposes.
```

## Excel File Structure
The fields are exact names. The first 9 fields are mandatory. (`Digital UPC` through `Performer 1 Main Role`)

Subsequent fields are optional, and must be fully filled out in the implied groups of three (`Type`, `Legal Name`, `Main Role`).
```
Digital UPC
Folder Name / Project Code
Product Code
Volume
Track No.
ISRC
Performer 1 Type
Performer 1 Legal Name
Performer 1 Main Role
Performer 2 Type
Performer 2 Legal Name
Performer 2 Main Role
Performer 3 Type
Performer 3 Legal Name
Performer 3 Main Role
Performer 4 Type
Performer 4 Legal Name
Performer 4 Main Role
Performer 5 Type
Performer 5 Legal Name
Performer 5 Main Role
```

Types and Roles are enumerations, explained somewhat further in [Field Mapping](#Field-Mapping)

## Logic
1. Confirm file is `.XLSX`
1. Open **S3** file stream
1. Parse file to dict of performers
1. Sift and merge data to form simple track/performer models as dicts, keyed off UPC
1. Query all tracks and get the needed keys to perform updates, instead of obliterating existing values.  (This is not fun, and these values should be included in other queries.)
    * Create label participants as needed, here.
1. Form a list of track update mutation payloads.
1. Execute mutations for all UPC's / Tracks / Performers with `saveTracks` mutation.

## Field Mapping
The values of the performer fields on the input `.xlsx` file are expected to be the same values which ingest successfully using the existing bulk uploader / VAPI input. Other values  or variations on these values will fail. The enumerations are strict.

The logic source (the `ddex-ingester`), uses a slighty different structure for some of these mapping, with small variations across both fields and values.

This structure matches (is a subset of) the current bulk-upload Excel template, which clients are familiar with. The value of working with this source Excel file structure, is in its ity and accessibility, providing a layman client experience. Many changes to the mapping fields were made to accomodate the particular `.xlsx` source file structure. 

In order to ensure parity between the expected field values, and the values governed by the constants, each mapped field's value had to be individually inspected. 

They were not the same in many cases. Case, structure, and sometimes the entire value varied between the structured `.xlsx` source, the logic source (`ddex-ingester`), and the expected values of the systems behind the GraphQL queries. 

There was some diaspora in the placement of the field constants in the logic source. This was consolidated into `role_mappings.py`

To clarify the detailed changes and obviate any complexity in comparison from the source, non-essential field values were retained in comment. 

## Original README:
This is a temp repo for the logic that will facilitate the needs of 
https://theorchard.atlassian.net/browse/SWITCH-3509

Currently contains code from:

https://github.com/theorchard/lambda-ddex-ingester/blob/master/lambda/set_track_metadata/index.py#L232

https://github.com/theorchard/lambda-ddex-ingester/blob/master/lambda/process_participants/index.py

https://github.com/theorchard/lambda-ddex-ingester/blob/master/lambda/parse_ddex/index.py