# bulk-insert-targeted-metadata This script facilitates ingestion of arbitrary track-level metadata via high-concurrency processing of bespoke workflows with large input datasets. ## Architecture The system is designed around a task queue pattern, where a single task is broken down into smaller sub-tasks that are processed concurrently. By defining a task in the `/tasks` directory, the system can be extended to handle new workflows with minimal effort. ### Task Definition A task is defined by a wrapping a function or group of functions in a method that is passed (eventually) to the [`concurrency_logic::process_data_async()`](./logic/concurrency_logic.py#421) method. This method is responsible for breaking down the task into smaller sub-tasks and processing them concurrently. ### Multiprocessing / Concurrency Implemented in [`concurrency_logic::process_data_async()`](logic/concurrency_logic.py#L421) The system uses the `pebble` library to manage the concurrency of the task. The `pebble` library is a wrapper around the `multiprocessing` library that provides a more intuitive interface for managing concurrent tasks. The task itself is not the method that is passed to the `process_data_async()` method, but rather a wrapper method that is responsible for handling the responses from the tasks. This method is defined outside of the task file and is passed to the `process_data_async()` method as a parameter. ## Data Ingestion ### Smithsonion Data Ingestion This task is enabled by sending `SMITHSONIAN` as the MAP_TASK environment variable. The Smithsonian data ingestion task involves the following steps: 1. Parse the input file for track contributions. 1. Query the Graph for the track-level participants for all tracks in the input file. 1. For each row in the input file: 1. Get all track-level participants needed for the mutation. 1. Format a section of the payload out of: 1. tuid/(upc, isrc) 1. Track-level Participants 1. Send the mutation to the Graph. ### Localization ingestion workflow This task is enabled by sending `LOCALIZATION` as the MAP_TASK environment variable. The localization ingestion task involves the following steps: 1. Parse input file for UPCS, and UPC/ISRC pairs. ignoring all rows missing either field. 1. Ensure that Track or Release-level metadata is present for all UPCS. 1. Query the Graph for the full product information for all UPCs in the doc. 1. Tracks: 1. Parse the gleaned product data for all existing track localization data. 1. For each row in the doc 1. For each track, get all localization keys needed for the mutation. 1. Format a section of the payload out of: 1. tuid/(upc, isrc) 1. Track-level Participants 1. Localization keys 1. Localization values 1. Combine the existing localization data with the new localization data. 1. Send the mutation to the Graph. 1. Releases: 1. Parse the gleaned product data for all existing product localization data. 1. For each row in the doc 1. For each product, get all localization keys needed for the mutation. 1. Format a section of the payload out of: 1. Product id / upc 1. Release-level Participants 1. Localization keys 1. Localization values 1. Send the mutation to the Graph. 1. Collect and aggregate all responses and errors from the process. 1. Generate a report of the process. ## Jira Ticket - INT-2077 ### Description To better serve our integration labels that require localizations, we need to find a way to import localization data in bulk to products as there is not a method currently. Localization data can be applied at both Release and Track-Level, and will most likely come in the form of a translated track title, though it could include Version information or Artist names. For the first two labels that we’re attempting this for, Young for You (Label ID: 790384) and Wing Hang Records (Label ID: 790383), I have provided the attached template for each to fill out. This will give us the minimum requirements for either release or track-level localizations: #### Release-level At the Release-Level, it gives us: - UPC - Release Name ( \* ) - Localization Language - Localized Release Name ( \* ) #### Track-level At the Track-Level, it gives us: - UPC - Volume - Track Number - ISRC - Track Name ( \* ) - Track Version ( \* ) - Localization Language - Localized Track Name ( \* ) - Localized Track Version ( \* ) This template can be expanded to include artist name information, but it has not been requested at this point by the labels. ### Technical #### Query/Functions required: - preservation of any product participants + any existing localization - query of existing data via `track` - preservation of any track participants + any existing localization - query of existing data via `product` - mutation query for product localization via `saveTracks` (input: `languageId`, `productName`, *optional*: `deliveredVersion`) - product input: `productId` - required: preservation of existing `labelParticipantUuid` - required: a blank list of `localizations` even if there are no localization - optional: include parameter for participant `localizations` (input `languageId`, `name`) ```json { "input": { "productId": 2821160, // REQUIRED "productLocalizations": [ { "deliveredVersion": "TEST-WC", // REQUIRED "languageId": 6, // REQUIRED "productName": "百鸟引-黄权笛子独奏" // OPTIONAL } ], "participations": [ { "labelParticipantUuid": "b3a133c5-6672-40f2-9161-bf8faa51096c", // REQUIRED "role": "PRIMARY_ARTIST", // REQUIRED "localizations": [ { "name": "百鸟引-黄权笛子", // OPTIONAL "languageId": 6 // OPTIONAL } ] // REQUIRED list, even if empty } ] } } ``` - mutation query for release localization via `updateProduct`(input: `languageId`, `trackName`, *optional*: `version`) - track input: `tracks`(this is the TUID, optionally accepts an array of TUIDs) - required: preservation of existing `labelParticipantUuid` - required: a blank list of `localizations` even if there are no localization - optional: include parameter for participant `localizations` (input `languageId`, `name`) #### Technical Notes - languageId keys off of the `arts_relations.itunes_languages` table - current GQL implementation does not allow for preservation of existing artist localization - However, we can explicitly set any artist and any localization to anything - Participation is a required value in product localization input - the value `localizations` under each artist is a required entry even as a blank List, otherwise the API call will fail - Participants **cannot be changed** while localization call is in use - Despite that, an UUID (any value can be used) and a`role` is still required - The role actually sets the artistType of the localized artist - Input of an UUID that do not already exist works but will not effect any participant value change - Input of a valid UUID is required to add localization to that artist - It is possible to add new roles to existing participant - Localization of an artist appears to create another `artistId`, and for some cases `anartistInfoId`, example response: ```json "artists": [ { "artistId": "112527720", "artistInfoId": null, "artistName": "永恒金曲第一", "artistType": "ensemble", "vendorId": null }, { "artistId": "112527721", "artistInfoId": null, "artistName": "永恒金曲第12345", "artistType": "orchestra", "vendorId": null } ] ``` #### Requirements 1. Using only provided localization data, be able to: 1. set localized product name 1. set localized track name 1. Ensure preservation of existing participant data 1. name, UUID + roleId 1. Ensure that data processing encoding supports the languages used for localization 1. Lookup of Language as language ID into an integer (keys off itunes_languages table) 1. Workstation front-end localization requires a Chinese Traditional input if a Chinese Simplified input is entered (Open issue with client/label) 1. Nice to have: (Out of scope) 1. set localized product participant name 1. set localized track participant name ### Process Steps #### Workflow Graph https://drive.google.com/file/d/1SlLCcBrHtkFTBLORZYkrcaHFkQ8ZTTwM/view ### Product 1. query productId participants 1. store participant data, add localization if needed 1. mutate productId and localization data + participant data ### Track 1. query trackId participants 1. store participant data, add localization if needed 1. mutate trackId and localization data + participant data ### Roles Definitions There are distinctive sets of Product-level roles and Track-level roles; the query returns a different value than the required input to set a Role, hence the definition table below key-value pair as: {returnedValue}: {requiredInputValue} ```json productRolesDef = { "performer": "PRIMARY_ARTIST", "featuring": "FEATURED_ARTIST", "feature_to_primary": "FEATURE_TO_PRIMARY", "remixer": "REMIXER", "producer": "PRODUCER", "composer": "COMPOSER", "conductor": "CONDUCTOR", "ensemble": "ENSEMBLE", "orchestra": "ORCHESTRA", "arranger": "ARRANGER", "lyricist": "LYRICIST", "cast": "CAST", "chorus": "CHORUS", "engineer": "ENGINEER", "singer": "SINGER", "soloist": "SOLOIST" } ``` ```json trackRolesDef = { "performer": "PERFORMER", "featuring": "FEATURED_ARTIST", "feature_to_primary": "FEATURE_TO_PRIMARY", "remixer": "REMIXER", "producer": "PRODUCER", "track_writer": "WRITER", "composer": "COMPOSER", "conductor": "CONDUCTOR", "ensemble": "ENSEMBLE", "orchestra": "ORCHESTRA" } ```