# YouTube Red Marketshare

This is intended as a quick reference to the flow. For more complete info, 
see [YouTube Red Marketshare Reference on Notion](https://www.notion.so/YouTube-Red-Marketshare-f40d62785ae440c5a4a228b84b156481).

## Data Sources

The flow  relies on the following 3 staging raw tables populated by the 
**youtube_monthly** flow instead of processing the drop files from the ftp 
bucket:

* 1\. **The Orchard Music** [theorchardmusic]:
    - [red_label_music_summary](https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/flows/youtube_red_marketshare/config.py#L50),
    - [red_label_subscriber_summary](https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/flows/youtube_red_marketshare/config.py#L52),
* 2\. **The Orchard Entertainment** [dmgi]: 
    - [red_music_summary](https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/flows/youtube_red_marketshare/config.py#L47).

## Workflow Description & Execution
- The flow checks whether the required [youtube monthly reports](https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/flows/youtube_red_marketshare/config.py#L44-L53) have been ingested by [looking into the statuses](https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/flows/youtube_red_marketshare/tasks.py#L61-L86) of the ********************************youtube_monthly******************************** flow executions for the necessary reports  on DynamoDB.
- If all three reports are ingested, the flow will proceed to populating its [staging raw table](https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/flows/youtube_red_marketshare/config.py#L42). 
To do so, the flow will combine data **(UNION ALL)** from the 
  _red_music_summary_ and 
  from the join (based on the country code column) of 
  _red_label_music_summary_ and _red_label_subscriber_summary_.

There are two possible approaches for executing the flow:
- triggering [swf-youtube-red-marketshare](https://scheduler.theorchard.io/job/swf-youtube-red-marketshare-exec/) with parameters:
  - **SKIP**: the number of days to skip,
  - **DAYS**: the number of days to run the flow for (should be 1).
- triggering [prod-swf-ingestion-exec](https://scheduler.theorchard.io/job/prod-swf-ingestion-exec/) with parameters:
     - **WORKFLOW**: `youtube_red_marketshare`,
     - **CONTEXT**: `{"context_date": "2021-12-03", "reload": "True"}` 
       (using any date from a month will work).

## Troubleshooting

If the **youtube_monthly** staging raw tables aren’t populated by the 
12th/13th/14th of each month (for the previous month), the following steps 
have to be undertaken:

1. We shall look into the execution statuses of **youtube_monthly** reports 
   for the 
respective period․ If the execution statuses aren't complete, we shall 
   rerun the necessary  **youtube_monthly** for the period.
2. If the execution statuses are complete, but the data isn’t available, we 
   shall look into the **[youtube ftp bucket on S3](s3://prod-orcd-ftp/ftp/youtube/)** for the following files (sample file names for October, 2022):
   - YouTube_theorchardmusic_M_20221001_20221031_red_label_summary_v1-1.csv.gz
   - YouTube_dmgi_M_20221001_20221031_red_summary_v1-1.csv.gz
3. If the mentioned files are not available on the ftp, we shall check for 
   them on **YouTube Studio**. More about this can be found on [YouTube Red Marketshare Reference on Notion](https://www.notion.so/YouTube-Red-Marketshare-f40d62785ae440c5a4a228b84b156481).