# delphi-apps-etl
Delphi Apps ETL

Environment:
* Apache Spark `3.1.2`
* Scala `2.12.10`

## Development Process
### Code Style
Scala language has an official code style guide. Everyone, involved in the current project development 
should read it: https://docs.scala-lang.org/style/. We should follow the current guide as much as possible. 
It will help us to keep our code readable and well-organized.

Code styling can be verified with following command:
```
sbt scalastyle
```
To verify styling in test sources use:
```
sbt test:scalastyle
```
For convenience, it's possible to check code formatting in the whole project:
```
sbt scalastyle test:scalastyle
```

### Code Formatting
The `scalafmt` tool was configured to the project in order to introduce the same code formatting
rules for all team members involved in the development and keep all code formatted identically.
The configuration might not 100% reflect the Scala style guide due to the tool limitation, but
it covers basic rules. 

##### IDE configuration
The following link provides a guide on how to configure 
the IDE to use `.scalafmt.conf` as a default formatting rule: https://scalameta.org/scalafmt/docs/installation.html

##### SBT usage
Use the following link for more information about usage with SBT: https://scalameta.org/scalafmt/docs/installation.html#sbt

To check formatting in the project:
```
sbt scalafmtCheckAll
```
To format a single file:
```
sbt scalafmtOnly <file>
```
Reformat all files according to .sclafmt config:
```
sbt scalafmtAll
```

##### Jenkins
The automatic code formatting check was added to `Code Style` step during the build. 
The build will be failed when the code was not formatted properly.

### Test Coverage
To run tests with enabled coverage tooling use: 
```
sbt clean coverage test
```
To generate coverage report use: 
```
sbt coverageReport
```

Coverage HTML and XML reports will be in `target/scala-2.12/coverage-report`.

## Branching Strategy
This project follows so-called "Feature Branching Strategy". It means, that all changes should be implemented in a
separate branch and then merged into the target branch after review and approval by other teammates.

All branches should have a Jira ticket number in their names. Additional information about the changes in that branch 
is welcomed, but not obligatory.

Examples of good branch naming:
* `task/XXX-1111/readme-update`
* `bug/XXX-1111/test-aggregation`
* `XXX-1111/naming-fixes`
* `XXX-1111`

### Typical development workflow 
The typical day-to-day flow includes normal changes that developers make to the code, changes that do not bring any 
heightened sense of urgency.
* The branch should be crated from the `develop` branch.
* PR should be targeted and merged to the `develop` branch.
* No project version update is required (because of `SNAPSHOT`).

### RC development workflow
Release candidate changes and fixes usually contains the last changes for the version of the project which is preparing 
for the production deployment and passing the QA verification on STAGE env. 
* The branch should be created from the `release/*` branch.
* PR should be targeted and merged to the `release/*` branch.
* RC version update is required (e.g. `1.2.0-RC1` -> `1.2.0-RC2`).

### Emergency hotfixes
An emergency hotfix is when a particular incident or issue has been expedited to deal with some emergent situation, 
normally bug fixes.
* The branch should be crated from the `master` branch.
* PR should be targeted and merged to the `master` branch.
* Version update is required (e.g. `1.2.0` -> `1.2.1`).

### Merge any branch to develop
From time to time we face with a need to apply some changes from any existing branch to the `develop` branch. This may
be a need to apply some hotfix from the `master` or `release/*` branch. The following steps describe how to do that
trick:
1. Make a new branch on top of the `develop` with the name `[source]-to-develop`. For example: `release-to-develop`
   or `master-to-develop`.
2. Do git merge from source branch (for example `release/X.X.X` or `master`) to the `[source]-to-develop` locally.
   Resolve conflicts if any.
3. Merge `[source]-to-develop` to the `develop` using a PR.

## Release Process
Octopus dashboard: https://octopus.delphi.zone/app#/Spaces-1/projects/delphi-etl-apps/deployments

### Low environment deployment
The deployment to the `dev` environment is done automatically by Octopus after each PR merge.

The deployment process to the `qa` environment is not automated and should be done manually using Octopus dashboard
**before** moving the ticket to the "Ready for QA" state.

### Staging environment deployment
In order to deploy the latest version of the project into the `STAGE` environment, there is a need to follow the
following steps:
1. Create `release/X.X.0` branch from the current state of the development branch, where `X.X.0` is a current project
   version, e.g. `1.2.0`.
2. Update project version from `X.X.0-SNAPSHOT` to `X.X.0-RC1` with a separate commit to the `release/X.X.0` branch.
3. Push `release/X.X.0` branch to the remote.
4. When Jenkins build will be completed, new release will be created and ready for the deployment in the Octopus.
5. Use Octopus dashboard for deployment to the `stage delphi` environment.
6. Update the current project version in the `develop` branch: from `X.X.0-SNAPSHOT` to `X.Y.0-SNAPSHOT`:
   1. Create a new branch `update-version`.
   2. Commit version update from `X.X.0-SNAPSHOT` to `X.Y.0-SNAPSHOT`.
   3.  Merge `update-version` to `develop` using a PR.

### Typical production environment deployment
Production deployment is very similar to the staging deployment process:
1. Do merge `release/X.X.X` branch to the `master` branch:
   1. Create new `release-to-master` branch on top of the `master`.
   2. Do git merge from `release/X.X.X` to `release-to-master` locally. Resolve conflicts if any.
   3. Update project version from `X.X.X-RCX` -> `X.X.X` with a separate commit to the `release-to-master` branch.
   4. Merge `release-to-master` to `master` via PR.
2. When Jenkins build will be completed, new release will be created and ready for the deployment in the Octopus.
3. Use Octopus dashboard for deployment to the `stage delphi` environment.
4. Promote the version from stage to prod environment using the same Octopus dashboard.
5. Do merge `master` to the `develop`:
   1. Create new `master-to-develop` branch on top of the `develop`.
   2. Do git merge from `masger` to `develop` locally. Resolve conflicts if any.
   3. Merge `master-to-develop` to `develop` using a PR.

### Emergency hotfix production deployment
Taking into account that usually hotfix branches are created and merged directly to the `master`, the deployment process
is slightly different and shorter than usual production deployment. 
1. Use Octopus dashboard for deployment to the `stage delphi` environment. 
2. Promote the version from stage to prod environment using the same Octopus dashboard.
3. Do merge `master` to the `develop`
   1. Create new `master-to-develop` branch on top of the `develop`.
   2. Do git merge from `masger` to `develop` locally. Resolve conflicts if any.
   3. Merge `master-to-develop` to `develop` using a PR.

## Development configuration
In order to use dev config, following java system property should be configured:
```
-Dconfig.resource=application.dev.conf
```
##### Databricks
To configure this property on Databricks this property should be added as an additional 
Spark driver option:
```
spark.driver.extraJavaOptions -Dconfig.resource=application.dev.conf
```
The property should be added to the cluster configuration. Please use the following links as a hint:
- https://docs.databricks.com/clusters/configure.html#spark-configuration
- https://kb.databricks.com/clusters/conf-overwrites-default-settings.html#solution

##### IntelliJ IDEA
To configure your local Intellij IDEA the following settings should be added to run config:
- VM options: `-Dspark.master=local[*] -Dconfig.resource=application.local.conf`
- Program arguments: application CLI arguments
- Environment Variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
- Include dependencies with "Provided" scope: checked (true)

> :information_source: The `-Dconfig.resource=application.local.conf` config is an alias for the combination of the 
following arguments: `-Dconfig.resource=application.dev.conf -Ddelphi.etl.log4j-config=classpath:log4j.properties`.
It's also possible to use these options directly if there is a need to connect to other environments.

## CLI arguments
| Argument        | Required?          | Multiple?          | Group | Description                                                                 |
| --------------- | ------------------ | ------------------ | ----- | ----------------------------------------------------------------------------|
| `--dsp`         | :white_check_mark: | :x:                |       | DSP to process                                                              |
| `--report-date` | :white_check_mark: | :x:                |       | Start date                                                                  |
| `--context`     | :white_check_mark: | :white_check_mark: | :one: | Context in format "licensor:context" or "licensor:[context1,context2,...]"  |
| `--licensor`    | :white_check_mark: | :white_check_mark: | :one: | Licensor name                                                               | 
| `--sf-token`    | :x:                | :x:                |       | AWS StepFunction token                                                      | 
| `--sf-name`     | :x:                | :x:                |       | AWS StepFunction name                                                       |

> :information_source: **Note:** Parameters in the same group are mutually exclusive.

#### For aggregations with contexts:
```bash
--dsp spotify --report-date 2019-10-20 --context theorchard:ad --context sme:ag --context sme:al
``` 
or
```bash
--dsp spotify --report-date 2019-10-20 --context theorchard:ad sme:ag sme:al
```
or
```bash
--dsp spotify --report-date 2019-10-20 --context theorchard:ad sme:[ag,al]
```


#### For aggregations without context
```bash
--dsp spotify --report-date 2019-10-20 --licensor theorchard --licensor sme
``` 
or
```bash
--dsp spotify --report-date 2019-10-20 --licensor theorchard sme
``` 

## Logging
Unfortunately, `log4j` is the only one alternative as a logging technology because Spark is strongly depends
on it.

### Logging Configuration

#### `src/main/resources/log4j.properties`
This is a usual `log4j 1.x` config for the current project. All third-party loggers must be configured here.

#### `src/main/resources/log4j.dbx.properties`
This config file is a copy of the Databrick's config. It should be used on Databricks. This file contains
modifications for writing logs into file in JSON format.
>Note. No changes should be done to this file. If there is a need to enable/disable any loggers, it should 
>be done in `src/main/resources/log4j.properties`. All logger configs will be appended to `log4j.dbx.properties`
>by SBT during the build.

### Databricks Logging
In order to use custom `log4j` config file, the following additional java option need to be configured: 
```
-Dlog4j.configuration=classpath:log4j.dbx.properties
```
Also, it is possible set custom log4j config in the `application.conf` file with a property: 
```
delphi.etl.apps.log4j-config
```
Absolute file path is supported as well as classpath. 

### Message Diagnostic Context
| MDC key                   | Mandatory           | Description                                   |
| ------------------------- | ------------------ | --------------------------------------------- |
| `delphi.etl.apps.env`     | :white_check_mark: | Current environment name                      |
| `delphi.etl.apps.traceId` | :x:                | Random UUID for trace logs from the same job  |
| `delphi.etl.apps.sfName`  | :x:                | SF sfName passed via `--sf-name` CLI argument | 
| `delphi.etl.dsp`          | :white_check_mark: | Current dsp name                              |
| `delphi.etl.appName`      | :white_check_mark: | Aggregation's main class simple name          |
| `delphi.etl.reportDate`   | :white_check_mark: | Raw file's report date                        |

##### JSON Logging Examples

Info:
```json
{
    "@timestamp": "2020-02-10T14:15:11.940Z",
    "source_host": "0210-141246-flub11-10-163-233-140",
    "level": "INFO",
    "thread_name": "main",
    "@version": 1,
    "logger_name": "com.sonymusic.delphi.etl.apps.aggregation.amazon.AmazonActivityAggregation$",
    "message": "Job AmazonActivityAggregation$ started with args [--dsp, amazonprime, --report-date, 2020-01-10, --context, sme:AT, --sf-name, test-sf-name]",
    "mdc": {
        "delphi.etl.apps.env": "dev",
        "delphi.etl.apps.traceId": "7ebe0ae8-bdd7-46e1-a5c0-348273928c57"
    }
}
```

Warn:
```json
{
    "@timestamp": "2020-02-10T12:22:34.414Z",
    "source_host": "0210-122006-junco10-10-163-231-123",
    "level": "WARN",
    "thread_name": "main",
    "@version": 1,
    "logger_name": "com.databricks.s3a.S3AFileSystem",
    "message": "Access denied: dev-sme-data-decompressed",
    "mdc": {
        "delphi.etl.apps.sfName": "test-sf-name",
        "delphi.etl.apps.env": "dev",
        "delphi.etl.apps.traceId": "237dbc56-f16b-4622-a653-c8e06be7974c"
    }
}
```

Error:
```json
{
    "exception": {
        "stacktrace": "java.lang.reflect.UndeclaredThrowableException\n\t ... 24 more",
        "exception_class": "java.lang.reflect.UndeclaredThrowableException"
    },
    "@timestamp": "2020-02-10T12:22:34.496Z",
    "source_host": "0210-122006-junco10-10-163-231-123",
    "level": "ERROR",
    "thread_name": "main",
    "@version": 1,
    "logger_name": "com.sonymusic.delphi.etl.apps.aggregation.amazon.AmazonActivityAggregation$",
    "message": "Job AmazonActivityAggregation$ failed. Args: [--dsp, amazonprime, --report-date, 2020-01-10, --context, sme:AT, --sf-name, test-sf-name]",
    "mdc": {
        "delphi.etl.apps.sfName": "test-sf-name",
        "delphi.etl.apps.env": "dev",
        "delphi.etl.apps.traceId": "237dbc56-f16b-4622-a653-c8e06be7974c"
    }
}
```

## Aggregations

### Amazon

Package: `com.sonymusic.delphi.etl.apps.aggregation.amazon`

| Simple Class Name                                    | Contexts           | CLI Arguments Examples                                                                                                                                        |
| ---------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AmazonArtistAggregation`                            | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `AmazonArtistPlaylistAggregation`                    | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |     
| `AmazonIsrcAggregation`                              | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |      
| `AmazonIsrcPlaylistAggregation`                      | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |     
| `AmazonPlaylistDateDayAggregation`                   | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |     
| `AmazonProductAggregation`                           | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |     
| `AmazonProductFamilyStreamsDaily`                    | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |     
| `AmazonProjectStreamsDaily`                          | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |     
| `AmazonTrackAggregation`                             | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |      
| `AmazonTrackPlaylistAggregation`                     | :white_check_mark: | `--dsp amazonprime --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp amazonprime --report-date 2021-12-12 --context xxx:[XX,YY]` |     

### Apollo

Package: `com.sonymusic.delphi.etl.apps.aggregation.apollo`

| Simple Class Name                                    | Contexts           | CLI Arguments Examples                                       |
| ---------------------------------------------------- | ------------------ | ------------------------------------------------------------ |
| `ApolloDailyAppleAggregation`                        | :white_check_mark: | `--dsp apollo --report-date 2021-12-12 --context xxx:XX`     |
| `ApolloDailySpotifyAggregation`                      | :white_check_mark: | `--dsp apollo --report-date 2021-12-12 --context xxx:XX`     |
| `ApolloPlaylistPositionsAppleSnapshotAggregation`    | :white_check_mark: | `--dsp apollo --report-date 2021-12-12 --context xxx:XX`     |
| `ApolloPlaylistPositionsSpotifySnapshotAggregation`  | :white_check_mark: | `--dsp apollo --report-date 2021-12-12 --context xxx:XX`     |

### Apple

Package: `com.sonymusic.delphi.etl.apps.aggregation.apple`

| Simple Class Name                                    | Contexts           | CLI Arguments Examples                                                                                                                            |
| ---------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AppleProductFamilyDemographicsDaily`                | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` | 
| `AppleProductFamilyStreamsDaily`                     | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` | 
| `AppleProjectDemographicsDaily`                      | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` | 
| `AppleProjectStreamsDaily`                           | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` | 
| `ArtistDateDayAggregation`                           | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` | 
| `ArtistDatePlaylistDayAggregation`                   | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `DemographicsArtistDateDayAggregation`               | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `DemographicsIsrcDateDayAggregation`                 | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `DemographicsProductDateDayAggregation`              | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` | 
| `DemographicsTrackDateDayAggregation`                | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `IsrcDateDayAggregation`                             | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `IsrcDatePlaylistDayAggregation`                     | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` | 
| `PlaylistDateDayAggregation`                         | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `PlaylistTrackDateDayAggregation`                    | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `ProductDateDayAggregation`                          | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `TrackDateDayAggregation`                            | :white_check_mark: | `--dsp apple --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp apple --report-date 2021-12-12 --context xxx:[XX,YY]` |

### Gras

Package: `com.sonymusic.delphi.etl.apps.aggregation.gras`

| Simple Class Name                                    | Contexts           | CLI Arguments Examples                                |
| ---------------------------------------------------- | ------------------ | ----------------------------------------------------- |
| `GrasAggregation`                                    | :x:                | `--dsp gras --report-date 2021-12-12 --licensor:xxx`

### Spotify

Package: `com.sonymusic.delphi.etl.apps.aggregation.spotify`

| Simple Class Name                        | Contexts           | CLI Arguments Examples                                                                                                                                |
|------------------------------------------| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SpotifyArtistDailyStreams`              | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyArtistDemographicsDailyStreams`  | :x:                | `--dsp spotify --report-date 2021-12-12 --licensor:xxx --licensor:yyy` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --licensor:[xxx,yyy]`      |
| `SpotifyArtistPlaylistDailyStreams`      | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyIsrcDailyStreams`                | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyIsrcDemographicsDailyStreams`    | :x:                | `--dsp spotify --report-date 2021-12-12 --licensor:xxx --licensor:yyy` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --licensor:[xxx,yyy]`      |
| `SpotifyIsrcPlaylistDailyStreams`        | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyPlaylistDailyStreams`            | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyProductDailyStreams`             | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyProductDemographicsDailyStreams` | :x:                | `--dsp spotify --report-date 2021-12-12 --licensor:xxx --licensor:yyy` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --licensor:[xxx,yyy]`      |
| `SpotifyProductFamilyDemographicsDaily`  | :x:                | `--dsp spotify --report-date 2021-12-12 --licensor:xxx --licensor:yyy` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --licensor:[xxx,yyy]`      |
| `SpotifyProductFamilyStreamsDaily`       | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyProjectDemographicsDaily`        | :x:                | `--dsp spotify --report-date 2021-12-12 --licensor:xxx --licensor:yyy` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --licensor:[xxx,yyy]`      |
| `SpotifyProjectStreamsDaily`             | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyTrackDailyStreams`               | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyTrackDemographicsDailyStreams`   | :x:                | `--dsp spotify --report-date 2021-12-12 --licensor:xxx --licensor:yyy` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --licensor:[xxx,yyy]`      |
| `SpotifyTrackPlaylistDailyStreams`       | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyIsrcNewListenersDaily`           | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyTrackPlaylistNewListenersDaily`  | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |
| `SpotifyIsrcPlaylistNewListenersDaily`   | :white_check_mark: | `--dsp spotify --report-date 2021-12-12 --context xxx:XX --context yyy:YY` <br/> <br/> `--dsp spotify --report-date 2021-12-12 --context xxx:[XX,YY]` |

### Youtube

Package: `com.sonymusic.delphi.etl.apps.aggregation.youtube`

| Simple Class Name                                    | Contexts           | CLI Arguments Examples                                              |
| ---------------------------------------------------- | ------------------ | ------------------------------------------------------------------- |
| `YoutubeIsrcDemographicsAggregation`                 | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `YoutubeIsrcViewsAggregation`                        | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `YoutubeProductFamilyDemographicsAggregation`        | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `YoutubeProductFamilyViewsAggregation`               | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `YoutubeProjectDemographicsAggregation`              | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `YoutubeProjectViewsAggregation`                     | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `YoutubeVideoIdDemographicsAggregation`              | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `YoutubeVideoIdViewsAggregation`                     | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `VevoMetadataAggregation`                            | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |
| `YoutubeMetadataAggregation`                         | :x:                | `--dsp youtubereporting --report-date 2021-12-12 --licensor:xxx`    |

### TikTok

Package: `com.sonymusic.delphi.etl.apps.aggregation.tiktok`

| Simple Class Name                                    | Contexts           | CLI Arguments Examples                                                                                                                                                |
| ---------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `TiktokIsrcContentTypeAggregationDaily`              | :x:                | `--dsp tiktokreporting --report-date 2021-12-12 --licensor:xxx --licensor:yyy` <br/> <br/> `--dsp tiktokreporting --report-date 2021-12-12 --licensor:[xxx,yyy]`      |