# Delphi Conversion ETL
Delphi Conversion ETL is a product which provides a functionality to convert decompressed
SLZ files to another format (parquet).

Environment:
* Apache Spark `3.1.2`
* Scala `2.12.10`

# Development Process
## Code Style
Scala language has an official code style guide. Everyone, involved in the current project development
should read it: https://docs.scala-lang.org/style/. We should follow the current guide as much as possible.
It will help us to keep our code readable and well-organized.

Code styling can be verified with following command:
```
sbt scalastyle
```
To verify styling in test sources use:
```
sbt test:scalastyle
```
For convenience, it's possible to check code formatting in the whole project:
```
sbt scalastyle test:scalastyle
```

## Code Formatting
The `scalafmt` tool was configured to the project in order to introduce the same code formatting
rules for all team members involved in the development and keep all code formatted identically.
The configuration might not 100% reflect the Scala style guide due to the tool limitation, but
it covers basic rules.

#### IDE configuration
The following link provides a guide on how to configure
the IDE to use `.scalafmt.conf` as a default formatting rule: https://scalameta.org/scalafmt/docs/installation.html

#### SBT usage
Use the following link for more information about usage with SBT: https://scalameta.org/scalafmt/docs/installation.html#sbt

To check formatting in the project:
```
sbt scalafmtCheckAll
```
To format a single file:
```
sbt scalafmtOnly <file>
```
Reformat all files according to .sclafmt config:
```
sbt scalafmtAll
```

#### Jenkins
The automatic code formatting check was added to `Code Style` step during the build.
The build will be failed when the code was not formatted properly.

## Test Coverage
To run tests with enabled coverage tooling use:
```
sbt clean coverage test
```
To generate coverage report use:
```
sbt coverageReport
```

Coverage HTML and XML reports will be in `target/scala-2.12/coverage-report`.

# Branching Strategy
This project follows so-called "Feature Branching Strategy". It means, that all changes should be implemented in a
separate branch and then merged into the target branch after review and approval by other teammates.

All branches should have a Jira ticket number in their names. Additional information about the changes in that branch
is welcomed, but not obligatory.

Examples of good branch naming:
* `task/XXX-1111/readme-update`
* `bug/XXX-1111/test-aggregation`
* `XXX-1111/naming-fixes`
* `XXX-1111`

## Typical development workflow
The typical day-to-day flow includes normal changes that developers make to the code, changes that do not bring any
heightened sense of urgency.
* The branch should be crated from the `develop` branch.
* PR should be targeted and merged to the `develop` branch.
* No project version update is required (because of `SNAPSHOT`).

## RC development workflow
Release candidate changes and fixes usually contains the last changes for the version of the project which is preparing
for the production deployment and passing the QA verification on STAGE env.
* The branch should be created from the `release/*` branch.
* PR should be targeted and merged to the `release/*` branch.
* RC version update is required (e.g. `1.2.0-RC1` -> `1.2.0-RC2`).

## Emergency hotfixes
An emergency hotfix is when a particular incident or issue has been expedited to deal with some emergent situation,
normally bug fixes.
* The branch should be crated from the `master` branch.
* PR should be targeted and merged to the `master` branch.
* Version update is required (e.g. `1.2.0` -> `1.2.1`).

# Release Process
Octopus dashboard: https://octopus.delphi.zone/app#/Spaces-1/projects/delphi-etl-conversion/deployments

## Low environment deployment
The deployment to the `dev` environment is done automatically by Octopus after each PR merge.

The deployment process to the `qa` environment is not automated and should be done manually using Octopus dashboard
**before** moving the ticket to the "Ready for QA" state.

## Staging environment deployment
In order to deploy the latest version of the project into the `STAGE` environment, there is a need to follow the
following steps:
1. Create `release/X.X.0` branch from the current state of the development branch, where `X.X.0` is a current project
   version, e.g. `1.2.0`.
2. Update project version from `X.X.0-SNAPSHOT` to `X.X.0-RC1` with a separate commit to the `release/X.X.0` branch.
3. Push `release/X.X.0` branch to the remote.
4. When Jenkins build will be completed, new release will be created and ready for the deployment in the Octopus.
5. Use Octopus dashboard for deployment to the `stage delphi` environment.
6. Update the current project version in the `develop` branch: from `X.X.0-SNAPSHOT` to `X.Y.0-SNAPSHOT` by a commit to
   the `develop` branch.
7. Push the `develop` branch to the remote.

## Typical production environment deployment
Production deployment is very similar to the staging deployment process:
1. Do merge `release/X.X.X` branch to the `master` branch.
2. Update project version from `X.X.X-RCX` -> `X.X.X` with a separate commit to the `master` branch.
3. Push `master` branch to the remote.
4. When Jenkins build will be completed, new release will be created and ready for the deployment in the Octopus.
5. Use Octopus dashboard for deployment to the `stage delphi` environment.
6. Promote the version from stage to prod environment using the same Octopus dashboard.
7. Do merge `master` to the `develop`.
8. Push the `develop` branch to the remote.

## Emergency hotfix production deployment
Taking into account that usually hotfix branches are created and merged directly to the `master`, the deployment process
is slightly different and shorter than usual production deployment.
1. Use Octopus dashboard for deployment to the `stage delphi` environment.
2. Promote the version from stage to prod environment using the same Octopus dashboard.
3. Do merge `master` to the `develop`.
4. Push the `develop` branch to the remote.

# Development configuration
In order to use dev config, following java system property should be configured:
```
-Dconfig.resource=application.dev.conf
```
#### Databricks
To configure this property on Databricks this property should be added as an additional
Spark driver option:
```
spark.driver.extraJavaOptions -Dconfig.resource=application.dev.conf
```
The property should be added to the cluster configuration. Please use the following links as a hint:
- https://docs.databricks.com/clusters/configure.html#spark-configuration
- https://kb.databricks.com/clusters/conf-overwrites-default-settings.html#solution

#### IntelliJ IDEA
To configure your local Intellij IDEA the following settings should be added to run config:
- VM options: `-Dspark.master=local[*] -Dconfig.resource=application.local.conf`
- Program arguments: application CLI arguments
- Environment Variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
- Include dependencies with "Provided" scope: checked (true)

> :information_source: The `-Dconfig.resource=application.local.conf` config is an alias for the combination of the
following arguments: `-Dconfig.resource=application.dev.conf -Ddelphi.etl.log4j-config=classpath:log4j.properties`.
It's also possible to use these options directly if there is a need to connect to other environments.


# Usage
Conversion ETL is implemented as a usual Spark Application. 
It supports the following list of application arguments:
   
| Argument        | Required?          | Multiple?          | Group | Description                                           |
|-----------------|--------------------|--------------------|-------|-------------------------------------------------------|
| `--dsp`         | :white_check_mark: | :x:                |       | DSP name.                                             |
| `--report-type` | :white_check_mark: | :x:                |       | Report name.                                          |
| `--report-date` | :white_check_mark: | :x:                |       | Report date in a format `YYYY-MM-DD`.                   |
| `--sf-token`    | :x:                | :x:                |       | AWS Step Function token.                              |
| `--sf-name`     | :x:                | :x:                |       | AWS Step Function name.                               |
| `--partitions`  | :x:                | :x:                |       | Number of output partitions                           |
| `--licensor`    | :white_check_mark: | :white_check_mark: | :one: | Licensor name (not allowed to use with `--context`)   |
| `--context`     | :white_check_mark: | :white_check_mark: | :one: | Report Context (not allowed to use with `--licensor`) |

#### Entry Point
Current ETl has a single entry point. In other words, this means that despite Apps ETl there is no
need to use multiple classes to process data for multiple DSPs. The entry point class is the following:
```scala
com.sonymusic.delphi.etl.conversion.ConversionEtl
```

#### Databricks execution
From the Databricks execution there is nothing special comparing to other Delphi ETLs.

Examples of the typical input arguments for all environments:

DEV:
```json
["--class","com.sonymusic.delphi.etl.conversion.ConversionEtl","--driver-java-options","-Dconfig.resource=application.dev.conf","s3://dev-delphi-etl-artifacts/delphi-etl-conversion.jar","--dsp","youtubereporting","--report-type","active_claims","--report-date","2020-11-08","--licensor","sme"]
```

QA:
```json
["--class","com.sonymusic.delphi.etl.conversion.ConversionEtl","--driver-java-options","-Dconfig.resource=application.qa.conf","s3://qa-delphi-etl-artifacts/delphi-etl-conversion.jar","--dsp","youtubereporting","--report-type","active_claims","--report-date","2020-11-08","--licensor","sme"]
```

STAGE:
```json
["--class","com.sonymusic.delphi.etl.conversion.ConversionEtl","--driver-java-options","-Dconfig.resource=application.stage.conf","s3://stage-delphi-etl-artifacts/delphi-etl-conversion.jar","--dsp","youtubereporting","--report-type","active_claims","--report-date","2020-11-08","--licensor","sme"]
```

PROD:
```json
["--class","com.sonymusic.delphi.etl.conversion.ConversionEtl","s3://prod-delphi-etl-artifacts/delphi-etl-conversion.jar","--dsp","youtubereporting","--report-type","active_claims","--report-date","2020-11-08","--licensor","sme"]
```

> :information_source: Note, there is no need to add `--driver-java-options` argument with config file despite other envs.

# S3 Buckets
Conversion ETL uses AWS S3 as source and destination storage for data:
 - src bucket: `<env>-sme-data-decompressed`
 - dst bucket: `<env>-delphi-sme-data-parquet`
 
#### Destination Bucket Partitioning
In general, Conversion ETL will write data to `<env>-delphi-sme-data-parquet` bucket using the same 
partitioning which used in SLZ. For example:
```
s3://dev-delphi-sme-data-parquet/youtubereporting/asset_basic/a2/report_date=2020-11-07/report_licensor=sme/
``` 

When report requires contexts, additional folder with context value will be added to the folder hierarchy.
```
s3://dev-delphi-sme-data-parquet/spotify/streams/v4/report_date=2020-11-07/report_licensor=sme/context=US/
```  
> The additional folder required because unfortunately, we cannot manage the file names for spark 
output, but Apps ETL matches files by masks, and context is strongly required to be in the path.

#### Number Of Output Files

The desired number of output files may be configured manually by `--partitions` application argument.
For example:
```
--partitions 500
```

When `--partitions` option is not set and source data were read into a single partition 
(source file is zip or `multiline` option set to `true`), dataset will be automatically repartitioned 
using `spark.sql.shuffle.partitions` as number of partitions.

#### Output Partitioning
By default, all generated parquet files are partitioned in the S3 bucket by `report_date` and `report_licensor` columns.

###### Context partitioning
Conversion ETL is able to partition output files by context. The mandatory condition is passing contexts as an input
arguments for the job. 
This behaviour is activated by default, but could be disabled by the following boolean feature flag:
```
delphi.etl.conversion.feature.context-partitioning-enabled
```

To switch the value of the flag, there is a need to add the following driver java option to `spark-submit`:
```
--driver-java-options "-Ddelphi.etl.conversion.context-partitioning-enabled=true"
```