# delphi-config

## Config management

How to add a new config or update an existing one? See [here](./src/gen_config/README.md)

## Repository management

`master` is the main branch in this repository.

`versions.json` contains versions of all configs in a repository.

Version of a config should be updated accordingly to (SemVer)[https://semver.org/]

## Config description

SLZ configs:

1. `dsp_config.json` - contains description of Unit Of Work
2. `dsp-specific-settings.json`
3. `dsp-complete-criteria.json` - contains set of contexts/business units for each Unit Of Work

APPS ETL configs:

1. `apps-etl-config.json` - contains description of APPS ETL Unit Of Work and its SLZ dependencies
2. `apps-etl-versions.json` - contains schemas and versions of reports used by APPS ETL

Exploration configs:

1. `exploration-dsp-config.json` - defines which reports should be transferred from slz to exploration(everything that is not present in config will be filtered
   out from exploration flow)
2. `exploration-etl-config.json` - contains dsp specific params for databricks job splitter

### SLZ configs

#### dsp-specific-settings.json

    {
        "apple": {
            "uow_active_days_limit": 14,
            [OPT]"concurrency_limit": 5,
            "implementations": [
              {
                "rule": "*/summary",
                "label": "apple-specific"
              },
              {
                "rule": "*/*",
                "label": "apple",
                [OPT]"flow": "standard"
              }
            ],
            "days_offset": 1,
            "validation_thresholds": {
              "low": 40,
              "high": 50
            }
          },
        ...
    }

- `apple` - dsp name
- `uow_active_days_limit` - timeframe in days, how many days since created_at date UOW will
  remain active and being processed by in slz flow. When
  today-created_at > uow_active_days_limit, UoW is force completed.
  - > defined here, value is used for all UoWs for specific DSP. To override
    value for a single UoW use dsp_config.json
- `implementations` - set of rules for mapping Client(which does download process) in
    slz_downloader
- [OPTIONAL]`flow` - which flow should JM start, if omitted, 'standard' is used in JM lambda
- `days_offset` - offset for UoW creation for Job Manager(JM) lambda. If we execute JM today,
  if creates UoW for today - days_offset date
- `validation_thresholds` - thresholds for historical validation done in slz_downloader
    - > defined here, value is used for all UoWs for specific DSP. To override
        value for a single UoW use dsp_config.json
- [OPTIONAL]`concurrency_limit` the maximum number of concurrenly running UoWs set per DSP.

|Value|Number of active UoW|
|--|--|
|NA | Unlimited|
|0|No units are allowed to be in IN_PROGRESS|
|`<N:int>`| No more than `N` units can be in IN_PROGRESS|


#### dsp_config.json

    {
        {
            "unit_of_work": "spotify-{yyyymmdd}-smecharity-users-v2",
            "uow_active_hours_threshold": 36,
            "schedule": "*/10 * * * *",
            [OPT]"days_offset": 9,
            [OPT]"frequency": "0 0 * * 7",
            [OPT]"uow_active_days_limit": 1,
            [OPT]"timeframe_hours": 4,
            [OPT]"hours_offset": 2,
            [OPT]"validation_thresholds": {
                "low": 40,
                "high": 50
            },
            [OPT]"uow_complete_after_hours": 24,
            "args": {
                "dsp": "spotify",
                "extension": "json",
                "type": "users",
                [OPT]"subtype": None
                "version": "v2",
                "licensor": "smecharity",
                "valid_from": "2018-01-22",
                [OPT]"valid_until": "2019-12-31"
            }
        },
        ...
    }

- `uow_active_hours_threshold` - how many hours UoW could stuck in
ACTIVE status, before uow_monitoring_lambda start notifying
- `schedule` - how often JM lambda start SF for this UoW
- [OPTIONAL] `frequency` - when report is not daily, this defines at which
  day it becomes available on source side

  - this param allows specifying recurrence of UoW
  - type: Cron expression
  - examples:
    ```
    '0 0 */3 * *' - every 3rd day of month
    '0 0 * * 5' - every Friday (isoweekday)
    '0 0 10 * *' - every month on the 10th
    '0 0 L * *' - every last day of the month
    ```
  - > Note : set "days_offset" to 0 to have UoW created on a specific day defined in the "frequency"
- [OPTIONAL] `uow_active_days_limit` - overrides value from `dsp-specific-settings.json`
- [OPTIONAL] `timeframe_hours` - for UoWs data frame for which is less than 1 day
- [OPTIONAL] `hours_offset` - [default=0]. Delay in hours for download start. By default, when UoW is created, 
  it's scheduled for next timeslot according to `schedule` param with no delays.
  
  **Examples**: 
    
  schedule = `"*/30 * * * *"`, timeframe of UoW = `(2021-01-01 12:00:00; 2021-01-01 13:59:59)`
  1. `hours_offset = 0` - then next_run_at for Uow will be `2021-01-01 14:00:00` - next datetime slot 
     according to cron scheduler after upper bound of timeslot.
  2. `hours_offset = 2` - then next_run_at for Uow will be `2021-01-01 16:00:00` - 2 hours 
     delay after upper bound of timeslot
  

- [OPTIONAL] `validation_thresholds` - overrides value from `dsp-specific-settings.json`
- [OPTIONAL] `uow_complete_after_hours` - used for complete(not force complete)
  units that have no complete criteria and logic for min_complete/complete is not applicable for them.
  In Job manager lambda when `(uow.created_at < now() - timedelta(hours=uow_complete_after_hours)) == True`
  unit will be completed
- [OPTIONAL]  `days_offset` - overrides value from `dsp-specific-settings.json`
- `args` - unique set of params defining UoW

#### dsp-complete-criteria.json

    {
        "youtubereporting-sme-content_owner_video_metadata": {
            "partner": "youtubereporting",
            "licensor": "sme",
            "report_type": "content_owner_video_metadata",
            "complete_criteria": [
                {
                    "context": "content_owner_video_metadata_a2",
                    "required": true,
                    [OPT]"job_id": "bd15242c-0c49-473c-bed1-51a5c9afeedb"
                },
                {
                    "context": "dim_label",
                    "required": true,
                    [OPT]"query": "gras-queries/dim_label.sql"
                },
                {
                    "context": "2382::80032614",
                    "required": false
                }
            ...
            ]
          },
        ...
    }

- `dsp-licensor-report_type` - youtubereporting-sme-content_owner_video_metadata unique identifier
- `partner` dsp
- `context` - context/business unit name
- `required` - bool flag defines if context is required
  for MIN_COMPLETE

- [OPTIONAL]`job_id` - youtubereporting specific flag used in
downloading process for generation url to source

how to get job id : 
 
using aws secret delphi/stage/slz/youtubereporting 

generate access_token : 
```
AUTH_URL = 'https://www.googleapis.com/oauth2/v4/token'
creds = {delphi/stage/slz/youtubereporting  secret }
response = requests.post(url=AUTH_URL, data=creds['credentials'])
result = response.json()
print(result)
```
go to [URL](https://developers.google.com/youtube/reporting/v1/reference/rest/v1/jobs/list?apix=true&apix_params=%7B%22includeSystemManaged%22%3Atrue%2C%22onBehalfOfContentOwner%22%3A%22Ta2MYghl4Z0lHK1qcf671g%22%7D) 

insert access token into Show_standard_parameters -> access_token 
remove [Google OAuth 2.0] / [API key checkboxes] 
press Execute

- [OPTIONAL]`query` - gras specific flag, contains path to gras query


### APPS ETL configs

#### apps-etl-config.json

    {
        "unit_of_work": "youtubereporting-{yyyymmdd}-metadata",
        "schedule": "*/10 * * * *",
        "frequency": "0 0 * * 7",
        "data_source": "slz",
        "args": {
            "dsp": "youtubereporting",
            "report_type": "youtubereporting_metadata",
            "unit_type": "snapshot",
            "dependencies_type": "normalized",
            "dbx_job_id": 600,
            "dbx_job_max_lifetime": 86400,
            "dbx_spark_params": [
                "--class",
                "com.sonymusic.delphi.etl.apps.aggregation.youtube.YoutubeMetadataAggregation",
                "--driver-java-options",
                "-Dconfig.resource=application.dev.conf",
                "s3://dev-delphi-etl-artifacts/delphi-etl-apps.jar"
            ],
            "valid_from": "2020-02-15",
            "valid_until": "2021-04-20",
            "include_on_hold": false
            "days_offset": 2
        },
        "dependencies": [
            {
                "name": "youtubereporting-{yyyymmdd}-active_claims_conversion",
                "type": "etl_uow"
            },
            {
                "name": "vevo-{yyyymmdd}-sme-active_claims-v1",
                "type": "slz_uow",
                "is_single_context": true,
                "is_optional": true,
            },
            {
                "name": "V_YT_VIDEO_ISRC",
                "type": "snowflake",
                "days_offset": 2
            },
        ]
    }

- `unit_of_work` - UoW name
- `schedule` - how often apps_etl_manager checks UoW dependecies for updates
- [OPTIONAL] `frequency` - when report is not daily, this defines at which day it's being processed

  - this param allows specifying recurrence of UoW
  - type: Cron expression
  - examples:
    ```
    '0 0 */3 * *' - every 3rd day of month
    '0 0 * * 5' - every Friday (isoweekday)
    '0 0 10 * *' - every month on the 10th
    '0 0 L * *' - every last day of the month
    ```
  - > Note : set "days_offset" to 0 to have UoW created on a specific day defined in the "frequency"
- `data_source` - source of dependencies (slz|chartmetrick)

`args` - unique set of params defining UoW:
- `args.dsp` - data source platform
- `args.report_type` - type of apps etl report
- `args.unit_type` - daily|weekly|monyhly|snapshot
- `args.dbx_job_id` - id of databricks job which will be used for aggregation processing
- `args.dbx_job_max_lifetime` - timeout of databricks aggregation processing in seconds
- `args.dbx_spark_params` - specific databricks params which will be passed to spark
- `args.valid_from` - UoW valid_from date
- [OPTIONAL] `args.valid_until` -  UoW valid_until date
- [OPTIONAL] `args.include_on_hold` - define if we will check slz dependendencies in ON_HOLD status
- [OPTIONAL] `args.days_offset` - UoW creation days offset
- [OPTIONAL] `args.dependencies_type` - "normalized"(default) for joinable deps, "denormalized" for deps which desn't need to be joined

`dependencies` - list of dependent slz UoWs with params:
- `dependency.name` - dependency name - slz UoW code pattern|name of SF view etc.
- `dependency.type` - slz_uow|etl_uow|snowflake
- `dependency.is_single_context` - determines if slz UoW has only one context
- [OPTIONAL] `dependency.is_optional` - determines if we can skip a specific dependency
- [OPTIONAL] `dependency.days_offset` - days_offset of slz UoW
- [OPTIONAL] `frequency` - frequency of slz UoW

#### apps-etl-versions.json

    {
        "amazonadsupported": {
            "activity": [
                {
                    "version": "v1",
                    "valid_from": "2019-04-01",
                    "valid_until": "2020-05-31"
                },
                {
                    "version": "v1_1",
                    "valid_from": "2020-06-01",
                    "valid_until": "2021-02-18",
                    "schema_version": "v1_1_20200601"
                },
                {
                    "version": "v2",
                    "valid_from": "2021-02-19",
                    "schema_version": "v1_20210219"
                }
            ]
            ...
        }
        ...
    }

- `amazonadsupported` - dsp name
- `activity` - report type
- `version` - specific report version
- `args.valid_from` - report version valid_from date
- [OPTIONAL] `valid_until` -  report version valid_until date
- [OPTIONAL] `schema_version` -  shema name for specific version


### EXP configs

#### exploration-dsp-config.json
    [
        "amazonprime": {
            "process_all": true,
            "report_types": {}
        },
        "youtubereporting": {
            "process_all": false,
            "report_types": {
              "active_claims": {
                "report_type": "active_claims",
                "versions": [
                  "a1"
                ],
                "action": "COPY"
              },
              "content_owner_asset": {
                "report_type": "content_owner_asset",
                "versions": [
                  "a2"
                ],
                [OPT]"action": "COPY"
              }
            }
        }
    ]
- `amazonprime` - dsp name
- `process_all` - True - all reports of all versions/licensors/report_types will be processed in exp.
when False - other params should be defined
- [OPTIONAL]`action` - which flow will be used in exp to process files