# Bulk decompress lambda

Launch bulk decompression archived files to decompressed bucket

## Execution notes
1. Lambda launch decompress process getting file from any env and landing to env it belongs to, so: when
you want to copy data from prod to stage, launch lambda for STAGE env
2. When do decompress, it stores files in destination bucket under the same path it gets original file from 
source bucket.

    For example: 
    ```
   you copy file from 
   "stage-sme-data-archive" -> "stage-sme-data-decompressed"
    in this case path of file will be:
   from
   s3://stage-sme-data-archive/tiktokreporting/Trends_TopSong/v1/report_date=2020-05-10/report_licensor=sme/TikTok_Trends_TopSong_20200510.txt.gz
   to
   s3://stage-sme-data-decompressed/tiktokreporting/Trends_TopSong/v1/report_date=2020-05-10/report_licensor=sme/TikTok_Trends_TopSong_20200510.txt
   ```

3. **Lambda works in 2 modes**:
    1. Generate prefixes and start Step Functions
    2. Generate prefixes
   
## Mode 1: Generate prefixes and start Step Functions
Could be used for slz/etl files that landed in s3 buckets under path like:

{licensor}/{report_type}/{version}/report_date=YYYY-MM-DD/report_licensor={licensor}/

Payload example
```
{
    "bucket_archive": "stage-sme-data-archive", 
    "archive_key": ".+US.+",
    "bucket_decompressed": "stage-sme-data-decompressed",
    "dsp_settings_path": "s3://dev-delphi-configs/dsp-specific-settings.json",
    "force": true,
    "debug": true,
    "prefixes_payload": {
        "dsp": "spotify",
        "licensor": ".+",
        "date": "2020-02-20",
        "type": ".+",
        "version": ".+"
    },
    "dsp_config": "s3://dev-delphi-configs/dsp_config.json"
}
```
Based on `dsp_config` and `prefixes_payload` we generate list of prefixes like 
`{licensor}/{report_type}/{version}/report_date=YYYY-MM-DD/report_licensor={licensor}/`, 
get list of available files under this prefix from `bucket_archive` from s3
and apply `archive_key` regexp. If found match, start SF for this file

Each param in `prefixes_payload` is regexp

####Examples
1. Decompress all apple, spotify files for sme,theorchard for 2020-03- 15-17
    ```
    {
        "bucket_archive": "stage-sme-data-archive", 
        "archive_key": ".+",
        "bucket_decompressed": "stage-sme-data-decompressed",
        "dsp_settings_path": "s3://dev-delphi-configs/dsp-specific-settings.json",
        "force": true,
        "debug": true,
        "prefixes_payload": {
            "dsp": "(apple|spotify)",
            "licensor": "(sme|theorchard)",
            "date": "2020-03-1[5-7]",
            "type": ".+",
            "version": ".+"
        },
        "dsp_config": "s3://dev-delphi-configs/dsp_config.json"
    }
    ```
2. Decompress all apple files for 2020-03 first 9 days
    ```
    {
        "bucket_archive": "stage-sme-data-archive", 
        "archive_key": ".+",
        "bucket_decompressed": "stage-sme-data-decompressed",
        "dsp_settings_path": "s3://dev-delphi-configs/dsp-specific-settings.json",
        "force": true,
        "debug": true,
        "prefixes_payload": {
            "dsp": "apple",
            "licensor": ".+",
            "date": "2020-03-0.+",
            "type": ".+",
            "version": ".+"
        },
        "dsp_config": "s3://dev-delphi-configs/dsp_config.json"
    }
    ```
3. Decompress all apple files for 2020-03 (1 month). 

    **ATTENTION: on production we have a lot of files 
    so processing whole month will take a lot of time and as we have timeout of lambda max 15 min it could not 
    process all files at once.** Use approach from p.4
    
    ```
    {
        "bucket_archive": "stage-sme-data-archive", 
        "archive_key": ".+",
        "bucket_decompressed": "stage-sme-data-decompressed",
        "dsp_settings_path": "s3://dev-delphi-configs/dsp-specific-settings.json",
        "force": true,
        "debug": true,
        "prefixes_payload": {
            "dsp": "apple",
            "licensor": ".+",
            "date": "2020-03-.+",
            "type": ".+",
            "version": ".+"
        },
        "dsp_config": "s3://dev-delphi-configs/dsp_config.json"
    }
    ```
4. Decompress all apple files for 2020-03 (1 month). - several launches

    For 1 month split launch for 3, each time replace 1 param in payload:
        
        `"date": "2020-03-0.+",`
        `"date": "2020-03-1.+",`
        `"date": "2020-03-2.+",`
        `"date": "2020-03-3.+",`
    
    ```
    {
        "bucket_archive": "stage-sme-data-archive", 
        "archive_key": ".+",
        "bucket_decompressed": "stage-sme-data-decompressed",
        "dsp_settings_path": "s3://dev-delphi-configs/dsp-specific-settings.json",
        "force": true,
        "debug": true,
        "prefixes_payload": {
            "dsp": "apple",
            "licensor": ".+",
            "date": "2020-03-0.+",
            "type": ".+",
            "version": ".+"
        },
        "dsp_config": "s3://dev-delphi-configs/dsp_config.json"
    }
    ```
## Mode 2: Generate prefixes
the same as mode 1 but we stop execution after prefix generation(Based on `dsp_config` and `prefixes_payload`) 
and return them.

Could be used as a pre-run for mode 1 in case you are not sure it payload you set and how 
much data will be processed

####Examples
1. Generate prefixes for apple, spotify files for 2020-03-1.+ (10 days)
    ```
    {
        "bucket_archive": "stage-sme-data-archive", 
        "archive_key": ".+",
        "bucket_decompressed": "stage-sme-data-decompressed",
        "dsp_settings_path": "s3://dev-delphi-configs/dsp-specific-settings.json",
        "force": true,
        "debug": true,
        "prefixes_payload": {
            "dsp": "(apple|spotify)",
            "licensor": ".+",
            "date": "2020-03-1.+",
            "type": ".+",
            "version": ".+"
        },
        "dsp_config": "s3://dev-delphi-configs/dsp_config.json",
        "prefixes_only": True
    }
    ```
    result will be:
    
    ```
    {
      "meta": {
        "status": "OK",
        "results": [
          "apple/artistdemographics/v1_0/report_date=2020-03-10/report_licensor=sme",
          "apple/artistdemographics/v1_0/report_date=2020-03-11/report_licensor=sme",
          "apple/artistdemographics/v1_0/report_date=2020-03-12/report_licensor=sme",
          "apple/artistdemographics/v1_0/report_date=2020-03-13/report_licensor=sme",
          "apple/artistdemographics/v1_0/report_date=2020-03-14/report_licensor=sme",
            ...
          "apple/contentdemographics/v1_0/report_date=2020-03-17/report_licensor=sme",
          "apple/contentdemographics/v1_0/report_date=2020-03-18/report_licensor=sme",
            ...
          "spotify/users/v4/report_date=2020-03-19/report_licensor=theorchard"
        ]
      }
    }
    ```

## PARAMS
Required params:
1. ```"bucket_archive"``` - path to bucket with source files
2. ```"archive_key"``` - python regexp to filter necessary files
3. ```"bucket_decompressed"``` - destination bucket
4. ```"dsp_settings_path"``` - path to settings file
5. ```"prefixes_payload" + "dsp_config"```: - payload to generate list of prefixes for preselect keys from s3.
For each prefix will be done preselect of prefixes from s3 and then applied regexp "archive_key". 

Optional params:
1. ```"force"``` - flag to launch decompression in force mode, if True - 
files in destination bucket will be overridden during decompression even if they are exist.
2. ```"debug"``` - DEBUG logging level
3. ```"prefixes_only"```: - stops lambda execution after prefix generation and return list of generated prefixes