# delphi-slz

## HOW TO ONBOARD NEW DSP

1. Update templates and generate config in delphi-config repo.

    1.1. Firstly read https://github.com/filtr/delphi-config/tree/master/src/gen_config#how-to-add-a-new-config.
    
    1.2. Update dsp_config, dsp-specific-settings, dsp-complete-criteria folders: 
    add template in templates and param in params(https://github.com/filtr/delphi-config/tree/master/files)
    
    1.3. Run make gen/all to generate config files for all environments
    (https://github.com/filtr/delphi-config/blob/master/src/gen_config/Makefile#L105)

2. Add dsp to slz-downloader. Generic flow: we download archived and decompressed version 
of the report to quarantine buckets, validate them and then move to final archive and decompressed bucket.
If validation fails we move archived file to corrupted bucket. 
(https://github.com/filtr/delphi-slz/blob/develop/step/task/slz_downloader/slz_downloader/worker/generic.py)
There are some other scenarios that override base flow. Files are located at this folder 
https://github.com/filtr/delphi-slz/tree/develop/step/task/slz_downloader/slz_downloader/worker
    
    2.1. https://github.com/filtr/delphi-slz/tree/develop/step/task/slz_downloader/slz_downloader/dsp 
    Add Client for new dsp in new file. Name of file should be equal to new dsp name lowercase.
    Extend base Client or HttpResponseClient from base.py located at the same folder.(HttpResponseClient
    already contains predefined logic for availability_check: check_source_is available(). If it does not
    suits new dsp - extend base Client and define this method(ex. dsp/gras.py))
    
    2.2. Connect new client to dsp factory https://github.com/filtr/delphi-slz/blob/master/step/task/slz_downloader/slz_downloader/dsp/factory.py#L8
    
    2.3. Do not forget to write tests
    
    2.4. [optional] If new dsp needs different download flow, it could be customised here
    https://github.com/filtr/delphi-slz/blob/develop/step/task/slz_downloader/slz_downloader/worker/factory.py#L20

3. Add json validation schema

    3.1 Add schema to data-dictionary repo according to this structure https://github.com/filtr/data-dictionary#structure
    
4. Add migration for database. Update data_source and report tables.
https://github.com/filtr/delphi-slz-db-schema/tree/master/liquibase/migrations 
(ex: https://github.com/filtr/delphi-slz-db-schema/blob/master/liquibase/migrations/20200423_1434_DLP-2756_slz-add-linkfire.sql)

5. At this step you should be able to download report using slz-downloader 
console script https://github.com/filtr/delphi-slz/tree/master/step/task/slz_downloader#examples

6. Update version of slz-downloader in related packages
    
    6.1. availability_check https://github.com/filtr/delphi-slz/tree/develop/step/lambda/slz_availability_check
    
    6.2. https://github.com/filtr/delphi-slz/tree/develop/step/lambda/slz_clean_up
    
    6.3. https://github.com/filtr/delphi-slz/tree/develop/step/lambda/slz_meta_retrigger
    
7. Add name of new dsp here https://github.com/filtr/delphi-slz/blob/develop/step/common/slz_dynamodb/slz_dynamodb/const.py#L5

8. Update version of check_complete lambda https://github.com/filtr/delphi-slz/tree/develop/step/lambda/slz_check_complete
with new version of slz_dynamodb from p.7