## Apps ETL Prefill
Command line tool for prefill 'apps_etl_unit_of_work' table with UoWs for ETL Backfill.

## Make commands

- `make venv` - create local virtual env(base.pip dependencies)
- `make testenv` - create local virtual env(test.pip dependencies)
- `make devenv` - create local virtual env(dev.pip dependencies)
- `make test` - run tests
- `make lint` - run pylint check
- `make build` - build package
- `make upload` - upload package on PyPI
- `make clean` - delete unnecessary files

- `make docker/login`  - login to AWS docker, aws configuration need to be set first
- `make docker/test/image` - pull image used for tests execution
- `make docker/test` - run tests in docker
- `make docker/lint` - run pylint check in docker
- `make docker/build` - build package in docker
- `make docker/upload` - upload package on PyPI from docker

- `make docker/image/build` - build Docker image
- `make docker/image/push` - push an image to a registry
- `make docker/image/pull` - pull an image from a registry

(when run `make test`, `make lint`  command, all dependencies from requirements/test will be installed.
When run `make venv` base.pip dependencies will be installed)

### configure AWS:
- to check if aws configuration was already set: `aws configure list`
- to configure `https://docs.aws.amazon.com/en_us/cli/latest/userguide/cli-chap-configure.html`

## Params:

    Required:
    --env                   dev                     Environment
    --dsp                   dsp                     Source DSP
    --mode                  init                    init/update

    Optional:
    --start_date            2017-04-12                  Start Date
    --end_date              2019-05-01                  End Date
    --excluded_dates        2018-10-20,2018-12-04       Excluded Dates
    --priority              8 (9 by default)            Priority
    --status                active(queued by default)   completeness_status
    --report_ids            4,5,6,7,8,10,11,12          List of report ids
    --validation_threshold  40                          Validation threshold in %



## Examples

Init:

    apps-etl-prefill --mode=init --env=dev --dsp=spotify --start_date=2019-11-01 --end_date=2019-12-31 --excluded_dates=2018-10-20,2018-12-04 --priority=8 --status=active --report_ids=4,5,6,7,8,10,11,12 --validation_threshold=35

Update:

    apps-etl-prefill --mode=update --env=dev --dsp=spotify --start_date=2019-11-01 --end_date=2019-12-31


Or run in aws batch:

Init:

    python3 apps_etl_prefill/run_batch.py --mode=init --env=dev --dsp=spotify --start_date=2017-05-11 --end_date=2019-12-25 --excluded_dates=2018-10-20,2018-12-04 --priority=8 --status=active

Update:

    python3 apps_etl_prefill/run_batch.py --mode=update --env=dev --dsp=spotify --start_date=2019-11-01 --end_date=2019-12-31


SQL query for single failed UoW update:

    update
       apps_etl_unit_of_work aeuow
    set
       completeness_status = 'QUEUED', next_run_at = now(), last_updated_at = now(), latest_job_id = replace(latest_job_id, split_part(latest_job_id, unit_of_work_id::text, 2), to_char(timezone('utc', now()), '_yyyymmddThh24.mi.ss'))
    where
       unit_of_work_id = 17146;


Query for update all failed backfill jobs:

    update
       apps_etl_unit_of_work aeuow
    set
       completeness_status = 'QUEUED', next_run_at = now(), last_updated_at = now(), latest_job_id = replace(latest_job_id, split_part(latest_job_id, unit_of_work_id::text, 2), to_char(timezone('utc', now()), '_yyyymmddThh24.mi.ss'))
    where
       completeness_status = 'FAILED'
       and unit_of_work_type = 'BACKFILL';