# Public Data Exporter

This service is designed:
* loading Units of Work meta information from S3;
* processing data export from DB to S3;
* updating Units of Work statuses;
* creating Audit Log event for each Unit of Work processed;

The service is designed to work in an AWS Batch environment, but local development is accepted as well.

```shell
. ./venv/bin/activate
export DB_URL=postgresql+psycopg2://localhost/etldb; ./dapd_exporter/entry.py \
  --env=dev --is-local --path='s3://dev-delphi-public-data-etl-export/meta/20201207T105446.json' \
  --bucket=dev-delphi-public-data-etl-export --db-secret-key='delphi/dev/pd_etldb/pg/user'
```

```
Available options:
    -h --help               Show this screen.
    --version               Show version.
    --env=<ENV>             Environment to run against: DEV|QA|STAGE|PROD.
    --path=<PATH>           Path to file containing export meta information.
    --bucket=<BUCKET>       S3 bucket for the application output.
    --db-secret-key=<KEY>   Path to DB Secrets.
    --is-local              Tells application it being started without AWS environment.

Usage:
    entry.py -h | --help | --version
    entry.py --env=<ENV> --path=<PATH> [--is-local]
```

## Make commands
Consider `make help` for supported commands:
```shell script
$ make help
Application: dapd_exporter

Run command:
  make <target>


  Local commands:
    venv                     create local virtual env (base.pip dependencies)
    testenv                  create local virtual env (test.pip dependencies)
    devenv                   create local virtual env (dev.pip dependencies)
    test                     run tests
    fmt                      reformat code with yapf
    fmtcheck                 dry run isort & yapf
    lint                     run pylint check
    typecheck                run mypy check
    build                    build package
    upload                   upload package on PyPI
    clean                    delete unnecessary files

  Docker commands:
    docker/login             login to AWS docker, aws configuration need to be set first
    docker/test/image        pull image used for tests execution
    docker/<local target>    universal docker wrapper for local targets
    docker/pg/start          start test pg instance
    docker/pg/stop           shutdown test pg instance
    docker/integration       run integration tests

  help                       this message
```

## Developers guide

Use `pre-commit` for git hooks managing (https://pre-commit.com/#installation). Hook configuration
can be found at .pre-commit-config.yaml. `isort` and `yapf` tools are included to run as a part
of precommit hook.

To register a new hook run:
```shell script
pre-commit install
```

To run hooks without code committing:
```shell script
pre-commit run
```

Sometimes these tools can have a conflict regarding import sorting. In this case consider
enabling `include_trailing_comma` option at `setup.cfg` file.

Don’t forget to run `pylint` and `mypy` before code committing manually. It will save you from
wasting time at reading CI logs for failed build.

In order to check everything is fine (and have a dry run for formatting and import sorting)
consider following commands, e.g. for `dapd_exporter`:
```shell script
cd dapd_exporter
make fmtcheck
make lint
make typecheck
```

It will run all checks inside a current directory.