# DAPD Exporter API

## Make commands
Consider `make help` for supported commands:
```shell script
$ make help
Application: dapd_exporter_api

Run command:
  make <target>


  Local commands:
    venv                     create local virtual env (base.pip dependencies)
    testenv                  create local virtual env (test.pip dependencies)
    devenv                   create local virtual env (dev.pip dependencies)
    test                     run tests
    fmt                      reformat code with yapf
    fmtcheck                 dry run isort & yapf
    lint                     run pylint check
    typecheck                run mypy check
    build                    build package
    upload                   upload package on PyPI
    clean                    delete unnecessary files

  Docker commands:
    docker/login             login to AWS docker, aws configuration need to be set first
    docker/test/image        pull image used for tests execution
    docker/<local target>    universal docker wrapper for local targets
    docker/pg/start          start test pg instance
    docker/pg/stop           shutdown test pg instance
    docker/integration       run integration tests

  help                       this message
```

## API examples:

#### Health check
```
curl -XGET 'http://localhost:9083/api/v1/health' -i
HTTP/1.0 204 No Content
content-type: application/json
content-length: 16
Server: Werkzeug/1.0.1 Python/3.7.7
Date: Fri, 30 Oct 2020 13:38:04 GMT
%
```

#### List of Units
```
curl -XGET 'http://localhost:9083/api/v1/uows/?limit=1&offset=2' -i
HTTP/1.0 200 OK
content-type: application/json
content-length: 284
Server: Werkzeug/1.0.1 Python/3.7.7
Date: Thu, 10 Dec 2020 18:08:58 GMT

[{"id": 429, "status": "EXPORT_COMPLETED", "project": "dapd", "fact": "fact_playlist_track_dynamics",
"dimensions": [], "version": "v1", "first_fact_id": 11801, "last_fact_id": 11900, "created_at": "2020-12-07T13:54:46.254465+03:00",
"updated_at": "2020-12-10T21:08:54.884726+03:00"}]%
```

#### Unit details
```
curl -XGET 'http://localhost:9083/api/v1/uows/429' -i
HTTP/1.0 200 OK
content-type: application/json
content-length: 282
Server: Werkzeug/1.0.1 Python/3.7.7
Date: Thu, 10 Dec 2020 18:27:13 GMT

{"id": 429, "status": "EXPORT_COMPLETED", "project": "dapd", "fact": "fact_playlist_track_dynamics", "dimensions": [],
"version": "v1", "first_fact_id": 11801, "last_fact_id": 11900, "created_at": "2020-12-07T13:54:46.254465+03:00",
"updated_at": "2020-12-10T21:08:54.884726+03:00"}%
```

#### Update mupltiple Units
```
curl -XPATCH 'http://localhost:9083/api/v1/uows/' -i -d '[{"id": 367, "status": "EXPORT_COMPLETED"}]'
HTTP/1.0 204 No Content
Server: Werkzeug/1.0.1 Python/3.7.7
Date: Thu, 10 Dec 2020 18:08:54 GMT
```

#### Update a single Unit
```
curl -XPATCH 'http://localhost:9083/api/v1/uows/367' -i -d '{"status": "EXPORT_COMPLETED"}'
HTTP/1.0 204 No Content
Server: Werkzeug/1.0.1 Python/3.7.7
Date: Thu, 10 Dec 2020 18:08:54 GMT
```

You can also update statuses asyncroniously by adding `"async": true` into the UoW's body

```
curl -XPATCH 'http://localhost:9083/api/v1/uows/367' -i -d '{"status": "EXPORT_COMPLETED", "async": true}'
curl -XPATCH 'http://localhost:9083/api/v1/uows/' -i -d '[{"id": 367, "status": "EXPORT_COMPLETED", "async": true}}]'
```

## Developers guide

Use `pre-commit` for git hooks managing (https://pre-commit.com/#installation). Hook configuration
can be found at .pre-commit-config.yaml. `isort` and `yapf` tools are included to run as a part
of precommit hook.

To register a new hook run:
```shell script
pre-commit install
```

To run hooks without code committing:
```shell script
pre-commit run
```

Sometimes these tools can have a conflict regarding import sorting. In this case consider to
enable `include_trailing_comma` option at `setup.cfg` file.

Don’t forget to run `pylint` and `mypy` before code committing manually. It will save you from
wasting time at reading CI logs for failed build.

In order to check everything is fine (and have a dry run for formatting and import sorting)
consider following commands, e.g. for `dapd_exporter_api`:
```shell script
cd dapd_exporter_api
make fmtcheck && make lint && make typechecker
```

It will run all checks inside a current directory.


## Example of env/local.env config

```
ENVIRONMENT=dev
# DB_SECRET_KEY=delphi/dev/pd_etldb/pg/user
# DB_URL=  # should be used when DB_SECRET_KEY is not set
DB_URL=postgresql+psycopg2://admin:admin@0.0.0.0:5432/etldb_test
IS_INTEGRATION_TESTING=true
```

`Huey` is used for performing backgroud tasks. SQLite is a backend. So, to do updates in DB.


## Scripts

Before starting you need to create a virtual environment

  make venv

Activate this environment

  source venv/bin/activate


To import parquet files into the local database:

  import-parquet \
    -p cm_charts_shazam -b qa-delphi-chartmetric \
    -f CHARTS_SHAZAM_TRENDING/report_date=2021-07-20/timestamp=2021-07-20T13:08:15/version=v1/shazam_fact_total_shazams.parquet \
    --table shazam_fact_total_shazams \
    -db postgres:postgres@0.0.0.0:5678/test_db

You can also use docker:

  docker run -v ~/.aws/:/app/.aws/ --rm -it --entrypoint= \
    475275892927.dkr.ecr.us-east-1.amazonaws.com/delphi/dapd-exporter-api /app/venv/bin/import-parquet \
    -p cm_charts_shazam -b qa-delphi-chartmetric \
    -f CHARTS_SHAZAM_TRENDING/report_date=2021-07-20/timestamp=2021-07-20T13:08:15/version=v1/shazam_fact_total_shazams.parquet \
    --table shazam_fact_total_shazams \
    -db postgres:postgres@apidb:5432/test_db