# Public Data Exporter Lambda

This Lambda function is designed for:
* creating Unit Of Work instances for data offload managing;
* storing meta information to S3 bucket;
* invoking batch computation for data offload;
* creating Audit Log event for each Unit of Work created;

Lambda payload is following:
```json
{
  "fact": "fact_playlist_track_dynamics",
  "dimensions": ["dim_track", "dim_playlist"],
  "project": "dapd",
  "priority": "medium",
  "chunk_size": 100,
  "max_iterations": 100
}
```

Note that `fact` is an ETL model to offload related to `dimensions`. Depending on `priority` value a
proper AWS Batch job is selected, available options are: `high`, `medium` and `low`. `chunk_size` defines
the amount of facts that can be processed in a single Unit of Work.

## Developers guide

Use `pre-commit` for git hooks managing (https://pre-commit.com/#installation). Hook configuration
can be found at .pre-commit-config.yaml. `isort` and `yapf` tools are included to run as a part
of precommit hook.

To register a new hook run:
```shell script
pre-commit install
```

To run hooks without code committing:
```shell script
pre-commit run
```

Sometimes these tools can have a conflict regarding import sorting. In this case consider enabling
`include_trailing_comma` option at `setup.cfg` file.

Don’t forget to run `pylint` and `mypy` before code committing manually. It will save you from
wasting time at reading CI logs for failed build.

In order to check everything is fine (and have a dry run for formatting and import sorting)
consider following commands, e.g. for `dapd_exporter_lambda`:
```shell script
cd dapd_exporter_lambda
make fmtcheck
make lint
make typecheck
```

It will run all checks inside a current directory.


## Local execution

Prerequisites:

- create event you want to pass into the lambda
- put this event into `./events/<my-event>.json, they are different for DAPD, Decibel or Linkfire


Activate virtualenv

  . venv/bin/activate

Execute lambda

  python dapd_exporter_lambda -e qa -p ./events/decibel.json

  usage: dapd_exporter_lambda [-h] -e ENV -p EVENT_PATH
  dapd_exporter_lambda: error: the following arguments are required: -e/--env, -p/--event-path


Keep in mind that for Decibel and Linkfire projects messge is coming from SQS and this is S3 onPut:* type of event