# SLZ DDEX Downloader lambda

## Lambda should:

- be invoked by `delphi-slzDdexReleaseFlow` stepfunction

- get prepared by `slz_job_manager` lambda payload with UnitOfWork from event

- scan DDEX S3 bucket

- get a list of Releases from the root of the bucket

- get a list of processed Releases from ContentStatus table

- check for unprocessed releases

- Create record in the content_status table, skip processing if it exist and has status COMPLETE.
    * Release folder name is slz_db.public.content_status.context

- Downloading process follows our standard flow: source → quarantine bucket → decompressed bucket. Copy the whole folder from s3(xml|jpg|jpeg|png files only)

- Validate release xml file against the XSD schema

- Check if the attachments described in the release file exist in {release_ts}/resources

- All checks passed →
    * move the folder into the decompressed bucket
    * set content_status.content_status to COMPLETE
    * create gz archive of the whole release, put it into the s3://{env}-sme-data-archive/ddex/release_message/v1/report_date={YYYY-MM-DD}/report_licensor=sme/{release_ts}.gz
    * create the Acknowledgment Success message

- If checks are not passed:
    * log the error into content_status_failure table,
    * increment content_status.failure_count value
    * set content_status.content_status to FAILED
    * move the content from quarantine to corrupted bucket (cleanup if there was something already)
    * If failure_count >= 3 → create the Acknowledgment Error message with corresponding file status
    * raise validation error to fail step function

- Put Acknowledgment message into the s3://{ddex-feed}/acknowledgements/

- Mark content_status record as completed

- Assign content_status.content_name to acknowledgment file name


#### Valid Acknowledgment file content statuses:

- FileOK - release is valid

- ResourceMissing - cover image is absent in resources folder, but described in release file

- NewReleaseMessageInvalid - release file is unparseable or XML content is malformed

- MetadataMissing - empty SoundRecordingId/ISRC tag

- InternalError - for all other errors(release file is not valid)


## Lambda Environment variables
To execute the DDEX Downloader Lambda he following environment variables must be configured:

| Variable                          | Description                        |
|-----------------------------------|------------------------------------|
| `ENVIRONMENT`                     | Environment name (`prod`, `dev`)   |
| `RDS_SECRETS_KEY`                 | Path to SLZ DB secrets             |
| `FEED_BUCKET`                     | DDEX Feed bucket name              |
| `ARCHIVE_BUCKET`                  | SLZ arhive bucket                  |
| `DECOMPRESSED_BUCKET`             | SLZ decompressed bucket            |
| `CORRUPTED_BUCKET`                | SLZ corrupted bucket               |
| `DECOMPRESSED_QUARANTINE_BUCKET`  | SLZ decompressed quarantine bucket |


## Lambda Input
```json
{
    "uow_code": "ddex-20210427-sme-release_message-v1",
    "dsp": "ddex",
    "report_type": "release_message",
    "version": "v1",
    "report_date": "2021-04-27",
    "licensor": "sme",
    "config_bucket": "dev-delphi-configs",
    "job_id": "ddex-20210427-sme-release_message-v1_28231_20210427"
}
```


## Lambda Output
In case of successful lambda execution the following output will be provided:
```json
{
    "meta": {
        "status": "OK"
    }
}
```


## Developers guide

Use `pre-commit` for git hooks managing (https://pre-commit.com/#installation). Hook configuration
can be found at .pre-commit-config.yaml. `isort` and `yapf` tools are included to run as a part
of precommit hook.

To register a new hook run:
```shell script
pre-commit install
```

To run hooks without code committing:
```shell script
pre-commit run
```

Sometimes these tools can have a conflict regarding import sorting. In this case consider to
enable `include_trailing_comma` option at `setup.cfg` file.

Don’t forget to run `pylint` and `mypy` before code committing manually. It will save you from
wasting time at reading CI logs for failed build.

In order to check everything is fine (and have a dry run for formatting and import sorting)
consider following commands, e.g. for `slz_ddex_downloader`:
```shell script
cd slz_ddex_downloader
workon slz_ddex_downloader # or './venv/bin/activate'
isort --check-only -rc . && yapf --recursive --verbose --parallel --diff . \
&& pylint --rcfile=pylintrc ./${PWD##*/} ./tests && mypy ${PWD##*/} && echo 'Exit code:' $?
```

It will run all checks inside a current directory.


## Testing

- Prepare CloudWatch event
- Put it into the `events/`
- Execute lambda:

```
    python <project_name> -e <environment> -p events/<event>.json
```