# Exploration Results Process lambda

It processes results from either `exp_archive_lambda` or Split and Compress DBX job.

## Lambda Environment variables
To execute Lambda the following environment variables must be configured:

| Variable                    | Description
|-----------------------------|-----------------------------------------------------------
| `ENVIRONMENT`               | Environment name (`prod`, `dev`)
| `RDS_SECRETS_KEY`           | Path to PG database credentials in AWS Secret Manager
| `SNOWFLAKE_SECRETS_KEY`     | Path to Snowflake database credentials in AWS Secret Manager


## Lambda Output
In case of successful lambda execution the following output will be provided:
```json
{
    "meta": {
        "status": "OK",
        "sf_status": "LOADED",
        "slz_status": "COMPLETED"
    }
}
```

In case of minor errors during lambda execution the following output will be provided:
```json
{
    "meta": {
        "status": "OK",
        "sf_status": "FAILED",
        "slz_status": "FAILED"
    }
}
```

If any kind of exception occurred, the response will look like the following:
```json
{
    "meta": {
        "status": "FAILURE",
        "message": <exception message>
    }
}
```

## Parameters for lambda start

```json
{
  "name": "gras-20200810-sme-dim_track_participant-v1-rp20200812T0_7500_2020-08-14T20.20.10",
  "uow_id": "gras-20200810-sme-dim_track_participant-v1-rp20200812T081003",
  "context": "dim_track_participant",
  "disassemble_content_status_id": 7500,
  "content_name": "gras_20200810_dim_track_participant.csv",
  "source_path": "s3://dev-sme-data-decompressed/gras/dim_track_participant/v1/report_date=2020-08-10/report_licensor=sme/gras_20200810_dim_track_participant.csv",
  "dest_bucket": "dev-delphi-exploration",
  "use_databricks": true,
  "trace_id": "4e68b5b9-ca6c-42e1-b516-29eaf6c29faf",
  "results": {
    "databricks": {
      "start": {
        "status": "ok",
        "splitoutput": {
          "total_split_chunks_number": 11,
          "total_file_row_count": 7361579
        }
      },
      "sync": {
        "meta": {
          "status": "OK",
          "dbx_run_state": "SUCCESS"
        }
      }
    }
  }
}
```

## Developers guide

Use `pre-commit` for git hooks managing (https://pre-commit.com/#installation). Hook configuration
can be found at .pre-commit-config.yaml. `isort` and `yapf` tools are included to run as a part
of precommit hook.

To register a new hook run:
```shell script
pre-commit install
```

To run hooks without code committing:
```shell script
pre-commit run
```

Sometimes these tools can have a conflict regarding import sorting. In this case consider to
enable `include_trailing_comma` option at `setup.cfg` file.

Don’t forget to run `pylint` and `mypy` before code committing manually. It will save you from
wasting time at reading CI logs for failed build.

In order to check everything is fine (and have a dry run for formatting and import sorting)
consider following commands, e.g. for `exp_results_process_lambda`:
```shell script
cd exp_results_process_lambda
workon exp_results_process_lambda # or './venv/bin/activate'
isort --check-only -rc . && yapf --recursive --verbose --parallel --diff . \
&& pylint --rcfile=pylintrc ./${PWD##*/} ./tests && mypy ${PWD##*/} && echo 'Exit code:' $?
```

It will run all checks inside a current directory.