Redshift unload
===============

This ETL process takes all the information from redshift and inserts it into Dynamodb (using S3 and EMR clusters.)

Requires:

* Docker.
* Access to Redshift, s3 and EMR.

Some preconfigured Docker images come with all dependencies installed. If not, you may have to complete the following steps:

1. Install gcc, postgresql, postgresql-devel, python3x-devel (matching your python version)
2. Symlink /usr/lib64/libncurses.so to /usr/lib64/libncurses.so.x (matching your installed library version)

### Backfill

#### Via Python

Whenever you need to backfill the data, you will need 3 informations: the list
of transaction ids you want to reprocess, the dates you want to reprocess and
if you want the backfill to happen right away (more expansive) or if it can wait
until our normal ETL runs.

```bash
python cli.py -s -c '{\
    "config": "label",\
    "start_date": "2015-10-01",\
    "end_date": "2015-10-15",\
    "transaction_type_ids": "8,30",\
    "unload_redshift_only": true}'
```

If the flag `unload_redshift_only` is set, only the redshift unload will run.
The unload data will be stored in the bucket, and when the ETL runs, it will be
processed. If the flag is not here, the full ETL will run. This includes a
DynamoDB throughput increase.

#### Via the AWS Console

Similar to the above, all you have to do to run a backfill would be to start a
SWF Execution directly in the console of `prod_emr_aggregate_load`, with the
JSON mentioned above. Make sure to use the latest version.

### Deployment

* Pull down the directory.
* Add a id_rsa with access the github.com/theorchard/garcon-contrib/ repository.
* Replace the environment values in the Dockerfile.show and move it to Dockerfile.
* Run: `sudo docker build -t redshift_unload .`
* Stop the previous running docker.
* Start the new one: `sudo docker run -p 80:80`