Kinesis POC
===========

POC for processing the accounting reports using Kinesis Streams and Lambda.

## Getting Started

### Deployment to lambda

Amazon Lambda supports Python 2.7 only. So you have to create virtualenv
for Python 2.7. After installing the requirements from `requirements.txt`
you have to copy the python packages from virtualenv to folder `lambda` 
in your project root. The folder structure should look like this:

```
\lambda 
  \aws_kinesis_agg
  \google
  \pymysql
  \redis
  \connectors
  \models
  config.py
  const.py
  lambda_consumer.py
  lambda_records_saver.py
  producer.py
  util.py
  requirements.txt
  README.md
```

To prepare the deployment package you have to compress all contents of
 `lambda` folder (but not the folder itself) into zip archive. This 
  archive can be uploaded to lambda via AWS console. 


### Running producer

When all dependencies have been installed, you can run the producer locally:

```
python producer.py
```

Don't forget to setup environment variables before running the script.

The producer will query records from source DB and send them to kinesis stream. To limit the number of records, please use the
 MAX_RECORDS environment variable.
After that it will keep printing the number of records left to process, 
until the records saving lambda process all records.


## Environment Variables
| Env Variable  | Purpose  |
| ------------- |--------- |
| MAX_RECORDS | Should be set in producer to determine the number of records to process. Note, that in current implementation it limits the amount of records, grouped by ISRC. |
| REDIS_HOST | Redis host to keep tracking the number of records processed |
| REDIS_PORT | Redis port (usually 6379) |
| REDIS_DB | Redis DB number |
| REDIS_PASSWORD | Redis DB Password |
| MYSQL_HOST | Source DB host |
| MYSQL_PORT | Source DB port (usually 3306) |
| MYSQL_USER | DB user |
| MYSQL_PASSWORD | DB password |
| MYSQL_DB | DB name (stmt-db) |
| SAVE_TO_S3 | Should be set in Lambda Consumer to configure should we save the CSV reports to S3, or just update the source DB. (default 1) |