Export of MastersRegistry data from DynamoDB to Snowflake 
=========================================================

POC for exporting the MR data from DynamoDB to Snowflake using Kinesis Streams and Lambda.


### Getting started

Prepare your Docker image (we will call it `ci-lambda`):

```
$ docker build -t ci-lambda /proj/lambda-mr-snowflake/build/
```

### Development

It may not be possible to run your Lambda function outside of the Lambda environment.  It should be
possible, however, to use the build script provided by this Docker container to ensure all
dependencies are satisfied and unit tests pass.

To prepare a deploy file and run tests:

##### Backfill Lambda
```
$ docker run -it \
> --volume /proj/lambda-mr-snowflake/snowflakeloader/:/reporoot \
> ci-lambda \
> /home/jenkins/build.sh dev /reporoot/lambda backfill
```

##### Running producer

To run the producer script locally you have to create virtualenv and install
 requirements first.
```
$ virtualenv /venv/lambda-producer
$ source /venv/lambda-producer/bin/activate
$ pip install -r /proj/lambda-mr-snowflake/snowflakeloader/lambda/common/requirements.txt
```

Then you need to set up and export environment variables.
```
$ cd /proj/lambda-mr-snowflake/snowflakeloader/lambda/common
$ cp env.shadow env
```
Fill-in all values in `env` file. Then export environment variables:
```
$ set -o allexport; source env; set +o allexport
```

Now you can run the producer locally:

```
$ python producer.py
```

The producer will query records from source DB and send them to kinesis stream.
After that it will keep printing the number of records left to process, 
until the backfill lambda process all records.

### Deployment

During development, it would be possible to deploy the function within dev
AWS account using the following command:

##### Backfill Lambda
Assuming that you have already create a lambda function in your dev account, and it's 
name is `dev-mr_snowflake_backfill`.
```
$ /proj/lambda-mr-snowflake/build/deploy.sh dev /proj/lambda-mr-snowflake/snowflakeloader/lambda/ backfill mr_snowflake_backfill
```

#### Environment Variables
| Env Variable  | Purpose  |
| ------------- |--------- |
| MAX_RECORDS | Should be set in producer to determine the number of records to process. Note, that in current implementation it limits the amount of records, grouped by ISRC. |
| TIMEOUT | Maximum timeout that producer will be waiting for a batch of records to process |
| REDIS_HOST | Redis host to keep tracking the number of records processed |
| REDIS_PORT | Redis port (usually 6379) |
| REDIS_DB | Redis DB number |
| REDIS_PASSWORD | Redis DB Password |
| SNOWFLAKE_ACCOUNT | Snowflake  account |
| SNOWFLAKE_USER | Snowflake user |
| SNOWFLAKE_PASSWORD | Snowflake password |
| SNOWFLAKE_DATABASE | Snowflake database name |
| SNOWFLAKE_TABLE | Snowflake table name |
| SNOWFLAKE_SCHEMA | Snowflake schema |
| SNOWFLAKE_WAREHOUSE | Snowflake warehouse |
| SNOWFLAKE_TIMEZONE | Snowflake timezone |
| MR_ACTIVE_TABLE_NAME | "Active" table name in DynamoDB |
| BACKFILL_KINESIS_STREAM | Kinesis stream that will be written by producer script |


This assumes that `dev` matches your AWS config profile pointing at Orchard's dev account.  You can
verify your AWS configuration as follows:

```
$ aws --profile dev configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      dev           manual    --profile
access_key     ****************IDJA shared-credentials-file    
secret_key     ****************osO9 shared-credentials-file    
    region                <not set>             None    None
```

The following would indicate a missing `dev` profile:

```
$ aws --profile dev configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      dev           manual    --profile

The config profile (dev) could not be found
```

Continuous integration
----------------------

Jenkins will be serve an interface for automatic building and deploying of Lambda functions outside
a local development environment.  A Jenkins job can be created with the following parameters:

- repo
- relative path to lambda dir (including lambda dir itself)
- Lambda function name (must match subdirectory)
- environment (dev, prod)

This job will perform the following steps:

1. clone the repo
2. install dependencies as per `requirements.txt`
3. create a zip file containing function's source code and its dependencies
4. install dependencies as per `requirements-dev.txt`
5. run pytests and flake8, halt on failure
6. deploy zip file into appropriate AWS account

In the example provided in this boilerplate, assuming this repo is checked out in `$REPO_HOME` and the image is `ci-lambda`
and we're targeting environment $ENV, docker command will look like this:
```
docker run -it --volume $REPO_HOME:/reporoot ci-lambda /etc/lambda/build.sh $ENV /reporoot/snowflakeloader/lambda/ backfill
```