Lambda function for Kinesis Events to Cloudsearch
============================================================

This lambda takes Kinesis events generated by [Maxwell's Daemon](http://maxwells-daemon.io/dataformat/) reading mysql bin logs, then constructs and uploads Cloudsearch documents to achieve near-instantaneous search results in the UI (via Cloudsearch) after a user modifies their data.

Developing
---------------------------

### Python environment

You can use a python virtual environment or use Docker as is preferred in the Orchard Python Lambda [boilerplate](https://github.com/theorchard/docs/tree/master/boilerplates/python-lambda).


### Code organization

- Lambda Python code must be placed in a repository of your choice, together with other
- Lambda functions in that repo under a common "lambda" directory.  Within this directory, there will be any number of functions and optionally one special directory called "common".
- The contents of "common" will be treated as another dependency that is injected into deployed zip file at build time.

The is one of the lambdas, `alooma_cloudsearch`. It has 3 functions `cloudsearch_upload`, `common`, and `kinesis_submit`. The `common` folder is packaged in with `cloudsearch_upload` and `kinesis_submit` when you run the build/deploy shell script `build/build.sh` so it's not really a function.

* **alooma_cloudsearch**
  * lambda
    * **cloudsearch_upload** - _Handles the Kinesis events, constructs a cloudsearch document and uploads to the correct corpus_
    * **common** _Shared modules that get packaged in with the other functions in ../lambda._
    * **kinesis_submit** - _Mock fire a Kinesis event (?)_

### Getting started

Build an image called `maxwells-cloudsearch` using the Dockerfile in `./build/`, which is based off of a python 3.6 image.

```bash
docker build -t maxwells-cloudsearch /path_to_local_checkout/lambda-alooma-cloudsearch/build
```

### Development

We cannot run the AWS Lambda environment locally, but we can run `pytest` and `flake8` against a built (zipped with all dependencies in same folder) environment.

To prepare a deploy file and run tests:

```bash
# login to docker if you haven't already
docker login
```

```bash
# shell into the maxwells-cloudsearch container and
# map the locally checked out repo to a folder in the container
docker run -it --volume /path_to_local_checkout:/reporoot maxwells-cloudsearch /bin/bash
```

Once you are in the container, you can run the build/test script for each lambda ad nauseam while developing.
Notice the 1st parameter `test` sets the environment to `test` for the unit tests, which affects test values for the assertions.

```bash
# run the build/test script against alooma-cloudsearch:cloudsearch_upload
/reporoot/build/build.sh test /reporoot/alooma-cloudsearch/lambda cloudsearch_upload
```

You should see familiar pytest and flake8 output after all the pip installs.

### Deployment

During development, it would be possible to deploy the function within dev
AWS account using the following command:

```
$ /proj/docs/boilerplates/lambda/build/deploy.sh dev /proj/docs/boilerplates/python-lambda/example/lambda/ myfunc aws_lambda_name
```

  - *dev* – environment, matches the AWS Lambda function name prefix
    (that is part of the name you've used during the Lambda function creation).
  - *myfunc* – python package name.
  - *aws_lambda_name* the AWS lambda function name without the *prefix-*.

This assumes that your default AWS credentials configured in *~/.aws/credentials* matche profile pointing at Orchard's dev account.  You can
verify your AWS configuration as follows:

```
$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      dev           manual    --profile
access_key     ****************IDJA shared-credentials-file
secret_key     ****************osO9 shared-credentials-file
    region                <not set>             None    None
```

The following would indicate a missing default profile:

```
$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      dev           manual    --profile

The config profile (dev) could not be found
```

If you have configured multiple AWS accounts in *~/.aws/*, `aws` command
(and the `deploy.sh` that uses it) should be prefixed with the env variable that
is set to the target account:

```
$ AWS_PROFILE=orchard-dev aws configure list
$ AWS_PROFILE=orchard-dev /proj/docs/boilerplates/lambda/build/deploy.sh dev /proj/docs/boilerplates/python-lambda/example/lambda/ myfunc aws_lambda_name
```

Continuous integration
----------------------

Jenkins will be serve an interface for automatic building and deploying of Lambda functions outside
a local development environment.  A Jenkins job can be created with the following parameters:

- repo
- relative path to lambda dir (including lambda dir itself)
- Lambda function name (must match subdirectory)
- environment (dev, prod)

This job will perform the following steps:

1. clone the repo
2. install dependencies as per `requirements.txt`
3. create a zip file containing function's source code and its dependencies
4. install dependencies as per `requirements-dev.txt`
5. run pytests and flake8, halt on failure
6. deploy zip file into appropriate AWS account

In the example provided in this boilerplate, assuming this repo is checked out in `$REPO_HOME` and the image is `ci-lambda`
and we're targeting environment $ENV, docker command will look like this:
```
docker run -it --volume $REPO_HOME:/reporoot ci-lambda /etc/lambda/build.sh $ENV /reporoot/boilerplates/python-lambda/example/lambda myfunc
```