# lambda-statement-attachments ## Tech Design https://docs.google.com/document/d/1EDfcAYA5a9_dq-dD8LYiYZ1iFuTk6FoU9kf9hwmFfbw As of Apr 18, 2017, AWS now supports Python 2.7 and 3.6 for lambda. Developing Lambda functions --------------------------- Lambda Python code must be placed in a repository of your choice, together with other Lambda functions in that repo under a common "lambda" directory. Within this directory, there will be any number of functions and optionally one special directory called "common". The contents of "common" will be treated as another dependency that is injected into deployed zip file at build time. For example: ``` repo_root |- ... -lambda | |- common (optional) | | |- | | |- tests | | | \- | | |- requirements-dev.txt | | \- requirements.txt | |- myfunc1 | | |- | | |- tests | | | \- | | |- requirements-dev.txt | | \- requirements.txt | |- myfunc2 | | \- ... | ... ... ``` During local development, consider using the Docker image to run unit tests in the same environment, as what will be run by CI. Let's assume that you are working with the example Lambda function provided and that your `docs` repo is cloned into `/proj/docs` on your dev environment. We will also assume that your Docker is installed and set up. ### Getting started Prepare your Docker image (we will call it `ci-lambda`): ``` $ docker build -t ci-lambda /var/www/html/docs/boilerplates/python-lambda/build/ ``` ### Development It may not be possible to run your Lambda function outside of the Lambda environment. It should be possible, however, to use the build script provided by this Docker container to ensure all dependencies are satisfied and unit tests pass. To prepare a deploy file and run tests: ``` $ docker run -it \ > --volume /var/www/html/docs/boilerplates/python-lambda/:/reporoot \ > ci-lambda \ > /reporoot/build/build.sh dev /reporoot/example/lambda myfunc ``` ### Deployment During development, it would be possible to deploy the function within dev AWS account using the following command: ``` $ /proj/docs/boilerplates/lambda/build/deploy.sh dev /proj/docs/boilerplates/python-lambda/example/lambda/ myfunc aws_lambda_name ``` - *dev* – environment, matches the AWS Lambda function name prefix (that is part of the name you've used during the Lambda function creation). - *myfunc* – python package name. - *aws_lambda_name* the AWS lambda function name without the *prefix-*. This assumes that your default AWS credentials configured in *~/.aws/credentials* matche profile pointing at Orchard's dev account. You can verify your AWS configuration as follows: ``` $ aws configure list Name Value Type Location ---- ----- ---- -------- profile dev manual --profile access_key ****************IDJA shared-credentials-file secret_key ****************osO9 shared-credentials-file region None None ``` The following would indicate a missing default profile: ``` $ aws configure list Name Value Type Location ---- ----- ---- -------- profile dev manual --profile The config profile (dev) could not be found ``` If you have configured multiple AWS accounts in *~/.aws/*, `aws` command (and the `deploy.sh` that uses it) should be prefixed with the env variable that is set to the target account: ``` $ AWS_PROFILE=orchard-dev aws configure list $ AWS_PROFILE=orchard-dev /proj/docs/boilerplates/lambda/build/deploy.sh dev /proj/docs/boilerplates/python-lambda/example/lambda/ myfunc aws_lambda_name ``` Continuous integration ---------------------- Jenkins will be serve an interface for automatic building and deploying of Lambda functions outside a local development environment. A Jenkins job can be created with the following parameters: - repo - relative path to lambda dir (including lambda dir itself) - Lambda function name (must match subdirectory) - environment (dev, prod) This job will perform the following steps: 1. clone the repo 2. install dependencies as per `requirements.txt` 3. create a zip file containing function's source code and its dependencies 4. install dependencies as per `requirements-dev.txt` 5. run pytests and flake8, halt on failure 6. deploy zip file into appropriate AWS account In the example provided in this boilerplate, assuming this repo is checked out in `$REPO_HOME` and the image is `ci-lambda` and we're targeting environment $ENV, docker command will look like this: ``` docker run -it --volume $REPO_HOME:/reporoot ci-lambda /etc/lambda/build.sh $ENV /reporoot/boilerplates/python-lambda/example/lambda myfunc ``` ## Moving existing statement attachments To move existing statement attachments from one S3 directory to another use script which is located at: lambda/common/move_attachments_per_label_dir.py ### Required permissions Before usage please double check that the AWS user which will be used to run the scrip has the following permissions: * **S3** `{env}-orcdbucket` - Read/Delete `{env}-statement-attachments` directory (prefix) - Read/Write `{env}-per-label-statement-attachments` directory (prefix) * **DynamoDB** Read/Write (update) on`{env}_statement_attachments_success` table ### Usage: 0. **Important**! Disable `{env}-statement-attachments-remove-dynamodb-item` lambda function (or it's triggers). Otherwise it will delete all the attachments that were moved by the script, because the move operation is actually copy and delete operations. 1. Create or use existing virtualenv with python >=3.4 - `python3 -m venv env` 2. Activate the virtualenv - `source env/bin/activate` 3. Install the boto3 and botocore libraries: - `boto3==1.4.6` - `botocore==1.6.8` - or `pip install -r lambda/common/requirements-dev.txt` 4. Execute the script python lambda/common/move_attachments_per_label_dir.py \ --env dev -s dev-statement-attachments \ -d dev-per-label-statement-attachments -l info It is recomended to redirect the `stderr` to file to keep the detailed log of errors. 5. Check that **DynamoDB** has been updated and **S3** files are at the expected location. 6. Enable `{env}-statement-attachments-remove-dynamodb-item` lambda function (or it's triggers) #### Notes Please take into account that if copy operation was successful but the update to DynamoDB record was not - the original (source_key) file nor the copy (dest_key) WON'T BE DELETED. In that case please solve the DynamoDB issue and re-run the script (steps 0, 4 - 6).