# publishing-bulk-ingestion

Lambdas for bulk ingestion of publishing spreasheets

## Setup
* Install [docker](https://docs.docker.com/get-docker/)
* Clone this repository
* Navigate to the lambda you want to work on in `lambda/`



## Building

Building the lambda depends on 2 build arguments needed to install JS and TS packages.

```
PACKAGECLOUD_USER (this is a user not a token)
GITHUB_TOKEN
```

The `PACKAGECLOUD_TOKEN` is needed for getting our older JS packages from our PackageCloud.
The `GITHUB_TOKEN` is needed for getting our newer TS packages from the Github monorepo.

These secrets are boud in the Jenkins job as environment variables and then saved to a temp file in the workspace. The build command would be as follows. The Jenkins job will use the Docker Build and Publish plugin to build and push the image to ECR. It needs to be instructed to us BuildKit and the location of these secrets. 

If you want to build the image locally and push it to the AWS Dev ECR, you can stage the secrets file locally ( `.env`) and run the following command

```
DOCKER_BUILDKIT=1 docker build --secret id=tokens,src=<temp_secret_file> -t <lambda_name_tag> .
```

The following build step consists of running the `setup.sh` script which will:

* sources the staged secrets by BuildKit `source /run/secrets/tokens`
* obtain a new fresh `PACKAGECLOUD_TOKEN`
* generate `.npmrc` file
* run `yarn install`
* delete .npmrc

## Running

### .env

You need the 2 secrets mentionned aboive if you want to build, run or test contairized lambda locally. Copy `.env.shadow` to `.env`

Given that DockerKit is not supported by compose yet, the `function` service uses a different Dockerfile, ie, `Dockerfile.local`.  This file is identical to the main `Dockerfile` but with a slightly different build step for installing packages. The secrets will just be passed as build args and the `setup-local.sh` script will be executed instead. Passing secrets as build args is considered insecure but is suitable for local development.  

### Start Container
```
$ docker-compose up --build -d function
```

### Execute Function
Use the HTTP client of your choice. The body of the request is the `event` passed into the function.

```sh
curl --request POST \
  --url http://localhost:9000/2015-03-31/functions/function/invocations \
  --header 'Content-Type: application/json' \
  --data $(cat tests/sample_event.json)
```

### Load Changes

Given that the TS files are compiled in the container, loading changes requires recompiling the TS files and restarting the function. There is an attempt to use `nodemon` and restart the container when the source changes but its not working quite right yet.

## Linting and Testing
Given that DockerKit is not supported by compose yet, its probably just fine running the lint and test tasks directly on the host, wether you local environement or Jenkins.

```sh
yarn
yarn test
```

The `lint-and-test` service will run the `yarn test` command which runs the linter and the unit tests. 

### Run Container
```sh
$ docker-compose up --build lint-and-test
```

### Capture Exit Code
This will not work on Jenkins unless we pass the secrets as CLI args which is insecure.

Unless the container itself crashes, or is killed, the exit code will be `0`. When the exit code of `lint-and-test.sh` script matters, a pull-request builder for example, run tests with the following flags.

```sh
$ docker-compose up --exit-code-from lint-and-test --abort-on-container-exit --build lint-and-test
```