# docker-image-build-cache-poc

When building docker images we can use the [--cache-from](https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources) option to use layers from previously built images instead of re-building every layer. This can give us the advantages of using the docker build cache regardless of the state of the local docker build cache to speed-up deployments.

## Notable Concepts

* Images must be built with `--build-arg BUILDKIT_INLINE_CACHE=1` to be used as cache references when building new images
* Dockerfiles should order operations from **least** often changed to **most** in order to take full advantage of caching
* [Multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/) are used to easily manage container dependencies, but are **not** required to take advantage of build caching, but would highly recommend using them so that test images can take advantage of base layers which are common with the application images
* We don't currently save test images to ECR, if we wanted to take full advantage of caching (requirements-test.txt and not just requirements.txt) we could follow the same process described below
* Some of our Jenkins builds in the "Docker Build and Push" block use `No Cache` and/or `Force Pull`, we should turn these options off if we want to use a image as cache


## Scripts

### build_and_push.sh

#### Parameters
* $1 => tag to build new image with
* $2 => tag of existing image to use as cache
* $3 => build stage target (default to "function")

#### Process
* create ECR repo if it does not exist
* grant docker access to ECR
* delete all local docker cache references for this project
* build image and tag with $1, using $2 as cache
* push image to ECR


### pull_and_run.sh

#### Parameters
* $1 => tag of image to pull down and run

#### Process
* run container based in ECR image reference
* send curl request to container, output response
* output logs
* kill container
## Proof of Concept

Follow these steps to observe how docker behaves. This process should be repeatable via Jenkins using git commit hashes.

:exclamation::exclamation: The local docker cache for this project is cleared during each run of `build_and_push.sh`, so subsequent builds act as if they are being run on different machines to simulate how Jenkins would act.

### 1. Setup

The following things are necessary to setup before getting started.
* [jq](https://formulae.brew.sh/formula/jq) - tested with v1.6
* [awscli](https://formulae.brew.sh/formula/awscli) - tested with v2.1.34
* [generated AWS credentials for dev](https://github.com/theorchard/collab/blob/7b610e3d44771ec4b6784d0820cc7d645e26a800/jcarrion/aws-creds-generator/README.md)
  * :warning: set `region = us-east-1` in `~/.aws/credentials` if not done already

### 2. Initial build without cache

"v1" does not exist, "v2" should build without any cache

* `./build_and_push.sh v2 v1`
* `./pull_and_run.sh v2`

### 3. Update source code

"v2" has the same packages, "v3" should build quickly and see code changes

* change `./src/app.py` and set `CHARACTER = 'dragon'`
* `./build_and_push.sh v3 v2`
* `./pull_and_run.sh v3`

### 4. Update package requirements

"v3" has different packages, "v4" should build without packages cached

* change `requirements.txt` to use `cowsay==4.0.0`
* `./build_and_push.sh v4 v3`
* `./pull_and_run.sh v4`

### 5. Build test container

"v4" has same base packages, but different test packages, should only install test packages

* `./build_and_push.sh v4-test v4 lint_and_test`


## Other Application Types

The above example uses the containerized lambda project template, but the same concepts can be applied to other types of applications to get the same benefits. Exploration was done to see what that level of effort would look like.

* [ows-sound-recordings](https://github.com/theorchard/ows-sound-recordings/compare/master...pdumoulin:pd.eng-week-june-2021?expand=1)
* [graphql-product](https://github.com/theorchard/graphql-product/compare/master...pdumoulin:pd.eng-week-june-2021?expand=1)