# Core Load Tester

core-load-tester is a tool designed for performing HTTP load testing
on SME-DnA services, including:

- core-images


## Purpose

Comprehensive HTTP load testing typically depends on a well-maintained
catalog of endpoints to subject ot testing and how each of them are
used, including authentication, authorization and permissible
parameters. Maintaining this catalog incurs a significant development
cost on all services to be load tested.

As a way to garner most of the benefits of load testing without
incurring much of that cost, this tool eschews hand-written endpoint
rules in favor of collecting that data from the usage patterns of the
System Under Test (SUT). Usage data is sampled from the CloudFront
distribution or load balancer that provides access from the Internet to the SUT.

Also is possibility manually describe the endpoint rules
and place them in the `sample-requests` directory.

## Overview

A local document is created with a tabulated record of endpoints
accessed via the CloudFront distribution or load balancer for a given calendar
day and lookback period (default 3 days).
That tabulated data is packaged into a Docker image along with
the [Locust](https://locust.io/) tool and a just enough Python to
authenticate with the SUT. The resulting Docker image is ready to be
deployed to the location from which the load testing will occur.


## Conditions

Usage of the core-load-tester tool is contingent on the following:

1. The SUT must be accessed from the Internet through an AWS
CloudFront distribution or load balancer.

2. The CloudFront distribution or load balancer for the SUT must be configured to
collect its access logs in AWS S3.

3. An AWS Athena database and table must be configured to process the
collected access logs of that CloudFront distribution or load balancer.


## Configuration

### High-level configuration
In Jenkins using build parameters can make the main configuration.

`ACCESS_LOGS_SOURCE_ENVIRONMENT` - set logs source from CloudWatch;\
`TARGET_ENVIRONMENT` - set testing environment (e.g. this allows testing separately of some environments `[apollo.stream/gate-api](http://apollo.stream/gate-api)` and `[apollo.stream/vendor-api](http://apollo.stream/vendor-api)` even though they are on the same domain);\
`CURRENT_DATE` - set date for getting logs data (e.g. 5/04/2023 means to get logs data by period from 5/01/2023 to 5/04/2023, by default we get data for 3 last days). You can also config this count of the last days in EXTRA_ARGS, just setting `--shift` param.\
`NUM_USER` - number of users, by default, is 100;\
`RUN_TIME` - load testing duration, by default, is 600s;\
`SPAWN_RATE` - service will spawn new users every second until it reaches the total number of users, by default, is 5;\
`S3_BUCKET_REQUESTS_EXTRA_CONFIG` - s3 bucket name where we can configure work with requests in more detail;\
`EXTRA_ARGS` - set additional arguments;

### EXTRA ARGS configuration
`-l`, `--limit` - limit for Athena query (the same as for SQL queries), by default, is 1000;\
`-s`, `--shift` - count of days to get data acquisition period, by default, is 3 days;

### Requests configuration
In the s3 bucket we can create 4 CSV files for extra config working with requests that we get from logs:

`add-requests.csv` - add specific requests;\
`replace-requests.csv` - replace specific requests (find URLs by regex term and query params, and replace both URLs and query params);\
`remove-requests.csv` - remove specific requests (find URLs by regex terms and query params, and exclude these requests from load testing);\
`add-http-headers.csv` - add headers for specific requests;

### Add requests configuration

*logs data*

| method | uri | query_string | count |
| --- | --- | --- | --- |
| GET | /api/v1/tracks | limit=200&offset=250 | 10 |
| GET | /api/v1/track/12345 |  | 2 |

*#add-requests.csv*

| method | uri | query_string | count | content_type | body |
| --- | --- | --- | --- | --- | --- |
| GET | /api/v1/track/55555 |  | 5 | application/json | {”key”: “value”} |
| GET | /api/v1/playlists | limit=50 | 100 |  |  |

*testing data*

| method | uri | query_string | count | content_type | body |
| --- | --- | --- | --- | --- | --- |
| GET | /api/v1/tracks | limit=200&offset=250 | 10 |  |  |
| GET | /api/v1/track/12345 |  | 2 |  |  |
| GET | /api/v1/track/55555 |  | 5 | application/json | {”key”: “value”} |
| GET | /api/v1/playlists | limit=50 | 100 |  |  |

### Replace requests configuration

1. Simple case.

*logs data*

| method | uri | query_string | count |
| --- | --- | --- | --- |
| GET | /api/v1/tracks | limit=200&offset=250 | 10 |
| GET | /api/v1/track/12345 |  | 2 |
| GET | /api/v1/playlists | limit=50 | 10 |

*#replace-requests.csv*

| method | regex_search_term | replacement | search_query_params | replacement_query_params |
| --- | --- | --- | --- | --- |
| GET | v1 | v2 |  |  |

*testing data*

| method | uri | query_string | count |
| --- | --- | --- | --- |
| GET | /api/v2/tracks | limit=200&offset=250 | 10 |
| GET | /api/v2/track/12345 |  | 2 |
| GET | /api/v2/playlists | limit=50 | 10 |


2. For multiple rules, follow the precedence in the *replace-requests.csv* file, specifying the more specific replacement rules first, then the more general ones. In the example below, we want to replace the API version, but in one endpoint we need to change the endpoint also, from api/v1/tracks to api/v2/old-tracks, but if we don't prioritize, we can get api/v2/tracks that is incorrect.

*logs data*

| method | uri | query_string | count |
| --- | --- | --- | --- |
| GET | /api/v1/tracks | limit=200&offset=250 | 10 |
| GET | /api/v1/track/12345 |  | 2 |
| GET | /api/v1/playlists | limit=50 | 10 |
| GET | /api/v1/artists | limit=10 | 5 |

*#replace-requests.csv*

| method | regex_search_term | replacement | search_query_params | replacement_query_params |
| --- | --- | --- | --- | --- |
| GET | v1\/tracks | v2/old-tracks |  |  |
| GET | v1 | v2 | limit=50 | limit=200 |
| GET | v1 | v2 |  |  |

*testing data*

| method | uri | query_string | count |
| --- | --- | --- | --- |
| GET | /api/v2/old-tracks | limit=200&offset=250 | 10 |
| GET | /api/v2/track/12345 |  | 2 |
| GET | /api/v2/playlists | limit=200 | 10 |
| GET | /api/v2/artists | limit=10 | 5 |


### Remove requests configuration

*logs data*

| method | uri | query_string | count |
| --- | --- | --- | --- |
| GET | /api/v1/tracks | limit=200&offset=250 | 10 |
| GET | /api/v1/track/12345 |  | 2 |
| GET | /api/v2/playlists | limit=50 | 10 |
| GET | /api/v2/artists | limit=10 | 5 |
| GET | /api/v2/tracks | limit=100 | 10 |
| GET | /api/v2/tracks | limit=5000 | 150 |

*#remove-requests.csv*

| method | regex_search_term | search_query_params |
| --- | --- | --- |
| GET | v1 |  |
| GET | v2\/tracks | limit=5000 |

*testing data*

| method | uri | query_string | count |
| --- | --- | --- | --- |
| GET | /api/v2/playlists | limit=50 | 10 |
| GET | /api/v2/artists | limit=10 | 5 |
| GET | /api/v2/tracks | limit=100 | 10 |


### Add HTTP headers configuration

*#add-http-headers.csv*

| method | regex_search_term | header |
| --- | --- | --- |
| GET | v1\/tracks | {”Accept-Language”: “de”} |
| GET | v1\/artists | {”Accept-Language”: “en”} |

Accept-Language (de) will be added for all tracks and (en) for all artists.

| method | regex_search_term | header |
| --- | --- | --- |
| GET | v1\/tracks | {”Accept-Language”: “de”} |
| GET | v1\/tracks | {”Cache-Control”: “no-cache”} |
| GET | v1\/artists | {”Accept-Language”: “en”} |

## Running core-load-tester

 - For running whole flow (getting testing data and run this data in the Locust) run:
    ```sh
    make docker/run/local DATE=2023-02-01 HOST=dev-images-api.atlas.stream
    ```
    Available params:\
    `DATE=2023-02-10` - Date by getting logs data from Athena (is required);\
    `HOST=dev-images-api.atlas.stream` - Host of testing service (is required);\
    `SAMPLE_DIR=<your-dir>` - Dir by save csv file with testing data for locust `(default=samples)`;\
    `REPORT_DIR=<your-dir>` - Dir by save report data generated by locust `(default=report)`;\
    `LIMIT=1000` - Number of urls items in the csv file with testing data `(default=1000)`;\
    `SHIFT=3` - Shift date in days. If shift is 7 days, it means get data by last 7 days regarding setting DATE    `(default=3)`;\
    `NUM_USERS=10` - Peak number of concurrent Locust users `(default=10)`;\
    `SPAWN_RATE=5` - Rate to spawn users at (users per second) `(default=5)`;\
    `RUN_TIME=60s` - Stop after the specified amount of time `(default=60s)`;


 - For getting testing data and crating csv file run:
    ```sh
    make docker/run/sample/aws DATE=2023-02-01 HOST=dev-images-api.atlas.stream
    ```
   Available params:\
    `DATE=2023-02-10` - Date by getting logs data from Athena (is required);\
    `HOST=dev-images-api.atlas.stream` - Host of testing service (is required);\
    `SAMPLE_DIR=<your-dir>` - Dir by save csv file with testing data for locust `(default=samples)`;\
    `LIMIT=1000` - Number of urls items in the csv file with testing data `(default=1000)`;\
    `SHIFT=3` - Shift date in days. If shift is 7 days, it means get data by last 7 days regarding setting DATE    `(default=3)`;\

    ```sh
    make docker/run/sample/datadog DATE=2023-02-01
    ```
   Available params:\
    `DATE=2023-02-10` - Date by getting logs data from Athena (is required);\
    `SAMPLE_DIR=<your-dir>` - Dir by save csv file with testing data for locust `(default=samples)`;\
    `SHIFT=3` - Shift date in days. If shift is 7 days, it means get data by last 7 days regarding setting DATE    `(default=1)`;\


 - For running locust using your csv file run:
    ```sh
    make docker/run/locust HOST=dev-images-api.atlas.stream CSV_FOLDER=<absolute-path-to-csv-files-folder> CSV_FILE_NAME=<filename(without ext)>
    ```
   Available params:\
    `HOST=dev-images-api.atlas.stream` - Host of testing service (is required);\
    `CSV_FOLDER=/Users/root/testing-data/` - Absolute path to csv files folder (is required);\
    `CSV_FILE_NAME=my-csv-file` - Filename of csv file without ext (is required);\
    `REPORT_DIR=<your-dir>` - Dir by save report data generated by locust `(default=report)`;\
    `NUM_USERS=10` - Peak number of concurrent Locust users `(default=10)`;\
    `SPAWN_RATE=5` - Rate to spawn users at (users per second) `(default=5)`;\
    `RUN_TIME=60s` - Stop after the specified amount of time `(default=60s)`;