Locust load testing for graphql-knowledge-search.

# Setup

```bash
$ make env

# Add a valid JWT to TOKEN
$ cp .env.shadow .env

# Get access to PDP test login secrets
$ awsume permissions-platform-qa
```


# Tests

### Full QA Tests

```bash
# 10 minute test (JWT expires in 15 mins :-/)
make load_test

# GlobalParticipantSearchES load test
make gp_search_test

# GlobalParticipantSearchESxPP load test
make gp_pp_search_test

# to run tests for an hour, use the zsh "repeat 6" command. 
# re-run `awsume` if the command fails due to an expired AWS session token
repeat 6 { make gp_search_test }

# or 
repeat 6 { make gp_pp_search_test }
```

### Datadog Metrics

- [graphql-knowledge-search](https://app.datadoghq.com/apm/services/graphql-knowledge-search/operations/express.request/resources?env=qa&resources=qson%3A%28data%3A%28visible%3A%21t%2Chits%3A%28selected%3Atotal%29%2Cerrors%3A%28selected%3Atotal%29%2Clatency%3A%28selected%3Ap50%29%2CtopN%3A%2110%29%2Cversion%3A%210%29&summary=qson%3A%28data%3A%28visible%3A%21t%2Cerrors%3A%28selected%3Acount%29%2Chits%3A%28selected%3Acount%29%2Clatency%3A%28selected%3Alatency%2Cslot%3A%28agg%3A75%29%2Cdistribution%3A%28isLogScale%3A%21f%29%29%2Csublayer%3A%28slot%3A%28layers%3Aservice%29%2Cselected%3Apercentage%29%29%2Cversion%3A%211%29&topGraphs=latency%3Alatency%2CbreakdownAs%3Apercentage%2Cerrors%3Aversion_count%2Chits%3Aversion_count&start=1685652529634&end=1685656129634&paused=false)

### pdp_test_profile json query

Fetch values stored in `locust_load_tests/graphql_knowledge_search/pdp_test_profiles.json`.

To regenerate this file:

1. Run this DB PR to add the QA test profiles: https://github.com/theorchard/database/pull/16700
2. Run the Neo4J query below and download the output as a JSON file.
3. Copy the file to `locust_load_tests/graphql_knowledge_search/pdp_test_profiles.json`

Query:
```cypher
MATCH(i:Identity)--(p:Profile)--(l:Label)
WHERE
  i.email in ["pdpteststresscomplicated@theorchard.com", "pdptest@theorchard.com", "pdpteststresssimple@theorchard.com"]
RETURN
    p.profileId as profile_id,
    p.uuid as profile_uuid,
    i.email as email,
    i.id as identity_uuid,
    l.uuid as tenant_uuid,
    l.id as vendor_id,
    p.profileType as profile_type,
    p.roles[0] as profile_role
ORDER BY profile_id;
```

### user_headers query
This is deprecated but keeping in case want to perf test more non-PP graphql endpoints.

The `header_values.json` file was generated using the neo4j query below and exported as `json`. The header values
are used ows-permissions endpoints: `GlobalParticipantSearchES GlobalSoundRecordingSearchES`.

```
MATCH
    (u:Identity)-[:HAS_PROFILE]->(p:Profile)-[:HAS_ACCESS_TO]->(v:Vendor)
    WHERE u.id in 
        ['02f3fd86-17e6-4555-8f4e-86d7b747cace', 'f415420b-2227-4d9f-8676-a7bddcd74fc8', 'c6942761-1bde-4f42-9cf1-813d30d669ef',
         'b77a1744-5da1-47a3-84c9-ae1421835d34', '6eb9d40e-d1f8-43e9-a1c0-f09b880e506f', 'f55ba766-2007-424a-9815-db4381fc47b3',
         '06846563-035a-49b0-ab9c-12f4f31bd597', 'b6efb925-2ec1-4e74-989d-41580f13033c', 'f104044b-6f3a-486f-841f-e44bf3915799',
         'f2a08c0a-dd3d-44e3-9cfe-08eddade72c1', '6de16fa1-12ef-4b5b-8dce-c4336ceab566', '02ce1fe8-c77a-4f04-b596-1a6f79a09cb6']
    AND p.profileType='LabelProfile'
RETURN
    u.id as identity_uuid,
    p.uuid as profile_uuid,
    v.uuid as vendor_uuid,
    "LabelProfile" as profile_type;
```

# References

- [Cerbos Performance Testing](https://www.notion.so/Cerbos-Performance-Testing-fd9ebea135f7439383c3ec9c8bf03386). Specifically the `gp_names.txt` and `gsr_names.txt` files.
- [Elasticsearch locust_load_tests in collabs](https://github.com/theorchard/collab/tree/master/buvarov/locust_load_tests)
- [locust.io docs](https://docs.locust.io/en/stable/writing-a-locustfile.html)