# Hive Regression Tracking

Scripting for running assets against HIVE AI API and storing their results for reporting purposes.
Assets are perpetually stored in S3 and included in each run. Subsequent runs should illustrate how results may
change as model versions change.

`app.py` is the main entrypoint for this repository. It orchestrates running a set of assets through the HIVE AI API, captures the API responses, and persists those results so they can be compared across runs and used for reporting/regression tracking.

At a high level it typically:
- Gets the list of assets in s3://prod-hive-regression-tracking/assets/
- Iterates over the listed assets and submits them to the HIVE AI API as tasks.
- Records JSON API responses under s3://prod-hive-regression-tracking/results/<run_timestamp>/raw/
- Records flattened CSV versions of the results under s3://prod-hive-regression-tracking/results/<run_timestamp>/flattened/

## Maintaining a set of assets for testing

The assets used in the scans will be stored in s3://prod-hive-regression-tracking/assets/ and are named `<asset_final_id>.flac`

To add assets to the set, copy them from the mezzanine bucket to the tracking bucket and rename them after their asset final id. e.g.

```bash
aws s3 cp s3://prod-orcd-mezzanine-assets/2966fc20_779c_4db7_abab_59e9198a6af3.flac s3://prod-hive-regression-tracking/assets/122239111.flac
```

## How to use

### 1) Set up a virtual environment and install dependencies from requirements.txt

### 2) Configure credentials
Set your HIVE API key
```bash
export HIVE_CREDENTIALS="YOUR_KEY_HERE"
```

> Ensure that you're using the correct project that's configured with the model we want to store results for.

### 3) Run the script

```bash
python app.py
```

### 4) Inspect results

Results will be stored [in S3](https://437795906767-jjxcu2vs.us-east-1.console.aws.amazon.com/s3/buckets/prod-hive-regression-tracking?region=us-east-1&prefix=results/&showversions=false)