ows-analytics
==========

Provides analytic data endpoints. 
Python flask application

## Install

ows-analytics uses Python with [pipenv](https://docs.pipenv.org/),

### Installation

1.  Install the python version used by the project set in `.python-version` with pyenv

```
$ brew update
$ brew install pyenv
$ pyenv install `cat .python-version`
```

To add pre-commit to your git hook:

```
brew install pre-commit
pre-commit install
```

To install dependencies, run

```bash
make pip_dev
```

----

Did you see something like this?
```bash
No local packages or working download links found for setuptools>=xx.x.x
```
If so, try this
```bash
env/bin/pip install --upgrade -vv setuptools
```
Then try again


## Run

Copy `.env.shadow` to `.env` and fill in variables. Then run the application with

```bash
make run
```

### Local Grass bypass

If you would like to patch your own local account information for every request,
fill in the following environment variables in the `.env` file with similar values,
depending on which account you would like to impersonate:

```
# Authorization overrides
GRASS_ACCOUNT_TYPE=subaccount
GRASS_ACCOUNT_ID=21989
ORCHARD_USER_ID=alw:1000
```

Note that if you leave these environment values defined while running integration
tests against the server, they will fail (as the integration tests are expecting
a particular label).

## Test

You must have a region in your default aws settings i.e. your 
`~/.aws/credentials` must have something like `region = us-east-1`

There are three kinds of tests: unit, lint, and integration.

### Unit

Simply do

```bash
make test_unit
```

To produce coverage reports, run `make test_unit_cov`.

### Lint

Run lint checks with

```bash
make lint
```

### Integration

Make sure the application is running, then do

```bash
make test_integation
```

## Clean up 

Remove the virtual env directory by running

```bash
make clean
```

## Snowflake authentication using key pair

To authenticate using SSH use following documentation:
https://docs.snowflake.net/manuals/user-guide/snowsql-start.html#using-key-pair-authentication

TL;DR:
```bash
mkdir ~/.ssh/snowflake && cd ~/.ssh/snowflake # recommended
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
```

Make sure to set a password and then put this into the `SNOWFLAKE_KEY_PASSPHRASE` environment
variable in your `.env` file.

Then add the public key to your user via Terraform: https://github.com/theorchard/terraform-infra/blob/master/prod/snowflake/orchard/human_users/variables.tf.

Note: if you put key not in ~/.ssh/snowflake, then enter location in SNOWFLAKE_PRIVATE_KEY_PATH variable.

## Application Handlers
If you create a new handler or update an existing one, please document this in the `ows-analytics-1.0.0.yaml` Swagger file.


## python-owsrequest and access_rules.yml

### What is access_rules.yml

`analytics/access_rules.yml` is a declarative RBAC (Role-Based Access Control) configuration file that maps endpoint paths to allowed profile types and roles. Each rule specifies:

- A **URL path pattern** using Flask-like syntax (e.g. `/product/<*>/streams-all`)
- Allowed **HTTP methods** (`GET`, `POST`)
- Allowed **profiles and roles** (e.g. `InsightsProfile: [administrator, analytics]`)

The file currently defines rules for ~65 endpoints covering sound recordings, products, participants, videos, channels, TikTok, and more.

### How it works

The file is consumed by the [python-owsrequest](https://github.com/theorchard/python-owsrequest) middleware. In `analytics/api.py`, `flask_request.setup()` is called with:

```python
flask_request.setup(
    app,
    ...
    verify_access=True,
    rules_file="analytics/access_rules.yml",
    access_log_only=config.ONLY_LOG_ACCESS_ERRORS,  # True
    ...
)
```

This creates an `EndpointRulesValidator` (from `owsrequest/rules.py`) that parses the YAML into `EndpointRule` objects, each with a compiled regex path, allowed methods, and allowed roles.

### Request lifecycle

The `verify_rules_access()` function runs as a Flask `before_request` hook on every incoming request:

1. **Skips health-check paths** (configured as `/hello/` in `exclude_paths`)
2. **Checks the caller's service name** against `ACCESS_CONTROL_SERVICES = ['graphql-<str>', 'ows-grass']`. If the request does NOT come from `graphql-*` or `ows-grass`, the check is **skipped entirely** (microservice-to-microservice calls bypass the rules).
3. **Matches the request path** against the YAML rules using regex.
4. **Checks** if the requestor's `profile_type` and `roles` are in the allowed set.
5. If the check fails and `access_log_only=False`, returns **403 Forbidden**. If `access_log_only=True` (current setting), the request **proceeds anyway**.

Note: if the request path does not match any rule in the YAML, `has_access()` returns `False`, so any endpoint not listed in `access_rules.yml` would be flagged as unauthorized.

### Current enforcement status

| Aspect                           | Status                                                     |
|----------------------------------|------------------------------------------------------------|
| File loaded at startup           | Yes — parsed into `EndpointRule` objects                   |
| `before_request` hook registered | Yes — runs on every request                                |
| Rules evaluated                  | Only for requests from `graphql-*` or `ows-grass`          |
| Violations blocked (403)         | **No** — `ONLY_LOG_ACCESS_ERRORS = True` (log-only mode)  |
| Could enforce                    | Yes — set `ONLY_LOG_ACCESS_ERRORS = False` in `config.py`  |

### Audit log visibility

The access rules system emits two log messages:

| Log message                | Level   | Visible in prod? |
|----------------------------|---------|-------------------|
| "skipping role check"      | `DEBUG` | No                |
| access denied from rules   | `INFO`  | No                |

**Both are suppressed in production** because `LOGGER_LEVEL` is set to `WARNING` for the prod environment (`analytics/config.py`). To see audit output in Datadog, either lower the production log level to `INFO` or change the access-denied log call in `owsrequest` from `.info()` to `.warning()`.

For comparison, JWT authentication failures are logged at `WARNING` level and _do_ appear in Datadog (`env:prod service:ows-analytics "The service has received an unauthorized request in"`).

## Cache pre-warming
The script in the `warmup_cache` dir allows to pre-warm the Redis cache for the top long running queries. This is designed for the employees (the users with the full access to the catalogue).