# TRD: ows-royalties API Hardening

**Date:** 2026-06-25
**Author:** Michael Rojas
**Status:** Proposed
**Epic:** [ACC-10607](https://theorchard.atlassian.net/browse/ACC-10607)
**Scope:** Pilot in `ows-royalties`; pattern then templated to the five `ows-abacus-*` services. Source of truth for the code is `upstream/master` of [theorchard/ows-royalties](https://github.com/theorchard/ows-royalties).

---

## Table of Contents

1. [Overview](#1-overview)
2. [Background & Motivation](#2-background--motivation)
3. [Goals / Non-Goals](#3-goals--non-goals)
4. [System Context](#4-system-context)
5. [Architecture](#5-architecture)
6. [Configuration Model](#6-configuration-model)
7. [Inbound Capabilities](#7-inbound-capabilities)
8. [Outbound Resilience](#8-outbound-resilience)
9. [Readiness & Health](#9-readiness--health)
10. [Observability](#10-observability)
11. [Security Model & Edge Precondition](#11-security-model--edge-precondition)
12. [Testing Strategy](#12-testing-strategy)
13. [Rollout Plan](#13-rollout-plan)
14. [Risks & Open Actions](#14-risks--open-actions)
15. [Decisions](#15-decisions)
16. [References](#16-references)

---

## 1. Overview

This project adds a production-grade **API hardening kit** to the Orchard Accounting (Abacus) Flask services: rate limiting, response compression, security headers, request body-size limits, a readiness probe, a custom thread-safe circuit breaker with per-resource outbound guards, and observability. It is delivered as a self-contained `core/hardening/` package wired in through a single `init_hardening(app, config)` call. Security headers, compression, and proxy handling are always on; rate limiting runs in a `shadow` mode (counts and logs without blocking) until enforced per environment, so the kit can be merged "dark" without a disable switch on any protective control.

The work pilots in `ows-royalties` and is intended to be a reusable **reference pattern** the same-scaffold `ows-abacus-*` services (and other teams) adopt.

## 2. Background & Motivation

The Accounting Flask services currently lack standard API hardening: today they have only health checks (`/hello/`), a uWSGI request path, and (royalties only) a dev-only CORS hook. Absent everywhere: rate limiting, compression, security headers, circuit breakers, body limits, and app-level graceful shutdown. Outbound timeouts exist only via `OwsClient` in some services; raw `requests.post` calls (e.g. to MWAA/Airflow) have none.

The services run as **Python / Flask under uWSGI (1 process × 15 threads), behind the HAProxy edge (`ows-grass`)**, all generated from one Orchard scaffold (`owsrequest` / `owsresponse` / `owslogger` / `abacus_common_logic`). The hardening capabilities are expressed in Flask idioms (`@after_request` hooks, Flask extensions, uWSGI flags, breaker wrappers around the outbound clients); the 1-process × 15-thread model drives several design choices, notably the custom circuit breaker.

## 3. Goals / Non-Goals

**Goals**
- Comprehensive, production-grade API hardening, idiomatic to Flask/uWSGI.
- Granular, per-endpoint / per-resource control for rate limits, breakers, and timeouts.
- A production-ready, exemplary reference others can adopt (full error handling, complete tests, type safety, a clean extraction seam).
- Ship safely: protective middleware is always on with safe defaults; rate limiting rolls out via a `shadow` → `enforce` mode per environment (never a plain on/off switch).
- A real seam to extract a shared `owshardening` library later.

**Non-Goals**
- Prod CORS / TLS termination — owned by the HAProxy edge.
- A service mesh. (Tenant-aware limiting and breakers around specific logical downstreams must live in-app regardless.)
- Dynamic/runtime config (Split.io). Config is fully static; a function seam keeps a Split override a later drop-in.
- Changing business logic, route contracts, or the `owsresponse` envelope.
- Bumping Orchard-lib versions (release tooling owns that).

## 4. System Context

The six in-scope services are HTTP Flask APIs on `:8080` (uWSGI), behind the shared HAProxy edge which terminates TLS and owns prod CORS. The hardening kit sits between the edge and the application's blueprints/connectors.

```mermaid
flowchart LR
  client[Clients / internal services] --> edge[HAProxy edge\nows-grass\nTLS, CORS, header strip]
  edge --> app[ows-royalties\nFlask / uWSGI 1x15]
  subgraph app[ows-royalties - Flask / uWSGI 1x15]
    hk[core/hardening kit]
    bp[blueprints]
    conn[connectors]
    hk --> bp
    bp --> conn
  end
  conn -->|call_downstream guards| mwaa[(Airflow / MWAA)]
  conn --> coll[(ows-collaborator)]
  conn --> acct[(ows-abacus-account)]
  conn --> sf[(Snowflake)]
  conn --> s3[(S3)]
```

**Affected services / repos:** `ows-royalties` (pilot), then `ows-abacus-account`, `ows-abacus-event`, `ows-abacus-legacy-sync`, `ows-abacus-state`, `ows-abacus-worksheet`.

**External integrations the kit guards (royalties):** `ows-collaborator` (owsrequest), `ows-abacus-account` (`OwsClient`/httpx), Airflow/MWAA (raw `requests`), Snowflake (driver). S3/boto3 keeps its native retries.

## 5. Architecture

A self-contained `core/hardening/` package with one entry point and single-phase init. This is the seam: when extracted to a shared `owshardening` lib, the package moves wholesale and the call site is unchanged.

```
core/hardening/
  __init__.py          # init_hardening(app, config) — thin orchestrator + validate_config
  policies.py          # Resource, RateCategory enums; DownstreamPolicy; DOWNSTREAMS; RATE_* maps
  breaker.py           # custom CircuitBreaker (lock-state-only) + CircuitBreakerError
  downstream.py        # ADAPTERS per Resource + GUARDS + call_downstream + DownstreamServerError
  client_ip.py         # ProxyFix wiring + principal/service/ip key function
  rate_limit.py        # flask-limiter + @rate_category + tier-aware limit callable
  compression.py       # flask-compress
  security_headers.py  # @after_request hook + @no_store marker
  errors.py            # CircuitBreakerError -> 503, RateLimitExceeded -> 429 (generic bodies)
  observability.py     # ddtrace dogstatsd metrics + structured logs
  readiness.py         # dedicated short-timeout engine + cached is_ready()
  README.md            # adopter guide + runbook
```

`init_hardening(app, config)` is a thin orchestrator over independently-testable `setup_*` functions, matching the scaffold's `setup_db`/`setup_cors` idiom. It is invoked as the **last** statement in `create_app()` (after blueprints register, since the rate limiter binds per-route) and after the identity `before_request` (so the limiter's key function sees `g.user_details`).

## 6. Configuration Model

Configuration is **typed Python, fully static** (validated at import via dataclass `__post_init__`), not YAML. The two concerns bind to different things:

- **Rate limiting binds to routes (code)** → a `@rate_category(RateCategory.X)` decorator + a small typed category map. A YAML map keyed by endpoint name duplicates routing knowledge and fails open silently on a view rename.
- **Timeouts + breakers bind to downstream resources (data)** → one typed `Resource → DownstreamPolicy` registry with cross-field invariants.

```python
class Resource(Enum):
    OWS_COLLABORATOR = "ows-collaborator"; OWS_ABACUS_ACCOUNT = "ows-abacus-account"
    AIRFLOW_MWAA = "airflow-mwaa"; SNOWFLAKE = "snowflake"

@dataclass(frozen=True)
class DownstreamPolicy:
    connect_timeout: float; read_timeout: float
    fail_max: int            # CONSECUTIVE failures that trip (deliberate; not a rolling window)
    reset_timeout: float
    success_threshold: int = 2   # consecutive half-open successes to re-close (anti-flap)
    retries: int = 0             # idempotent calls only; blocked on long reads
    # __post_init__: reset_timeout >= read_timeout; no retries when read_timeout >= 30
```

Env vars cover only genuinely-environmental values (`RATELIMIT_MODE` = shadow|enforce, `RATELIMIT_STORAGE_URI`, `RATELIMIT_DEFAULT`, `PROXYFIX_X_FOR`) — there are no enable/disable flags for the protective middleware, and fixed routes stay constants (see [DECISION_LOG.md](DECISION_LOG.md) D7). The rationale for typed-over-YAML is in [DECISION_LOG.md](DECISION_LOG.md) (D1).

## 7. Inbound Capabilities

| Capability | Mechanism | Notes |
|---|---|---|
| **Rate limiting** | `flask-limiter` (storage-agnostic, `sliding-window-counter`) | Per-endpoint `@rate_category` (callable limits) applied to **every key** — user `identity_id`, `requestor-service`, or client IP (no separate service tier); `memory://` now, `redis://` later (fail-open). Each category carries a **burst + sustained** pair (the native equivalent of a token bucket). **Calibrated from 7d prod Datadog APM** (per-principal, `identity_id`-keyed; the `graphql-abacus` gateway forwards end-user identity across ~900 users): READ `150/s`+`1000/min`, WRITE `30/s`+`200/min`, EXPENSIVE `25/min`. Reads fan out on page load to ~106 req/s then settle; sustained minutes are usually 200–300 with a rare 1342/min bulk spike, so the caps catch abuse, not normal load — real backend protection comes from the breaker + per-resource timeouts + the tight `expensive` limit. Ships in `shadow`, confirmed against would-be-429 counts before `enforce`. **Full calibration + API-landscape comparison → [RATE_LIMITS.md](RATE_LIMITS.md).** |
| **Compression** | `flask-compress` | gzip JSON, level 5, `min_size=500`, `COMPRESS_STREAMS=False` so `stream_with_context` contract exports are left uncompressed (no per-view exempt API in 1.24). |
| **Security headers** | `@after_request` | App owns `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`, `COOP`, and `Cache-Control` (`no-cache` default; `no-store` enforced on mutating methods + a `@no_store` marker). HSTS and `Server` stripping are the edge's job. |
| **Body-size limit** | `MAX_CONTENT_LENGTH` (2 MB) | Verified JSON-only: uploads/downloads are presigned-direct-to-S3 (no file bytes through Flask), bulk arrays bounded by `OWS_BATCH_LIMIT`; 2MB is ~10x headroom and bounds per-thread memory under 15 threads; 413 via the existing handler. |
| **Graceful shutdown** | uWSGI flags | `--harakiri 90`, `--max-requests` + delta, `--reload-mercy`. |

```mermaid
sequenceDiagram
  participant C as Client
  participant PF as ProxyFix
  participant ID as identity before_request
  participant RL as rate limiter
  participant V as view
  participant SH as security headers / compress
  C->>PF: request (XFF, identity headers from edge)
  PF->>ID: corrected remote_addr
  ID->>RL: g.user_details set
  RL-->>C: 429 (generic) if over limit
  RL->>V: else dispatch
  V->>SH: response
  SH-->>C: headers + gzip
```

## 8. Outbound Resilience

The core abstraction is `call_downstream(resource, call, *, idempotent=False)`: it dispatches to a per-`Resource` **adapter** (which applies that client's native timeout and classifies failures) and wraps it in the resource's circuit breaker + a bounded retry for idempotent calls.

Two verified realities shape this:

1. **The four clients have incompatible calling conventions.** `owsrequest.process` builds a `requests.Request` that accepts no `timeout` and never passes one to `session.send`; the Snowflake executor is a context manager with a construction-time int; httpx needs an `httpx.Timeout`. A uniform `fn(timeout=tuple)` therefore cannot work — each `Resource` carries its own adapter + failure classifier (e.g. Snowflake counts `OperationalError` but passes `ProgrammingError`; HTTP adapters raise `DownstreamServerError` on a returned 5xx because `owsrequest`/`requests` return 5xx as a value rather than raising).

2. **Off-the-shelf breakers are unusable here.** `pybreaker` holds its lock during the wrapped I/O, serializing every call through a breaker to concurrency 1 under uWSGI's 1-process × 15-thread model (benchmarked). The no-serialization libraries lack a half-open gate and state hooks. So the kit ships a **small custom circuit breaker** whose lock guards only state transitions — I/O runs outside it — with an atomic single half-open probe and an `on_state_change` callback. The design was prototyped and stress-tested (15-thread concurrency, exactly-one-probe, gate-leak, classification). See [DECISION_LOG.md](DECISION_LOG.md) (D2, D3).

```mermaid
sequenceDiagram
  participant V as view
  participant CD as call_downstream
  participant B as CircuitBreaker
  participant A as Resource adapter
  participant D as downstream
  V->>CD: call_downstream(SNOWFLAKE, fn)
  CD->>B: breaker.call(...)
  alt OPEN (not recovered)
    B-->>V: CircuitBreakerError -> 503 (generic)
  else CLOSED / single HALF_OPEN probe
    B->>A: run outside the lock
    A->>D: native timeout, classify
    D-->>A: result / failure
    A-->>B: raise on failure (classifier)
    B-->>V: result / re-raise
  end
```

## 9. Readiness & Health

- `/hello/` stays **liveness** (unchanged).
- `/ready/` is added to the base blueprint and does a shallow `SELECT 1` on a **dedicated short-timeout engine** (`connect_timeout=1`, `read_timeout=2`, `NullPool`, `pool_pre_ping=False`) — the shared engine bakes 10/30s timeouts plus a pre-ping that would defeat a statement cap. A `MAX_EXECUTION_TIME(2000)` hint is belt-and-suspenders (verify the MySQL flavor honors it). The result is cached ~2–5s so probe storms don't add DB load; the body is a static minimal `ok`/`unavailable`.
- Both paths are excluded from access logging via a list (`HEALTH_CHECK_PATHS`).

## 10. Observability

Metrics are emitted via the dogstatsd client vendored in `ddtrace` (the `datadog` package is not installed). Counters for rate-limit 429s (by category / key-type), body-limit 413s, and downstream failures; a per-state breaker gauge (`hardening.breaker.state` tagged `state:closed|halfopen|open`) so "how many pods are open" is answerable. **Hard guardrail:** tags and structured-log fields use only the key **type** (`principal`/`service`/`ip`) — never the key value (PII + Datadog cardinality). Sentry is reserved for anomalies (e.g. a breaker stuck open). A starter Datadog dashboard + monitors ship in-repo.

## 11. Security Model & Edge Precondition

Per-principal rate limiting depends on identity the app reads from request headers (`Orchard-Identity-Id`, `Orchard-User-Id`, `Orchard-Requestor-Service`) plus the client IP from `X-Forwarded-For`. These headers are **client-settable**, so they are only trustworthy if the HAProxy edge strips and re-injects them.

> **Blocking precondition (tracked as [ACC-10614](https://theorchard.atlassian.net/browse/ACC-10614)):** the edge must strip/re-inject `X-Forwarded-For` and `Orchard-Identity-Id` / `Orchard-User-Id` / `Orchard-Requestor-Service`, and `PROXYFIX_X_FOR` must be set to the actual proxy hop count per environment. Without this, per-principal limits are spoofable (mint unlimited keys) or weaponizable (set a victim's id to burn their bucket). The kit ships with rate limiting in `shadow` mode (counts/logs, never blocks) so it can merge dark; **prod rate-limit enforcement waits on this ticket.**

Client-facing 429/503/413 bodies are generic (no `Resource` name, no policy string); the resource/category go only to internal logs/metrics.

## 12. Testing Strategy

`pytest` with the existing `respx` / `moto` / `freezegun`. Because the suite caches one app via `@lru_cache` and the limiter/breaker hold process-global state, three isolation prerequisites apply: a hardening-scoped autouse fixture that resets breaker + limiter + readiness cache; features default **off** under the suite's `TestConfig` so the existing suite is unaffected; and a non-cached `make_app` for config-variation tests. Behavior coverage includes: breaker concurrency + single-probe + gate-leak + classification; 5xx-returned-trips-breaker; `CircuitBreakerError → 503` and `RateLimitExceeded → 429` generic bodies; per-key limits (a `service` key is bucketed separately but gets the same finite limit); compression streaming-stays-uncompressed; `no-store` enforcement; readiness 503-while-live; policy-import invariants; and the observability tag-cardinality guardrail.

## 13. Rollout Plan

1. **Pilot — `ows-royalties`:** implement the kit end to end (Epic [ACC-10607](https://theorchard.atlassian.net/browse/ACC-10607), tickets 01–06); rate limiting `enforce` in dev, `shadow` under `TestConfig`, shadow → enforce per env.
2. **Edge precondition — [ACC-10614](https://theorchard.atlassian.net/browse/ACC-10614):** confirm header strip/re-inject + `PROXYFIX_X_FOR` per env (gates prod rate-limit enablement).
3. **Template — five abacus services:** copy `core/hardening/`, the wiring, uWSGI flags, config keys, and `/ready/`. Per-service adaptation is limited to the `Resource` set + `DOWNSTREAMS` table + the `call_downstream` wrap points. Note `ows-abacus-account` is on Flask 2.3 — verify extension compatibility or bump to 3.x first.
4. **Extract `owshardening`:** once stable across all six, lift the package into a versioned lib.

**Staged enable order** (per env, bake/observe each before the next): security headers → compression → rate limiting → breakers.

## 14. Risks & Open Actions

- **Edge header-trust + ProxyFix hop count** — blocking, [ACC-10614](https://theorchard.atlassian.net/browse/ACC-10614).
- **Snowflake statement timeout** — the in-use executor is an Orchard lib we must not edit; a repo-local subclass forwards the timeout if feasible, else Snowflake ships breaker-only and an upstream ticket is filed.
- **MySQL flavor** — confirm `MAX_EXECUTION_TIME` is honored (else MariaDB `max_statement_time`).
- **Per-pod state** — `memory://` limits and breaker `fail_max` are per-pod; expensive/low-limit tiers need Redis for a real cross-pod guarantee (storage-agnostic seam already in place).
- **Flask 2.3 on `ows-abacus-account`** — extension compatibility during rollout.

## 15. Decisions

Key, non-obvious decisions (each justified through four review rounds) are recorded in [DECISION_LOG.md](DECISION_LOG.md): typed-Python config over YAML (D1), a custom circuit breaker over pybreaker/library options (D2, D3), per-resource adapters over a uniform call signature (D4), and the edge header-trust precondition (D5).

## 16. References

- **Jira:** Epic [ACC-10607](https://theorchard.atlassian.net/browse/ACC-10607); tickets [ACC-10608](https://theorchard.atlassian.net/browse/ACC-10608) … [ACC-10614](https://theorchard.atlassian.net/browse/ACC-10614).
- **Repositories:** [theorchard/ows-royalties](https://github.com/theorchard/ows-royalties), and the `ows-abacus-*` services.
- **Decision Log:** [DECISION_LOG.md](DECISION_LOG.md).
- The detailed task-by-task implementation plan is maintained by the engineer locally (TDD steps) and is not part of this doc set.