# Caller Analysis and Rollout Safety

> ⚠️ **Adding an enforcing PP check to a live endpoint will reject requests from any
> caller that cannot send a valid JWT.** Always follow this three-step rollout:

## Step 1 — Identify callers (Datadog APM, required before any code change)

1. Open [Datadog APM](https://sonymusic-pde.datadoghq.com/apm/home) → find this service.
2. Navigate to each endpoint resource and open the **Dependencies** tab.
3. Set the time window to **1 month** to capture infrequent callers.
4. Classify each caller using **only** the types in this table (do not add rows for caller types not listed here):

| Caller Type | JWT Support | Rollout Approach |
|---|---|---|
| **SPA / Suite Application (frontend)** | ✅ Yes if authenticated via Auth0; ❌ No if unauthenticated session | Confirm JWT presence in the shadow-phase `would_deny` metric before enforcing; unauthenticated SPAs must log in first |
| **Lambda** | ✅ Yes if M2M JWT provisioned; ❌ No if not yet provisioned | File a ticket to provision a dedicated M2M JWT before enforcing |

## Step 2 — Deploy in shadow mode first (`MigrationAuthorizationBackend` wired)

Wire `MigrationAuthorizationBackend` (Phase 2 in the migration templates) and add the side-effect
`is_authorized()` call. The wrapper always allows traffic and emits the Datadog metric
`pp_auth.rollout.would_deny` whenever the real PP decision *would* have denied. Monitor it over
1–2 weeks. Each increment identifies a caller that *would* have been rejected. The metric is tagged
with `environment`, `service_name`, `action`, `resource_type`, `reason`
(`pp_denied` | `unauthenticated` | `exception`), plus the `extra_tags_getter` tags
(`method`, `endpoint`, `has_authorization_header`, and `profile_type` on Flask). Break down by
`reason`, `endpoint`, and `has_authorization_header` to identify which callers lack a valid JWT.
Resolve every would-deny source before enabling enforcement.

> **Tracing individual denials**: the metric gives you counts, not per-request detail. To
> investigate a specific denial, use Datadog APM → find the service → filter by the endpoint in
> question and look for requests that correlate with a metric spike.

## Step 3 — Enforce (follow-up PR)

We can move to the `enforce` step when we're confident that enabling PP will not result in legitimate traffic being denied.

### PP Enforce Readiness Criteria

1. PP [resource policies](https://app.notion.com/p/Writing-Cerbos-resource-policies-dbe5cc1d2ffd4ea6aa70bea59e772e6e) and [derived roles](https://app.notion.com/p/Derived-roles-tenants-tenant-hierarchy-13e84204dbdf48f99ce8cf06209d4836) 
   are defined for the application and downstream services. (`/endpoint-resource-action-pp-authorization-table` skill)
2. Human identities in requests from JWT-enabled applications have derived roles attached in PP. Configured using SettingsV2 or pdp-backfill.
3. Machine identities have dedicated M2M tokens and Principal policies defined in PP.
4. Traffic that does not meet the PP criteria can be authorized by the fallback method, if available.
5. All other traffic is rejected.

The `pp_auth.rollout.would_deny` metric is how you verify the criteria above took effect: each
increment is a request PP *would* deny. You're ready to enforce when the only remaining increments
are traffic you intend to reject (criterion 5) — i.e. every legitimate caller (criteria 1–4) already
passes the PP check or is covered by the fallback.

Then ship the Phase 3 enforce change: swap `MigrationAuthorizationBackend` → `PdpAuthorizationBackend`
and restructure the handler to PP-first + legacy fallback (Templates A/B/C for Flask, FA/FB/FC for
FastAPI). This is a code change, not an env-var flip.