# Platform Service Operations Runbook

## Configuration

### Required environment variables

| Variable                       | Purpose                                             | Required   |
| ------------------------------ | --------------------------------------------------- | ---------- |
| `PORT`                         | HTTP listen port (default 8082)                     | No         |
| `CODA_DB_HOST`                 | Aurora MySQL hostname                               | Yes (prod) |
| `CODA_DB_USER`, `CODA_DB_PASS` | Database credentials                                | Yes (prod) |
| `CODA_DB_DATABASE`             | Database name (default `coda`)                      | No         |
| `REDIS_URL`                    | ElastiCache URL for permission cache + audit buffer | Yes (prod) |
| `AUDIT_MASTER_KEY`             | 256-bit hex key for audit PII encryption            | Yes (prod) |
| `SENTRY_DSN`                   | Error tracking                                      | Yes (prod) |
| `SHADOW_MODE`                  | Log-only permission checks (default `true`)         | No         |

### Pipeline feature flags

All default to `false` in Phase 1 (stub implementations):

| Flag                       | Pipeline Step              |
| -------------------------- | -------------------------- |
| `PIPELINE_STEP_5_ENABLED`  | Consent + DPA checks       |
| `PIPELINE_STEP_6_ENABLED`  | IP allowlist enforcement   |
| `PIPELINE_STEP_12_ENABLED` | Resource scope checks      |
| `PIPELINE_STEP_14_ENABLED` | Data classification checks |
| `PIPELINE_STEP_15_ENABLED` | Step-up auth requirements  |

---

## Health Checks

### `GET /health`

Always returns 200. Used by container health checks and load balancer probes.

```json
{ "status": "ok" }
```

### `GET /health/ready`

Returns the status of dependencies.

| Status     | HTTP | Meaning                                   |
| ---------- | ---- | ----------------------------------------- |
| `ready`    | 200  | Cache and DB are operational              |
| `degraded` | 503  | Cache is unavailable (DB fallback active) |

```json
{
  "status": "ready",
  "cache": "ok"
}
```

---

## Cache Management

### Cache key reference

| Pattern                                | TTL   | Purpose                        |
| -------------------------------------- | ----- | ------------------------------ |
| `perm:{tenantId}:{userId}:effective`   | 30s   | User's effective permissions   |
| `perm:{tenantId}:{userId}:deny`        | 30s   | User's deny overrides          |
| `perm:{tenantId}:role:{roleId}`        | 15min | Role's resolved permissions    |
| `perm:{tenantId}:plan`                 | 1hr   | Plan feature keys              |
| `perm:{tenantId}:tenant_status`        | 30s   | Tenant active/suspended status |
| `perm:{tenantId}:{userId}:user_status` | 30s   | User active/suspended status   |
| `perm:superadmin:{userId}`             | 5min  | Super admin level              |
| `perm:{tenantId}:conditions:{perm}`    | 15min | ABAC policy conditions         |
| `access:{tenantId}:{userId}:groups`    | 30s   | User's group memberships       |
| `access:{tenantId}:dpa_valid`          | 5min  | DPA validity flag              |

### Manual cache invalidation

**Flush all caches for a tenant** (e.g., after bulk permission change):

```bash
redis-cli --scan --pattern "perm:{tenantId}:*" | xargs redis-cli DEL
redis-cli --scan --pattern "access:{tenantId}:*" | xargs redis-cli DEL
```

**Flush a specific user's caches:**

```bash
redis-cli DEL "perm:{tenantId}:{userId}:effective"
redis-cli DEL "perm:{tenantId}:{userId}:deny"
redis-cli DEL "perm:{tenantId}:{userId}:user_status"
```

**Flush super admin cache:**

```bash
redis-cli DEL "perm:superadmin:{userId}"
```

### Cache failure behavior

When Redis is unreachable, the service falls through to Aurora DB for all reads. This degrades latency from ~5ms to ~50ms but preserves correctness. Rate limit counters fail closed (429). Audit writes fall back to pino JSON logs.

---

## Audit Monitoring

### Audit buffer health

The audit buffer lives in Redis at key `audit:buffer`. Monitor its length:

```bash
redis-cli LLEN audit:buffer
```

| Buffer depth | Status   | Action                                                               |
| ------------ | -------- | -------------------------------------------------------------------- |
| < 100        | Normal   | None                                                                 |
| 100-1,000    | Elevated | Monitor -- drainer may be slow                                       |
| 1,000-10,000 | Warning  | Check drainer health, DB connection                                  |
| > 10,000     | Critical | Drainer is stopped or DB is unreachable. Buffer will grow unbounded. |

### Drainer behavior

The audit drainer runs every 1 second, popping up to 1,000 entries per tick from the Redis buffer and batch-inserting into the `audit_logs` table.

**Symptoms of drainer issues:**

- `audit:buffer` length growing continuously
- "Audit drainer: batch insert failed" in logs
- Missing audit entries in `QueryAuditLog` results

**Recovery:** The drainer auto-recovers when the DB becomes available. Entries buffered in Redis are preserved. If Redis was also down, entries were logged via pino as fallback -- these can be replayed manually.

### Crypto-shredding verification

To verify a user's audit data is properly shredded after key deletion:

1. Query audit entries by `userIdHash` (HMAC of the userId)
2. Verify PII fields (`userId`, `ipAddress`, `userAgent`) show `[redacted]`
3. Verify structural fields (`action`, `resource`, `outcome`, `createdAt`) are readable

---

## Shadow Mode

### How it works

When `SHADOW_MODE=true` (default in Phase 1), the permission pipeline runs but results are not enforced. The platform service logs each check result but the consuming middleware does not block requests on DENIED outcomes.

### Validating shadow mode

1. Check application logs for `permission.checked` events
2. Compare shadow denials against actual access patterns
3. Look for unexpected DENIED outcomes that would break existing workflows

### Transitioning to enforcement

1. Validate shadow mode results over a representative traffic period
2. Set `SHADOW_MODE=false`
3. Monitor for 403 errors in the consuming services
4. Roll back by setting `SHADOW_MODE=true` if issues arise

---

## Troubleshooting

### Permission check returning unexpected DENIED

**Step 1: Identify which pipeline step denied.**

Check the response `step` field or the audit log entry. Common causes by step:

| Step | Denial reason           | Fix                                                                                                         |
| ---- | ----------------------- | ----------------------------------------------------------------------------------------------------------- |
| 3    | Tenant suspended        | Reactivate tenant via `ReactivateTenant` RPC                                                                |
| 4    | User not active         | Check `TenantUser.status` -- may be `invited`, `suspended`, or `deactivated`                                |
| 7    | Feature not in plan     | Check `PlanFeature` rows for the tenant's plan. The permission's module slug must match a plan feature key. |
| 8    | Deny override active    | Check `UserPermission` rows with `type=deny` for this user. Remove if unintended.                           |
| 11   | Not in any role         | Check `UserRole` assignments. Verify the role has the required permission via `RolePermission`.             |
| 13   | Policy condition failed | Check `PolicyCondition` rows. Verify the request context contains the required fields.                      |

**Step 2: Check cache staleness.**

If a permission change was just made, the cache may still hold stale data. Wait 30 seconds (effective permission TTL) or manually invalidate:

```bash
redis-cli DEL "perm:{tenantId}:{userId}:effective"
redis-cli DEL "perm:{tenantId}:{userId}:deny"
```

### Redis connection failures

**Symptoms:** `Permission cache: Redis error` in logs, elevated response latency.

**Impact:** Permission checks fall through to DB (slower but correct). Rate limits fail closed (429). Audit writes fall back to pino logs.

**Fix:**

- Check ElastiCache endpoint accessibility from the Fargate task VPC
- Verify Redis URL format in `REDIS_URL` env var
- The cache auto-reconnects with exponential backoff (max 5 retries, 200ms-2s)

### Database connection failures

**Symptoms:** All permission checks return DENIED (fail-closed). `Permission resolution failed` errors in logs.

**Impact:** Complete denial of service for all users.

**Fix:**

- Check Aurora MySQL connectivity
- Verify database credentials
- Check connection pool health

### Audit master key issues

**Symptoms:** `Audit writer: encryption failed` in logs.

**Fix:**

- Verify `AUDIT_MASTER_KEY` is a 64-character hex string (256 bits)
- In dev without the key, a random key is generated (data not recoverable across restarts)
- In production, the key MUST be persistent across deployments

### Tenant suspension cascade

When a tenant is suspended:

1. All permission checks for the tenant return DENIED (step 3)
2. ALL cache keys for the tenant are flushed via prefix scan
3. All active sessions should be revoked

To reactivate:

1. Call `ReactivateTenant` RPC
2. The cache rebuild happens automatically on the next permission check
3. Users may need to re-authenticate

---

## Monitoring Checklist

| Metric                       | Threshold                       | Alert                                 |
| ---------------------------- | ------------------------------- | ------------------------------------- |
| Permission check p99 latency | > 50ms (cached), > 200ms (cold) | Warning                               |
| Cache hit ratio              | < 90%                           | Warning (expected > 95% steady state) |
| Audit buffer depth           | > 10,000                        | Critical                              |
| DENIED outcomes (unexpected) | Spike > 10x baseline            | Warning                               |
| Redis connection errors      | Any                             | Warning                               |
| DB connection errors         | Any                             | Critical                              |