# Infrastructure & Deployment ## Purpose & Audience This document describes the infrastructure, deployment strategy, and monitoring for the platform service. It is intended for engineers provisioning infrastructure, deploying the service, or configuring monitoring. For the service architecture, see [platform architecture](../architecture/platform.md). For operational runbooks, see the [platform runbooks](runbooks/). --- ## Deployment Topology ### Initial: Single Service All 4 domains (access, platform, compliance, growth) run in one ECS Fargate service (`apps/platform`). ``` VPC (shared with server + search) ┌─────────────────────────────────────────────────────┐ │ │ │ ┌─────────────────────────────────────────────┐ │ │ │ Platform Service (Fargate) │ │ │ │ Port 8082 (ConnectRPC: gRPC + Connect JSON)│ │ │ │ │ │ │ │ ┌─────────┐ ┌──────────┐ │ │ │ │ │ access │ │ platform │ │ │ │ │ │ domain │ │ domain │ │ │ │ │ └─────────┘ └──────────┘ │ │ │ │ ┌──────────┐ ┌─────────┐ │ │ │ │ │compliance│ │ growth │ │ │ │ │ │ domain │ │ domain │ │ │ │ │ └──────────┘ └─────────┘ │ │ │ │ │ │ │ │ Health: GET /health, GET /health/ready │ │ │ │ Min tasks: 2 (HA) │ │ │ └─────────────────────────────────────────────┘ │ │ │ │ Aurora MySQL (shared) ◄──── Redis (shared) │ │ S3 (shared bucket) │ └─────────────────────────────────────────────────────┘ ``` ### Future: Decomposed Services When team size warrants, each domain becomes its own Fargate service: | Domain | Port | ECR Repo | Min Tasks | Rationale | | ------------ | ---- | --------------------- | --------- | ----------------------------- | | `access` | 8082 | `ows-coda-access` | 3 | Hot path, higher availability | | `platform` | 8083 | `ows-coda-platform` | 2 | Admin operations | | `compliance` | 8084 | `ows-coda-compliance` | 2 | Audit + privacy | | `growth` | 8085 | `ows-coda-growth` | 1 | Low traffic | Consumers update their ConnectRPC endpoint per domain. Proto imports remain unchanged. --- ## Dependencies | Dependency | Shared/New | Notes | | ------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | Aurora MySQL | Shared | Same cluster as main server. All domain tables in one schema. Table ownership by convention (prefix), not separate databases. | | Redis (ElastiCache) | Shared | Same cluster. Key prefixes by domain: `access:`, `platform:`, `compliance:audit:`, `growth:`. | | S3 | Shared bucket | Partitioned by concern: `/audit/`, `/attachments/`, `/branding/`, `/exports/`. | | RDS Proxy | New (at decomposition) | Connection pooling and multiplexing. Not needed initially. | --- ## Database Connection Strategy ### Current: Direct Connections Each service creates a Prisma connection pool directly to Aurora MySQL. ``` apps/server ──(pool: 20)──┐ apps/search ──(pool: 20)──┼──> Aurora MySQL (shared cluster) apps/platform──(pool: 20)──┘ ``` | Phase | Services x Tasks | Pool/Task | Total Connections | Aurora Headroom | | ----------------- | ----------------- | --------- | ----------------- | -------------------------------------- | | **Today** | 3 svc x 2 tasks | 20 | ~120 | Comfortable (db.r6g.large: ~1,000 max) | | **Decomposed** | 7 svc x 2-3 tasks | 20 | ~300 | Fine, less margin | | **Scaled** (peak) | 7 svc x 3-5 tasks | 20 | ~500+ | Approaching 50% -- time for proxy | The hot path (permission checks) is Redis-cached. Most requests never hit Aurora. DB connections are primarily used for cache misses, write operations, and audit log batch inserts. ### Future: RDS Proxy Adopt RDS Proxy when decomposing into 4 separate services. The trigger is the combination of: 1. **Task churn** -- Fargate deploys and auto-scaling open/close connections rapidly. RDS Proxy absorbs this by multiplexing. 2. **Aurora failover transparency** -- without proxy, all connections drop during failover. RDS Proxy maintains application connections and re-routes transparently. 3. **IAM authentication** -- RDS Proxy supports IAM auth, eliminating DB credentials in env vars. ``` apps/server ──(pool: 20)──┐ apps/search ──(pool: 20)──┤ apps/access ──(pool: 20)──┤ apps/platform ──(pool: 20)──┼──> RDS Proxy ──(pool: 100)──> Aurora MySQL apps/compliance(pool: 20)───┤ (multiplexes) apps/growth ──(pool: 10)──┘ ``` **Migration path:** 1. **Now:** Direct connections. `CODA_DB_HOST=qa-ows-coda-db.theorchard.io` 2. **Decomposition:** Terraform provisions RDS Proxy. `CODA_DB_HOST=qa-ows-coda-proxy.theorchard.io` 3. **IAM auth (optional):** Configure RDS Proxy for IAM authentication. Remove `CODA_DB_USER`/`CODA_DB_PASS` from env vars. **Cost:** ~$0.015/vCPU-hour (~$22/month for db.r6g.large with 2 vCPUs). **Prisma compatibility:** Prisma's MariaDB adapter works with RDS Proxy. Connection pinning occurs during transactions (expected). Prepared statements handled correctly. Only change: swap `CODA_DB_HOST` to the proxy endpoint. --- ## Deployment Strategy **Strategy:** Blue/green with canary. A bad deploy to the platform service denies all access -- this is the highest-risk deployment in the platform. 1. Deploy new version to green environment 2. Route 5% traffic to green (canary) 3. Monitor: permission check success rate, p99 latency, denial rate, error rate 4. If metrics stable for 10 minutes, shift to 50%, then 100% 5. Keep blue environment running for 30 minutes for instant rollback ### Schema Migrations For tables on a shared Aurora instance: | Operation | Approach | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | New tables | Migrate before code deploy (additive, backward-compatible) | | Column additions | Add as nullable first, deploy code, backfill, then enforce NOT NULL | | High-write tables (AuditLog, CreditEntry) | Use `ALGORITHM=INSTANT` for column additions. For index/structural DDL on tables with >1M rows, use `pt-online-schema-change` or `gh-ost`. | | Rollback | Maintain manual DOWN migration scripts for critical changes. Prisma migrations are not easily reversible. | --- ## Terraform Resources New resources in `terraform-infra/qa/ows-coda/`: | Resource | Module | Notes | | ------------------------- | --------------------------------------------- | ---------------------------------------------------------- | | Fargate service | `terraform-fargate` (6.4.2) | One service initially, split at decomposition | | ECR repo | `terraform-ecr` (2.0.1) | `terraform-infra/shared/prod/ecr/repos/ows-coda-platform/` | | Datadog service dashboard | `terraform-datadog//modules/service` (6.16.0) | Per-domain dashboards | | RDS Proxy | (future) | At service decomposition | No new RDS or ElastiCache resources -- shared with existing infrastructure. --- ## Monitoring ### Per-Domain Dashboards Metrics are tagged by domain even while running as one service. | Domain | Key Metrics | | -------------- | ------------------------------------------------------------------------------------------------------------------------ | | **Access** | Permission check latency (p50/p95/p99), cache hit ratio, denial rate, credit consumption rate, rate limit rejection rate | | **Platform** | Invitation acceptance rate, SCIM sync success rate, domain verification status | | **Compliance** | Audit log buffer depth, consent collection rate, privacy request SLA adherence | | **Growth** | Promotion redemption rate, referral conversion rate | ### Alerts | Domain | Alert Condition | | ---------- | ------------------------------------------------------------------------------------------------ | | Access | Error rate > 1%, cache hit ratio < 90%, p99 latency > 10ms | | Compliance | Audit buffer depth > 1,000 (drain stalled), privacy request SLA breach risk (< 7 days remaining) | | Audit log | Datadog log pipeline for structured permission events (separate from application logs) | ### SLOs and Error Budgets | SLO | Target | Error Budget (30-day) | Burn Rate Alert | | --------------------------------- | --------- | ---------------------- | ---------------------------- | | Permission check availability | 99.95% | 21.6 min downtime | 5x burn rate -> page | | Permission check latency (cached) | <5ms p99 | <1% exceed 5ms | 3x burn rate -> warn | | Permission check latency (cold) | <50ms p99 | <5% exceed 50ms | 3x burn rate -> warn | | Audit log completeness | 99.99% | 4.3 min of lost events | Any loss -> page | | Audit log drain latency | <2s p99 | <1% exceed 2s | Buffer > 5,000 -> warn | | Cross-tenant data leakage | 0 events | 0 tolerance | Any event -> page + incident | --- ## Disaster Recovery | Component | RPO | RTO | Mechanism | | -------------------- | ----------------------- | ------ | ------------------------------------------------------------------------ | | **Aurora MySQL** | 5 min | 30-60s | Multi-AZ automated failover. Point-in-time recovery for data corruption. | | **Redis** | N/A (cache) | 30s | Multi-AZ failover. Cache rebuilds from Aurora on miss. | | **S3** | 0 (11 nines durability) | N/A | Cross-region replication for critical archives. | | **Platform service** | N/A (stateless) | <60s | ECS auto-restarts failed tasks. Minimum 2 tasks for HA. | **During Aurora failover (30-60s):** All permission checks deny (fail-closed). The error budget allows 21.6 minutes/month. One failover per month is within budget. **Backup verification:** Weekly automated restore test to non-production. Verify table counts, audit log integrity, and permission resolution on restored data. --- ## Testing Strategy | Level | What | Coverage Target | | ---------------------- | ----------------------------------------------------------- | -------------------------------------------------- | | Named invariant tests | One test per `inv-*` constraint (TDD) | 100% of invariants | | Unit tests | Each pipeline step in isolation | 90%+ line coverage on `permission-resolver.ts` | | Integration tests | Cross-domain flows (invitation -> role -> permission check) | All 3 user story walkthroughs as executable suites | | Cross-tenant isolation | Every tenant-scoped query verified | One test per model with `tenantId` | | Load tests | 1,000 concurrent permission checks/second | Meet <5ms p99 cached, <50ms cold targets | | Chaos tests | Redis kill, Aurora failover, event loss simulation | All Redis failure modes behave as documented |