# Security Incident Response Playbook ## Purpose & Audience This playbook defines the procedures for detecting, triaging, containing, and recovering from security incidents affecting the Coda platform. It is intended for on-call engineers, platform team leads, and security responders. For general operations, see [platform-operations.md](platform-operations.md). For the security architecture, see [security.md](../../compliance/security.md). **Last reviewed:** 2026-04-10 --- ## Severity Levels | Severity | Definition | Examples | Response Time | Notification | | -------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | -------------------- | ---------------------------- | | **SEV-1** (Critical) | Active data breach, complete service compromise, encryption key exposure | Master key exfiltrated, cross-tenant data leakage confirmed, DB credentials compromised | Immediate (< 15 min) | VP Engineering + Legal + DPO | | **SEV-2** (High) | Confirmed unauthorized access, privilege escalation, audit trail integrity compromise | Super admin grant without authorization, audit log tampering detected, OAuth token theft | < 1 hour | Engineering Lead + Security | | **SEV-3** (Medium) | Suspicious activity, potential vulnerability exploitation, anomalous access patterns | Spike in permission denials, repeated failed auth from single IP, rate limit bypass attempt | < 4 hours | Platform Team | | **SEV-4** (Low) | Minor policy violation, single failed exploit attempt, configuration drift | Single 403 from unexpected source, CSP violation report, dependency CVE disclosure | Next business day | Ticket created | --- ## Phase 1: Detection ### Automated Detection Sources | Source | What It Detects | Alert Channel | | ---------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------- | | **Sentry** | Unhandled exceptions, error spikes, new error types | Sentry alerts → Slack | | **Datadog APM** | Latency anomalies, error rate spikes, AppSec events | Datadog monitors → PagerDuty | | **Datadog AppSec** | SQL injection attempts, XSS attempts, known attack patterns | Datadog Security Signals → Slack | | **Audit Log Analysis** | Spike in denials, cross-tenant access attempts, super admin abuse | **Not yet automated** — manual review of `audit_logs` table | | **Rate Limit Alerts** | Sustained 429 responses, distributed attack patterns | Application logs (pino) | | **AWS CloudTrail** | KMS key usage anomalies, IAM role assumption, S3 access patterns | CloudTrail → CloudWatch → SNS | ### Manual Detection Triggers - User reports unexpected access denial or data they shouldn't see - Code review reveals security vulnerability - Third-party disclosure (CVE, Auth0 advisory, AWS security bulletin) - Penetration test finding ### Detection Gaps (Phase 1) These are not yet automated and require manual monitoring: 1. **Audit log anomaly detection** — no automated alerting on permission denial spikes 2. **Super admin activity monitoring** — no alerts on unusual super admin patterns 3. **Concurrent session detection** — no alerts on user sessions from multiple geolocations 4. **Crypto-shredding verification** — no automated check that erased users' PII is truly unreadable --- ## Phase 2: Triage ### Initial Assessment Checklist When a potential incident is detected: 1. **Scope determination** - [ ] Is this affecting a single user, a single tenant, or multiple tenants? - [ ] Is the attack ongoing or was it a past event? - [ ] Is data at rest compromised, data in transit intercepted, or access control bypassed? 2. **Data classification** - [ ] Does the incident involve PII (identity data, IP addresses, user agents)? - [ ] Does the incident involve credentials (OAuth tokens, session tokens, API keys)? - [ ] Does the incident involve encryption keys (AES key, HMAC secret, audit master key)? - [ ] Does the incident involve financial data (royalties, credits)? 3. **Assign severity** using the table above. 4. **Assign incident commander** (IC): - SEV-1/2: Engineering Lead or designated security responder - SEV-3/4: On-call engineer 5. **Open communication channel:** - SEV-1/2: Dedicated Slack channel `#incident-{date}-{short-name}` - SEV-3/4: Thread in `#platform-alerts` --- ## Phase 3: Containment ### Immediate Actions by Incident Type #### A. Encryption Key Compromise **Applies to:** `AUDIT_MASTER_KEY`, `CODA_DB_IDENTITY_AES_KEY`, `CODA_DB_IDENTITY_HMAC_SECRET` 1. **Rotate the compromised key immediately:** - Generate new key: `node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"` - Update in AWS Secrets Manager (or environment variable source) - Deploy with new key value 2. **Assess blast radius:** - `AUDIT_MASTER_KEY`: All audit PII is derivable. See [platform-key-rotation.md](platform-key-rotation.md) for re-encryption procedure. - `CODA_DB_IDENTITY_AES_KEY`: All encrypted identity copies and OAuth tokens are decryptable. - `CODA_DB_IDENTITY_HMAC_SECRET`: Identity hashes can be brute-forced from known identity ID space. 3. **Re-encrypt affected data** using the key rotation runbook. 4. **Revoke all sessions** if credential theft is suspected: ```sql UPDATE sessions SET revoked_at = NOW() WHERE revoked_at IS NULL; ``` 5. **Notify affected tenants** if PII was exposed (GDPR Art. 34 — 72-hour window). #### B. Cross-Tenant Data Leakage 1. **Enable shadow mode** (kill switch) to stop enforcement changes while investigating: - Set `SHADOW_MODE=true` and redeploy 2. **Identify the leak path:** - Query audit logs: `SELECT * FROM audit_logs WHERE tenant_id = '{affected_tenant}' AND outcome = 'success' ORDER BY created_at DESC LIMIT 100` - Check for queries missing `tenantId` filter - Check cache keys for cross-tenant pollution: `redis-cli KEYS "perm:*"` 3. **Flush all permission caches:** ```bash redis-cli KEYS "perm:*" | xargs redis-cli DEL redis-cli KEYS "access:*" | xargs redis-cli DEL ``` 4. **Suspend affected tenant** if active exploitation: ```sql UPDATE tenants SET status = 'suspended' WHERE id = '{tenant_id}'; ``` This triggers automatic cache purge for the tenant. 5. **Fix the query/code path** before re-enabling enforcement. #### C. Unauthorized Super Admin Activity 1. **Revoke the super admin grant immediately:** ```sql UPDATE super_admins SET revoked_at = NOW(), revoked_by = '{your_user_id}' WHERE user_id = '{suspect_user_id}' AND revoked_at IS NULL; ``` 2. **Flush super admin cache:** ```bash redis-cli DEL "access:superadmin:{user_id}" ``` 3. **Review audit trail:** ```sql SELECT * FROM audit_logs WHERE is_super_admin = true AND user_id = '{suspect_user_id}' ORDER BY created_at DESC LIMIT 500; ``` 4. **Check for impersonation tokens issued:** - Review logs for `action = 'session.impersonate'` with the suspect user ID - Revoke any active impersonation sessions 5. **Review permission changes made by this user:** ```sql SELECT * FROM permission_change_logs WHERE changed_by = '{suspect_user_id}' ORDER BY created_at DESC; ``` #### D. Rate Limit Bypass / DDoS 1. **Verify rate limit infrastructure:** - Check Redis connectivity: `redis-cli PING` - Check rate limit keys exist: `redis-cli KEYS "ratelimit:*" | head -20` - If Redis is down, rate limiting fails closed (429 for all requests) 2. **Add IP to blocklist** at ALB/HAProxy level (not application level): - Contact infrastructure team for WAF rule update 3. **If application-level only:** Temporarily reduce rate limits: - `API_RATE_LIMIT_MAX=50` (from 100) - `STREAM_RATE_LIMIT_MAX=5` (from 10) #### E. Auth0 / JWT Compromise 1. **If Auth0 is compromised:** - Rotate Auth0 signing keys in the Auth0 dashboard - Clear JWKS cache (restart server pods) - Force all users to re-authenticate 2. **If a specific JWT is stolen:** - JWTs are short-lived (Auth0 default: 1 hour) - Revoke the user's session in the platform service - If long-lived: contact Auth0 to invalidate the grant --- ## Phase 4: Eradication After containment: 1. **Identify root cause:** - Code vulnerability → fix and deploy - Configuration error → correct and document - Credential leak → rotate and re-encrypt - Dependency vulnerability → update and patch 2. **Verify fix:** - Write a regression test for the specific attack vector - Run full test suite: `pnpm test` - Deploy to staging and verify containment is effective 3. **Verify no persistence:** - Check for unauthorized DB modifications - Check for unauthorized role/permission changes - Verify audit log integrity (no gaps in timestamps) --- ## Phase 5: Recovery 1. **Restore normal operations:** - Re-enable enforcement mode (disable shadow mode) if it was activated - Remove temporary rate limit reductions - Unsuspend tenants if they were suspended for investigation - Re-enable disabled features 2. **Verify service health:** - Permission check latency within SLO (p99 < 5ms cached, < 50ms cold) - Audit log completeness (no gaps) - Cache hit ratio > 90% - Zero false denials 3. **Communication:** - SEV-1/2: User-facing incident report within 48 hours - GDPR breach: Supervisory authority notification within 72 hours (Art. 33) - CCPA breach: Consumer notification "without unreasonable delay" --- ## Phase 6: Post-Incident 1. **Post-mortem** (required for SEV-1/2, recommended for SEV-3): - Timeline of events - Root cause analysis (5 Whys) - What went well, what didn't - Action items with owners and deadlines 2. **Update documentation:** - Update this playbook if procedures were insufficient - Update [threat-model.md](../../compliance/threat-model.md) with new threat scenarios - Update [compliance-matrix.md](../../compliance/compliance-matrix.md) if compliance status changed 3. **Improve detection:** - Add alerts for the attack pattern that was missed - Add invariant tests if access control assumptions were violated --- ## GDPR Breach Notification Timeline If the incident involves personal data of EU residents: | Deadline | Action | Owner | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | | T+0 | Incident detected. IC assigned. | On-call engineer | | T+24h | Scope and impact assessed. Data subjects identified. | IC + platform team | | T+48h | Internal report to Legal and DPO. | IC | | T+72h | **Supervisory authority notification** (GDPR Art. 33). Must include: nature of breach, categories of data, approximate number of records, name of DPO, likely consequences, measures taken. | DPO + Legal | | T+72h+ | **Data subject notification** if high risk (GDPR Art. 34). Plain language description of breach and recommended protective measures. | DPO + Legal | --- ## Contact List | Role | Responsibility | Escalation | | ------------------ | ----------------------------------------------------------------- | ------------------ | | On-call engineer | First responder. Triage and initial containment. | Platform team lead | | Platform team lead | Incident commander for SEV-2+. Technical decisions. | VP Engineering | | VP Engineering | Escalation for SEV-1. Resource allocation. | CTO | | Legal | GDPR/CCPA notification compliance. | External counsel | | DPO | Data protection impact assessment. Supervisory authority liaison. | Legal | --- ## Maintaining This Document - **Review cadence:** After every SEV-1/2 incident, or quarterly. - **Test cadence:** Conduct a tabletop exercise annually using Scenario 1-3 from the threat model. - **Owner:** Platform team lead.