# Key Rotation Runbook

## Purpose & Audience

This runbook documents procedures for rotating cryptographic keys used by the Coda platform. It covers routine rotation (planned maintenance) and emergency rotation (key compromise). It is intended for platform engineers and on-call responders. For the encryption architecture, see [security.md](../../compliance/security.md). For incident response, see [platform-incident-response.md](platform-incident-response.md).

**Last reviewed:** 2026-04-10

---

## Key Inventory

| Key                            | Purpose                                                      | Storage                                          | Rotation Cadence                        | Blast Radius if Compromised                                         |
| ------------------------------ | ------------------------------------------------------------ | ------------------------------------------------ | --------------------------------------- | ------------------------------------------------------------------- |
| `CODA_DB_IDENTITY_AES_KEY`     | AES-256-GCM encryption of user identity IDs and OAuth tokens | Environment variable (Secrets Manager in prod)   | Annual or on compromise                 | All encrypted identities and OAuth tokens decryptable               |
| `CODA_DB_IDENTITY_HMAC_SECRET` | HMAC-SHA256 hashing of user identity IDs for DB lookup       | Environment variable (Secrets Manager in prod)   | Annual or on compromise                 | Identity hashes can be reversed via brute force from known ID space |
| `AUDIT_MASTER_KEY`             | Per-user audit encryption key derivation via HMAC-SHA256     | Environment variable (Secrets Manager in prod)   | Annual or on compromise                 | All audit log PII decryptable (all per-user DEKs derivable)         |
| AWS KMS CMK                    | Wraps per-user audit encryption DEKs (envelope encryption)   | AWS KMS                                          | Automatic (AWS-managed yearly rotation) | Per-user DEKs that are actively wrapped can be decrypted            |
| Auth0 signing key              | RS256 JWT signature                                          | Auth0 dashboard                                  | Managed by Auth0                        | All JWTs forgeable until JWKS cache expires                         |
| Snowflake private key          | mTLS-style auth to Snowflake                                 | File or env var (`SNOWFLAKE_READER_PRIVATE_KEY`) | Annual                                  | Read-only Snowflake access                                          |

---

## Rotation Procedures

### 1. `CODA_DB_IDENTITY_AES_KEY` Rotation

**Impact:** All `identityEncrypted` values in `users` table and `accessTokenEncrypted`/`refreshTokenEncrypted` in `oauth_connections` must be re-encrypted.

**Downtime:** Zero (dual-key strategy).

#### Procedure

1. **Generate new key:**

   ```bash
   node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
   ```

2. **Store both keys:**
   Add the new key alongside the old one. Use a migration-period configuration:

   ```
   CODA_DB_IDENTITY_AES_KEY=<new_key>
   CODA_DB_IDENTITY_AES_KEY_OLD=<old_key>
   ```

3. **Deploy with dual-key support:**
   During the migration period, the decryption path should:
   - Try decrypting with the new key first
   - Fall back to the old key if GCM auth tag verification fails
   - Re-encrypt with the new key on successful fallback decryption (lazy migration)

   > **Note (Phase 1):** The current codebase does NOT support dual-key decryption. This must be implemented before the first rotation. The change is in `packages/db/src/crypto.ts` — add a `decryptWithFallback(ciphertext, newKey, oldKey)` function.

4. **Run batch re-encryption migration:**

   ```sql
   -- Count records needing re-encryption (still using old key)
   -- After lazy migration has run for 1+ weeks, run a batch job:
   SELECT COUNT(*) FROM users;
   SELECT COUNT(*) FROM oauth_connections;
   ```

   Write a migration script that:
   - Reads each row
   - Decrypts with old key
   - Re-encrypts with new key
   - Updates the row
   - Processes in batches of 100 with 100ms delay between batches

5. **Verify completion:**

   ```sql
   -- Attempt to decrypt a sample with only the new key
   -- If all succeed, old key is no longer needed
   ```

6. **Remove old key:**
   - Remove `CODA_DB_IDENTITY_AES_KEY_OLD` from configuration
   - Deploy without fallback path
   - Delete old key from Secrets Manager

#### Rollback

If the new key causes issues:

- Swap `CODA_DB_IDENTITY_AES_KEY` back to the old value
- Redeploy
- Any records re-encrypted with the new key will need the fallback path to be re-enabled temporarily

---

### 2. `CODA_DB_IDENTITY_HMAC_SECRET` Rotation

**Impact:** All `identityHash` values in `users` change. This is the lookup key — rotation requires rehashing ALL users.

**Downtime:** Brief (minutes) during cutover. Dual-hash strategy minimizes impact.

#### Procedure

1. **Generate new secret:**

   ```bash
   node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
   ```

2. **Add secondary hash column:**

   ```sql
   ALTER TABLE users ADD COLUMN identity_hash_new VARCHAR(64) NULL;
   CREATE UNIQUE INDEX idx_users_identity_hash_new ON users(identity_hash_new);
   ```

3. **Batch rehash migration:**
   Write a script that:
   - For each user: decrypt `identityEncrypted` → raw ID → HMAC with new secret → write to `identity_hash_new`
   - Process in batches of 100

4. **Deploy with dual-lookup:**
   Update `hashIdentity()` to use the new secret. Update user lookup to check `identity_hash_new` first, fall back to `identity_hash`.

5. **Cutover:**

   ```sql
   -- Verify all users have new hash
   SELECT COUNT(*) FROM users WHERE identity_hash_new IS NULL;
   -- Should be 0

   -- Swap columns
   ALTER TABLE users
     DROP INDEX users_identity_hash_key,
     CHANGE COLUMN identity_hash identity_hash_old VARCHAR(64),
     CHANGE COLUMN identity_hash_new identity_hash VARCHAR(64);

   CREATE UNIQUE INDEX users_identity_hash_key ON users(identity_hash);
   ```

6. **Deploy with single-lookup** (new hash only). Remove fallback code.

7. **Clean up:**

   ```sql
   ALTER TABLE users DROP COLUMN identity_hash_old;
   ```

8. **Remove old secret** from Secrets Manager.

#### Rollback

Before cutover (step 5): remove `identity_hash_new` column, revert code.
After cutover: restore from DB backup (the column swap is the point of no return).

---

### 3. `AUDIT_MASTER_KEY` Rotation

**Impact:** All per-user audit encryption keys (DEKs) are derived via `HMAC-SHA256(masterKey, userId)`. Changing the master key changes all DEKs. Existing encrypted audit PII becomes undecryptable unless re-encrypted.

**Downtime:** Zero (dual-key derivation).

#### Procedure

1. **Generate new master key:**

   ```bash
   node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
   ```

2. **Store both keys:**

   ```
   AUDIT_MASTER_KEY=<new_key>
   AUDIT_MASTER_KEY_OLD=<old_key>
   ```

3. **Deploy with dual-derivation:**
   When reading audit entries:
   - Derive DEK with new master key first
   - If decryption fails (GCM auth tag mismatch), derive with old master key
   - On successful old-key decryption, re-encrypt with new-key-derived DEK (lazy migration)

   When writing new audit entries:
   - Always use new-key-derived DEK

4. **Batch re-encryption (optional but recommended):**
   For each user with audit entries:
   - Derive old DEK: `HMAC-SHA256(oldMasterKey, userId)`
   - Derive new DEK: `HMAC-SHA256(newMasterKey, userId)`
   - Decrypt PII fields with old DEK
   - Re-encrypt with new DEK
   - Update rows in batches

   > **Warning:** Audit logs are append-only by design. Re-encryption requires a temporary relaxation of the UPDATE restriction on `audit_logs`. Document this exception in the change log.

5. **Verify completion:**
   - Sample 100 random audit entries across 10 different users
   - Verify all decrypt successfully with new-key-derived DEK only

6. **Remove old key** and fallback code.

#### Alternative: KMS-Based Per-User DEKs (Recommended for Phase 2)

Instead of HMAC derivation from a master key, use AWS KMS `GenerateDataKey` to create independent per-user DEKs:

1. Each user gets a unique DEK generated by KMS
2. The DEK is stored in `audit_encryption_keys.encrypted_key_data` (KMS-wrapped)
3. No master key exists — compromising one DEK doesn't expose others
4. KMS handles automatic CMK rotation

This eliminates the single-point-of-failure of the master key entirely.

---

### 4. Auth0 Signing Key Rotation

**Impact:** All existing JWTs become invalid when Auth0 rotates its signing key and the old key is removed from the JWKS endpoint.

**Procedure:**

1. **In Auth0 Dashboard:** Rotate the signing key. Auth0 adds the new key to the JWKS endpoint immediately.
2. **JWKS cache:** The `express-oauth2-jwt-bearer` library caches the JWKS. Restart server pods to clear the cache, or wait for TTL expiry.
3. **Grace period:** Auth0 keeps the old key in the JWKS endpoint for a configurable period. During this time, both old and new JWTs are valid.
4. **After grace period:** Remove the old key from Auth0. Old JWTs are rejected.

No application code changes needed. The library handles multi-key JWKS automatically.

---

### 5. Snowflake Private Key Rotation

**Impact:** Snowflake connections fail until the new key is registered.

**Procedure:**

1. **Generate new key pair:**

   ```bash
   openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 aes-256-cbc -inform PEM -out rsa_key_new.p8
   ```

2. **Register public key with Snowflake:**

   ```sql
   ALTER USER DEV_ENGINEERING SET RSA_PUBLIC_KEY_2='<new_public_key>';
   ```

   Snowflake supports two concurrent public keys for zero-downtime rotation.

3. **Deploy with new private key:**
   Update `SNOWFLAKE_READER_PRIVATE_KEY_PATH` (or `SNOWFLAKE_READER_PRIVATE_KEY`) to point to the new key.

4. **Verify connectivity:**

   ```bash
   pnpm dev  # or health check endpoint
   ```

5. **Remove old key from Snowflake:**
   ```sql
   ALTER USER DEV_ENGINEERING UNSET RSA_PUBLIC_KEY;
   ```

---

## Emergency Rotation (Key Compromise)

When a key is known or suspected to be compromised:

1. **Follow the incident response playbook** ([platform-incident-response.md](platform-incident-response.md)) — assign IC, open channel.

2. **Rotate immediately** — do not wait for a maintenance window. Follow the applicable procedure above, but skip the lazy migration and go straight to batch re-encryption.

3. **Assess blast radius:**
   - Which data was decryptable with the compromised key?
   - How long was the key exposed? (Check CloudTrail, Secrets Manager access logs, deployment history)
   - Was data actually accessed? (Check audit logs, Snowflake query history)

4. **Notify affected parties:**
   - If PII was exposed: GDPR Art. 33 (72-hour supervisory authority notification)
   - If OAuth tokens were exposed: revoke all affected tokens with the third-party provider
   - If Snowflake key was exposed: check Snowflake login history for unauthorized access

5. **Post-mortem:**
   - How was the key leaked? (Log exposure, env dump, insider threat, dependency vulnerability)
   - Add monitoring for the leak vector

---

## Rotation Schedule

| Key                            | Last Rotated               | Next Rotation     | Owner         |
| ------------------------------ | -------------------------- | ----------------- | ------------- |
| `CODA_DB_IDENTITY_AES_KEY`     | Never (initial deployment) | Before Phase 1 GA | Platform team |
| `CODA_DB_IDENTITY_HMAC_SECRET` | Never (initial deployment) | Before Phase 1 GA | Platform team |
| `AUDIT_MASTER_KEY`             | Never (not yet created)    | Before Phase 1 GA | Platform team |
| AWS KMS CMK                    | AWS-managed                | Automatic         | AWS           |
| Auth0 signing key              | Auth0-managed              | Quarterly         | Auth0 admin   |
| Snowflake private key          | Never (initial deployment) | Annual            | Platform team |

---

## Pre-Rotation Checklist

Before any key rotation:

- [ ] Verify dual-key support is implemented in the codebase
- [ ] Test the rotation procedure in the QA environment first
- [ ] Ensure database backup exists (point-in-time recovery enabled)
- [ ] Schedule during low-traffic window (if batch re-encryption)
- [ ] Notify on-call that rotation is in progress
- [ ] Prepare rollback plan and test it

---

## Maintaining This Document

- **Review cadence:** Before each rotation, and after any key compromise incident.
- **Owner:** Platform team lead.