# Token Encryption Strategy

## Context

Resonance Engine handles Spotify OAuth refresh tokens for ~159K fans (dev) / 73M fans (production). These tokens flow through multiple systems:

```
DDB Export -> SQS -> Collector Worker -> Kafka -> Snowflake Landing -> RE_FAN_TOKENS -> S3 (recollection) -> back to Collector Worker
```

This document evaluates encryption strategies for protecting refresh tokens at rest across the pipeline, based on research into org patterns in `terraform-infra`.

## Org Patterns (terraform-infra)

### KMS Key Strategy: Per-Service CMKs

The org uses customer-managed KMS keys per service, not shared keys.

- `ows-dmp/kms.tf` — two CMKs: one for Songwhip DSP tokens (cross-account), one for internal DMP data
- `art-relations/aurora/kms.tf` — per-service PII key with admin/service user separation
- `rds-refresh/kms.tf` — cross-account RDS snapshot sharing key

**Naming convention:** `alias/{environment}-{service_name}[-suffix]`

### Secrets Management: Secrets Manager, Not Environment Variables

Secrets are **never** passed as Lambda environment variables. The org uses `terraform-secrets-manager` module:

```hcl
module "service_secrets" {
  source   = "git@github.com:theorchard/terraform-secrets-manager.git//?ref=1.5.1"
  for_each = toset(var.secret_names)

  environment  = var.environment
  service_name = var.service_name
  secret_name  = each.value  # Creates {env}/{service_name}/{secret_name}
}
```

Lambdas fetch secrets at runtime via `boto3.client('secretsmanager').get_secret_value()`.

**Lambda environment variables** contain only public config: service names, table names, Kafka brokers, topic names, Snowflake account/warehouse/schema (not passwords).

### Kafka Sink Connectors: Secrets Manager for Snowflake Auth

From `prod/kafka-infra/snowflake_sink_fanresponse/main.tf`:
- Snowflake private key + passphrase stored in Secrets Manager
- Public config (host, user, database, topics) in environment variables
- Docker image fetches secrets at container startup

### No Application-Layer Field Encryption

No Lambda or service in the org encrypts individual data fields before sending to Kafka or SQS. The org relies entirely on:

- **Transit security:** TLS for all connections (Kafka port 9094, Snowflake HTTPS, Secrets Manager HTTPS)
- **At-rest encryption:**
  - S3: `apply_server_side_encryption_by_default` with AWS-managed keys (not CMK in most cases)
  - RDS/Aurora: Encrypted with CMKs where sensitive data is involved
  - Secrets Manager: Encrypted with AWS-managed key (default)
  - Kafka/MSK: At-rest encryption (AWS-managed)
  - SQS: AWS-managed key (`aws/sqs`) by default; no custom `kms_master_key_id` observed in most queues

### SQS Encryption

Most SQS queues use AWS-managed encryption (default). No custom CMK usage observed on SQS queues except in edge cases.

### Snowflake Data Masking

No Dynamic Data Masking policies were observed in the `terraform-infra` codebase during this review. Access control is handled via Snowflake roles scoped to specific schemas.

## Approaches Evaluated

### Option A: Application-Layer KMS Encryption (Field-Level)

Encrypt each refresh token with KMS before it enters any downstream system. Only the Collector Worker Lambda has `kms:Decrypt`.

**Flow:**
```
Manifest Parser [KMS Encrypt] -> SQS (ciphertext) -> Collector Worker [KMS Decrypt]
  -> Spotify API -> [KMS Encrypt] -> Kafka (ciphertext) -> Snowflake (ciphertext)
  -> S3 recollection (ciphertext) -> Collector Worker [KMS Decrypt] -> ...
```

**Pros:**
- Strongest protection — tokens are ciphertext everywhere at rest
- Even Snowflake ACCOUNTADMIN or Kafka consumers with topic access see only ciphertext
- KMS key policy enforces that only Collector Worker can decrypt
- Defense-in-depth beyond infrastructure-layer encryption

**Cons:**
- Not consistent with org patterns (no precedent in terraform-infra)
- Per-record KMS API calls: ~318K calls for 159K fan backfill (~$1), ~146M calls for 73M fans (~$440/cycle)
- Added code complexity in both Lambdas (encrypt/decrypt logic, auto-detection for migration)
- KMS API latency added to every fan processing (~5-10ms per call)
- Envelope encryption optimization needed at 73M scale to reduce API calls

### Option B: Infrastructure-Layer Encryption (Org Standard)

Use AWS-managed or CMK encryption on the infrastructure components themselves. Tokens remain plaintext within the application but are encrypted at rest by the service.

**Flow:**
```
Manifest Parser -> SQS [SSE-KMS] -> Collector Worker -> Kafka [MSK at-rest]
  -> Snowflake [Snowflake encryption + masking] -> S3 [SSE-KMS] -> Collector Worker
```

**Components:**

| Layer | Mechanism | Implementation |
|-------|-----------|----------------|
| Spotify credentials | Secrets Manager | `terraform-secrets-manager` module; Lambda fetches at cold start |
| SQS | SSE-KMS with CMK | `kms_master_key_id` on `terraform-sqs` module; auto-encrypts message bodies |
| Kafka/MSK | TLS in transit + MSK at-rest | Already TLS (port 9094); MSK handles at-rest encryption |
| S3 recollection | SSE-KMS with CMK | `apply_server_side_encryption_by_default` with CMK on recollection prefix |
| Snowflake | Dynamic Data Masking | Masking policy on `RECORD_CONTENT:refresh_token` and `RE_FAN_TOKENS.REFRESH_TOKEN`; scoped to roles below admin |
| DLQ | SSE-KMS with CMK | Same CMK as SQS queue; auto-encrypts failed messages |

**Pros:**
- Consistent with org patterns and conventions
- Zero per-record KMS cost (SSE is transparent to the application)
- No code changes to Lambdas (encryption is at the infrastructure layer)
- Simpler to maintain and reason about
- Secrets Manager for Spotify creds is already in the production readiness checklist

**Cons:**
- Anyone with SQS `ReceiveMessage` + KMS `Decrypt` permission sees plaintext tokens
- Anyone with Snowflake SELECT on the schema sees plaintext tokens (mitigated by masking)
- Kafka consumers with topic access see plaintext tokens
- Relies on IAM/Snowflake role boundaries for access control (not cryptographic enforcement)

### Option C: Application-Layer Symmetric Encryption (Fernet) *(added 2026-04-17 — post-decision brainstorm)*

This option was not evaluated during the original decision. It surfaces the gap in the Option A vs Option B framing: Option A conflated "application-layer field encryption" with "per-record KMS API calls." A shared symmetric key stored in Secrets Manager decouples the two — encryption is field-level and cryptographically enforced, but there are no per-record API calls.

Encrypt each refresh token with a [Fernet](https://cryptography.io/en/latest/fernet/) symmetric key fetched once at Lambda cold start from Secrets Manager. The `cryptography` package performs AES-128-CBC + HMAC-SHA256 in pure Python.

**Flow:**
```
Manifest Parser [cold start: fetch FERNET_KEYS from Secrets Manager, build MultiFernet]
  -> SQS (ciphertext: "gAAAAA...") -> Collector Worker [cold start: same fetch]
  -> [decrypt in memory] -> Spotify API -> [re-encrypt] -> Kafka (ciphertext)
  -> Snowflake (ciphertext) -> S3 recollection (ciphertext) -> Collector Worker [decrypt] -> ...
```

**Pros:**
- Tokens are ciphertext everywhere at rest — SQS, Kafka, Snowflake, S3 recollection exports
- Snowflake `ACCOUNTADMIN` sees ciphertext; masking policies alone cannot protect against a compromised superuser
- SQS DLQ messages (14-day retention) contain ciphertext — no plaintext exposure in failed batches
- Zero per-record API cost (pure Python crypto; no KMS call per token)
- Negligible latency (~microseconds vs 5–10ms for KMS `GenerateDataKey*`)
- Simpler code than KMS option: two lines per encrypt/decrypt call
- Secrets Manager fetch is shared across all fans in a Lambda invocation (cold start only)

**Cons:**
- Key rotation requires careful handling (see below)
- Adds `cryptography>=42.0.0` to both Lambda `requirements.txt` (binary wheel dependency)
- Tokens stored as Fernet ciphertext in Snowflake are opaque to any SQL query or BI tool; useful SQL operations on token values (e.g. dedup by token value) become impossible
- Not consistent with org patterns (no precedent, same as Option A)

**Key Rotation with MultiFernet:**

Secrets Manager stores a JSON array of base64-encoded Fernet keys, ordered newest-first:

```json
{ "keys": ["NEW_KEY_b64==", "OLD_KEY_b64=="] }
```

Both Lambdas build a `MultiFernet` at cold start:

```python
import json
from cryptography.fernet import Fernet, MultiFernet

secret = json.loads(boto3.client("secretsmanager").get_secret_value(
    SecretId=f"{secrets_prefix}/FERNET_KEYS"
)["SecretString"])
_fernet = MultiFernet([Fernet(k) for k in secret["keys"]])

# Encrypt (always uses first/newest key)
ciphertext = _fernet.encrypt(refresh_token.encode()).decode()

# Decrypt (tries keys in order; succeeds on whichever key encrypted it)
plaintext = _fernet.decrypt(ciphertext.encode()).decode()
```

**Rotation procedure:**

1. Generate a new key: `Fernet.generate_key().decode()`
2. Update the Secrets Manager secret: prepend the new key → `["NEW_KEY", "OLD_KEY"]`
3. Deploy updated Lambda images. New encryptions use `NEW_KEY`; old ciphertexts decrypt via `OLD_KEY`
4. After one full re-collection cycle, all Snowflake tokens will have been re-encrypted with `NEW_KEY`
5. Remove `OLD_KEY` from the array → `["NEW_KEY"]`

The window of dual-key support matches the re-collection cadence (e.g. 30 days). No re-encryption job is needed: the pipeline re-encrypts tokens naturally as fans are re-collected.

**Comparison to Option A (KMS):**

| Dimension | Option A (KMS) | Option C (Fernet) |
|-----------|---------------|-------------------|
| Per-record API cost | ~$440/cycle at 73M fans | $0 |
| Latency per token | 5–10ms (KMS network call) | ~microseconds |
| Key access control | KMS key policy (IAM-enforced) | Secrets Manager IAM policy |
| Key rotation | KMS automatic rotation (annual) | Manual via Secrets Manager + deploy |
| Org precedent | None | None |

Option C achieves the same cryptographic guarantee as Option A at the cost profile of Option B. The original decision rejected Option A primarily on cost and latency grounds; both objections are resolved here.

**Why this was not in the original evaluation:**

The original analysis framed the choice as "KMS per-record (expensive) vs infrastructure-layer (free)." This conflated *field-level encryption* with *KMS API calls*, which are not equivalent. The AWS-first lens made KMS the natural answer to "encrypt a field in a Lambda," leaving symmetric encryption unexamined.

**Status:** Not adopted. The original Option B decision (2026-02-28) stands. This option is documented for future consideration, particularly if the threat model expands to include Snowflake superuser compromise or Kafka consumer unauthorized access as credible risks.

---

## Spotify Credentials: Current vs Target State

Currently Spotify `client_id` and `client_secret` are passed as Lambda environment variables (in Terraform). The org pattern is to use Secrets Manager instead:

**Current (dev):**
```hcl
lambda_function_environment_variables = {
  SPOTIFY_CLIENT_ID     = var.spotify_client_id
  SPOTIFY_CLIENT_SECRET = var.spotify_client_secret
}
```

**Target (prod):**
```hcl
# Secrets Manager
module "resonance_engine_secrets" {
  source       = "git@github.com:theorchard/terraform-secrets-manager.git//?ref=1.5.1"
  for_each     = toset(["SPOTIFY_CLIENT_ID", "SPOTIFY_CLIENT_SECRET"])
  environment  = var.environment
  service_name = var.service_name
  secret_name  = each.value
}

# Lambda fetches at cold start via boto3
lambda_function_environment_variables = {
  SECRETS_PREFIX = "${var.environment}/${var.service_name}"
}
```

## KMS Key Policy Template (If CMK Is Used)

If a CMK is created for SQS/S3 encryption (Option B), the key policy follows the org pattern:

```hcl
data "aws_iam_policy_document" "token_kms_key_policy" {
  # Root account — allows IAM policies to grant access, prevents lockout
  statement {
    sid    = "EnableIAMPolicies"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
    actions   = ["kms:*"]
    resources = ["*"]
  }

  # SQS service principal — required for SSE-KMS on SQS
  statement {
    sid    = "AllowSQSServiceAccess"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["sqs.amazonaws.com"]
    }
    actions = [
      "kms:Decrypt",
      "kms:GenerateDataKey*"
    ]
    resources = ["*"]
  }

  # Lambda roles — send/receive encrypted SQS messages
  statement {
    sid    = "AllowLambdaAccess"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [
        module.lambda_manifest_parser.lambda_role_arn,
        module.lambda_collector_worker.lambda_role_arn,
      ]
    }
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:GenerateDataKey*",
      "kms:DescribeKey"
    ]
    resources = ["*"]
  }

  # S3 service principal — required for SSE-KMS on S3
  statement {
    sid    = "AllowS3ServiceAccess"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["s3.amazonaws.com"]
    }
    actions = [
      "kms:Decrypt",
      "kms:GenerateDataKey*"
    ]
    resources = ["*"]
  }
}
```

## Decision

**Option B: Infrastructure-Layer Encryption (Org Standard)** — decided 2026-02-28.

Rationale: Consistent with org patterns, zero per-record KMS cost, no Lambda code changes for encryption. Access control handled by IAM roles and Snowflake role boundaries.

### Implementation Scope

1. **Snowflake Dynamic Data Masking** — masking policy on `RE_FAN_TOKENS.REFRESH_TOKEN` and `RE_SPOTIFY_DATA_LANDING.RECORD_CONTENT` token fields. Only the service role used by the sink connector and transformation tasks can see plaintext tokens; all other roles see masked values.
2. **SQS SSE-KMS with CMK** — deferred to production (per production readiness checklist).
3. **S3 SSE-KMS for recollection exports** — deferred to production.
4. **Secrets Manager for Spotify credentials** — deferred to production.

Application-layer field encryption (Option A) was evaluated but rejected as inconsistent with org conventions and unnecessary given infrastructure-layer protections.