Now I need to explore the terraform work needed setup the lambda to 
consume from the topics.

Here are some TF examples related to the examples in `PP-1411/notes_about_lambda.md`


* lambda-store-api: /Users/tcalhoun/work/terraform-infra/qa/lambda-store-api
* lambda-preference-sforce-sync: /Users/tcalhoun/work/terraform-infra/fansifter/prod/lambda-preference-sforce-sync
* lambda-metadata-consistency: /Users/tcalhoun/work/terraform-infra/qa/lambda-metadata-consistency


Also, here's some example lambdas from our project, but these lambdas don't read from kafka.

- lambda-pp-audit-ddbs-kafka: /Users/tcalhoun/work/terraform-infra/permissions-platform/qa/lambda-pp-audit-ddbs-kafka
- lambda-auth0-m2m-token-secret-rotation: /Users/tcalhoun/work/terraform-infra/permissions-platform/qa/lambda-auth0-m2m-token-secret-rotation

Also, this project has some of the connectors we use to connect to kafka.


permissions-platform/qa/kafka-pp/connectors: /Users/tcalhoun/work/terraform-infra/permissions-platform/qa/kafka-pp


Its also worth noting that the CDC kafka clusters lives in a separate AWS account.

The CDC topics live in the `prod` account: `437795906767`.
The Permissions Platform QA account is in: `591204808501`. Prod is: `031099521156`

---

## Cross-account challenge

The `cdc.musicGraphV5.profile` topic lives on the CDC MSK cluster in the shared prod account
(`437795906767`). The PP Lambda would normally live in the PP account (`591204808501` QA /
`031099521156` PROD), but Lambda's native MSK event source mapping (`msk_event_enabled`) requires
the Lambda and MSK cluster to be in the same account. Two options:

### Option A: Self-managed Kafka Lambda in PP account

The Lambda uses `kafka_self_managed_event_source` with CDC cluster broker addresses in env var —
same pattern as `lambda-store-api`. Lambda lives in the PP account (`031099521156`).

Cross-account requirements:
- Network: PP VPC → CDC cluster via Transit Gateway
- Auth: SASL/MSK IAM credentials in Secrets Manager
- CDC account (`437795906767`) needs a resource-based policy on the MSK cluster

`lambda-store-api` uses self-managed Kafka against a same-account cluster. This would be the
first cross-account self-managed case.

---

### Option B: Lambda in prod account (same account as MSK)

Deploy the Lambda in the prod account (`437795906767`) alongside the CDC MSK cluster. Uses
`msk_event_enabled = true` — identical to `lambda-store-api`. No cross-account MSK complexity.

HTTP calls to ows-permissions and ows-pdp still go cross-account (prod → PP account), but:
- **This path is already established.** Almost all microservices live in the prod account and
  already reach ows-pdp via existing devops CIDR/SG cross-account networking.
- A VPC-attached Lambda gets an ENI with a VPC IP — the same SG rules that apply to
  ECS/EC2 services should apply. **Needs devops confirmation** that Lambda ENIs aren't
  excluded from the current rules.

Tradeoff: puts a new PP Lambda in the shared prod account, going against the multi-account
direction PP is adopting. Ownership and deployment live outside the PP account CI.

---

## New Lambda TF directory

**Option A (PP account):**
```
permissions-platform/prod/lambda-pp-cdc-workstation-roles/
```

**Option B (prod account):**
```
prod/lambda-pp-cdc-workstation-roles/
```

Both follow the same module shape. The key difference is the Kafka trigger:

```hcl
module "lambda_pp_cdc_workstation_roles" {
  source = "git@github.com:theorchard/terraform-lambda.git?ref=<version>"

  environment         = var.environment
  application_family  = "permissions-platform"
  lambda_name         = "lambda-pp-cdc-workstation-roles"
  use_container_image = true

  vpc_enabled    = true
  vpc_id         = module.vpc_info.vpc_id
  vpc_subnet_ids = module.vpc_info.default_private_subnet_ids

  # Option A: self-managed (PP account, cross-account broker access)
  kafka_self_managed_event_source        = true
  kafka_topics                           = ["cdc.musicGraphV5.profile"]
  event_source_mapping_batch_size        = 100
  event_source_mapping_starting_position = "LATEST"
  lambda_function_environment_variables = {
    ENVIRONMENT   = var.environment
    KAFKA_BROKERS = var.cdc_kafka_brokers
  }

  # Option B: native MSK ESM (prod account, same-account cluster — like lambda-store-api)
  # msk_event_enabled                      = true
  # kafka_topics                           = ["cdc.musicGraphV5.profile"]
  # event_source_mapping_msk_cluster_name  = var.cdc_msk_cluster_name
  # event_source_mapping_msk_cluster_uuid  = var.cdc_msk_cluster_uuid
  # event_source_mapping_batch_size        = 100
  # event_source_mapping_starting_position = "LATEST"
  # lambda_function_environment_variables = {
  #   ENVIRONMENT = var.environment
  # }

  iam_managed_policy_attachments = [
    aws_iam_policy.workstation_roles_lambda_policy.arn
  ]
}
```

## Lambda IAM policy

The Lambda execution role needs:

```hcl
data "aws_iam_policy_document" "workstation_roles_lambda_policy_document" {

  # SASL credentials for the CDC MSK cluster + ows-permissions/ows-pdp service auth
  # (kafka:/kafka-cluster: IAM actions are not needed for self-managed Kafka mode)
  statement {
    effect    = "Allow"
    actions   = ["secretsmanager:GetSecretValue"]
    resources = ["arn:aws:secretsmanager:us-east-1:${data.aws_caller_identity.current.account_id}:secret:${var.environment}/lambda-pp-cdc-workstation-roles/*"]
  }
}
```

For **Option B**, the IAM policy also needs MSK read actions against the CDC cluster (same as
`lambda-store-api`). HTTP calls to ows-pdp are cross-account VPC via existing CIDR/SG rules —
no additional IAM policy needed for those.

## Open questions

1. **Which account?** — Option B (prod account) is simpler for MSK access and follows the proven
   cross-account networking to ows-pdp. Option A (PP account) stays in the multi-account direction.
   Decision needed before cutting tickets.

2. **Lambda ENI + cross-account SG rules (Option B)** — Confirm with devops that the existing
   CIDR/SG rules allowing prod-account services to reach ows-pdp also cover Lambda ENIs. This
   would be the first Lambda (vs ECS/EC2) using that path.

3. **Option A: SASL auth** — If Option A is chosen, confirm what auth mechanism the CDC MSK
   cluster uses (SASL/SCRAM, MSK IAM) and how credentials are stored. `lambda-store-api` is the
   reference for how self-managed Kafka auth is handled.

4. **QA environment for Option B** — If the Lambda lives in the prod account, how is QA testing
   handled? Confirm whether a non-prod CDC MSK cluster exists or if QA is skipped for this Lambda.