# Kafka Topic Naming Rules

Source of truth: [Kafka Naming Conventions (Notion)](https://www.notion.so/49d48bb6a549442cb65c6d36120c7069).

The skill **must** validate every topic name against these rules. A topic that doesn't match any of the patterns below is a **hard error** — block generation and report the offending names back to the user.

## Confirm the middle segment with the user

The middle segment — `<sourceName>` / `<dataSetName>` (e.g. `fingerprinting`, `musicGraphV5`, `chartmetric`, `facts`) — is the source/namespace and is **not** necessarily the connector's `purpose`/`connector_name`. Always confirm it with the user (propose a default, let them override) and apply it consistently across every topic generated for the connector.

## Patterns

Every topic name starts with one of five lowercase prefixes followed by `.`-separated camelCase parts.

| Prefix | Use case | Pattern | Examples |
|---|---|---|---|
| `cdc.` | Change Data Capture from a source database/service | `cdc.<sourceName>.<dataSetName>[.<additionalInfo>]` | `cdc.chartmetric.charts`, `cdc.musicGraph.globalSoundRecording`, `cdc.analytics.soundRecordingsStreams.v02` |
| `stream.` | Intermediate or destination-formatted stream (KSQL output, transformations) | `stream.<dataSetName>.<additionalInfo>.<typeOfStream>` | `stream.musicGraph.globalSoundRecording.destination`, `stream.nrGraph.mysql.nr.sound.recording` |
| `event.` | Business event from a service or business object | `event.<source>.<businessObject>` | `event.musicEvent.socialMedia`, `event.spotifyapi.audioFeatures`, `event.sr.newVersion` |
| `etl.` | Bulk chunked source extraction (when CDC is not supported) | `etl.<sourceName>.<dataSetName>[.<additionalInfo>]` | `etl.snowflake.factsStreams` |
| `dlq.` | Dead-letter queue for failed messages | `dlq.<dataSetName>[.<additionalInfo>]` | `dlq.gdaSignup`, `dlq.snowflakeSink.noGSR` |

## Validation regex (single combined)

```
^(cdc|stream|event|etl|dlq)\.[a-zA-Z][a-zA-Z0-9]*(\.[a-zA-Z0-9][a-zA-Z0-9_]*)+$
```

Notes:
- All parts after the prefix should be camelCase or simple identifiers. Underscores tolerated for compatibility with existing topics (e.g. `stream.nrGraph.mysql.nr.sound.recording`).
- Hyphens are not allowed.
- All-lowercase prefix is mandatory (`CDC.foo` is invalid).
- Topic must have at least 2 dot-separated segments after the prefix (prefix + source + dataset minimum).

## Hard rules

1. Every topic referenced in a connector's `topics` / `KAFKA_TOPICS` env var must be either:
   - **Already terraformed** in the relevant `kafka-cluster*/topics/*.tf` file, **or**
   - Generated by this skill alongside the connector (see `topic_block.tf.j2`).
2. **Source connectors** (`direction: "source"` in registry) must always emit topic resources — they create new topics that need to exist before the connector starts.
3. **Sink connectors** that reference a topic not yet terraformed should warn — the topic may exist but isn't tracked, which the Notion guidebook flags as risky.
4. DLQ topic names must use the `dlq.` prefix and be terraformed too.

## Default topic config (when generating new topics)

Match the conventions seen in `prod/kafka-infra/kafka-cluster/topics/`:

```hcl
resource "kafka_topic" "<sanitized_name>" {
  name               = "<topic_name>"
  replication_factor = var.kafka_minimum_replication_factor  # typically 3
  partitions         = 1   # default 1; ask the user to bump if they need throughput / parallelism
  config = {
    "max.message.bytes"   = "15728640"     # 15 MiB
    "retention.ms"        = "604800000"    # 7 days
    "cleanup.policy"      = "delete"
    "min.insync.replicas" = "2"
  }
}
```

The Terraform resource name (left side of `"<sanitized_name>"`) is the topic name with `.` replaced by `_`.

## When unsure

If a topic name doesn't fit any pattern but the user insists it's correct:
1. Surface the conflict explicitly to the user.
2. Link them to the Notion conventions page.
3. Do not auto-override. Either the user fixes the name, or they explicitly confirm "ignore the naming rule" — only then proceed and emit a warning in the validation report.