# json_to_changelog

Converts a JSON array of Kafka DLQ (or similar) records into a Liquibase‑style changelog XML. Each input element becomes one `<changeset>` with an embedded `<kafkamessage>`.

## 1. Current CLI

```bash
python -m json_to_changelog.json_to_changelog INPUT.json --author AUTHOR --id BASE_ID [--output-dir DIR] [--suffix SUFFIX]
```

Positional:

* `INPUT.json`  JSON file whose top‑level is an array of record objects.

Required flags:

* `--author`  Author applied to every changeset.
* `--id`      Base changeset id. For multiple records, `:n` (1-based index) is appended per record.

Optional flags:

* `--output-dir`  Defaults to `~/code/kafka-db-deploy/kafka/build/changelog/dml`. Created if missing.
* `--suffix`      Filename suffix appended to the base id (default: `_redrive_dql_messages.xml`).

Resulting file path (default settings):

```text
<output-dir>/<BASE_ID>_redrive_dql_messages.xml
```

### 2. Input Record Schema

Only the `value` field is strictly needed. The script supplies `id` and `author` from the CLI. A `topic` field is optional; if present it becomes a `<changeset>` attribute.

Recognized optional fields with defaults:

* `kafka-cluster-name` (default: `managed-kafka-cdc-destination`)
* `key` (default: `null`)
* `tombstone` (default: `false`)

`key` and `tombstone` become attributes of `<kafkamessage>`. Any other (non‑reserved) top‑level fields in the record (e.g. Kafka metadata like `offset`, `partition`, `timestamp`, headers, etc.) are copied as additional `<changeset>` attributes.

Reserved (not copied as `<changeset>` attributes):

* `value` (its parsed payload becomes the body of `<kafkamessage>`)
* `key`, `tombstone` (used on `<kafkamessage>` itself)

### 3. Payload Extraction

If the `value` string contains a fragment like:

```text
payload={FOO=bar, ANSWER=42, TEXT=Hello World}
```
the inner `payload={...}` portion is parsed into JSON object form and emitted as the message body. The parser:

* Handles values with spaces, commas inside parentheses, URLs.
* Converts `null` to JSON null, and numeric digit sequences to integers.
* Leaves anything ambiguous as a string.

Fallback: If no `payload={...}` match exists, the original `value` string is used verbatim.

### 4. Minimal Example

Input (`input.json`):

```json
[
  { "value": "payload={CREATED_BY=ingest, UPC=12345, TRACK_NAME=Song Title}" },
  { "value": "payload={CREATED_BY=ingest, UPC=67890, TRACK_NAME=Another}" }
]
```

Run:

```bash
python -m json_to_changelog.json_to_changelog input.json --author cbeesley --id IN-15986
```

Output file (default path): `~/code/kafka-db-deploy/kafka/build/changelog/dml/IN-15986_redrive_dql_messages.xml`

Each changeset id: `IN-15986:1`, `IN-15986:2`, ...

### 5. Customizing Output Name / Location

```bash
--output-dir /tmp/out --suffix _custom.xml
```
Produces: `/tmp/out/IN-15986_custom.xml`

### 6. Development & Install

Create venv & install locally (from repo root):

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install .
```

Or use Make (if present):

```bash
make dev
```

### 7. Tests

```bash
pytest -q
```

### 8. Limitations / Notes

* Payload parsing is heuristic; complex nested structures are not fully supported.
* Trailing or unexpected characters in the source `payload={...}` may lead to slightly altered string values; verify critical fields.
* Additional Kafka metadata fields will inflate the `<changeset>` attribute list—filter upstream if undesired.

### 9. Troubleshooting

| Issue | Explanation | Action |
|-------|-------------|--------|
| File not created | Wrong working dir or missing write perms | Use absolute path; verify `--output-dir` |
| Garbled payload | Parser couldn't interpret edge cases | Inspect raw `value`; adjust or simplify |
| Missing topic | Topic is optional; supply if desired | Add `"topic": "your.topic"` per record |

### 10. Quick Run With Downloaded File

```bash
cd /path/to/repo && python -m json_to_changelog.json_to_changelog \
"/Users/you/Downloads/dlq.example.json" --author you --id IN-12345
```

---

Internal utility (add license / ownership details here if required).