---
slug: kafka-topic-migration
title: How to migrate event topics between clusters
authors: ssavva
tags: [ topics, migrations, mirror-maker ]
---

### Introduction and Historical Contenxt

Originally, our Kafka ecosystem revolved around a single cluster architecture. This centralized cluster became a hub for
different kinds of topics – cdc messages, ksqlDB streams and table, business logic events. Now we are considering moving
the event topic to a dedicated Kafka cluster. There are several potential reasons. First, the logical grouping of
similar data streams – bringing all event-related topics into a dedicated space ensures more transparent data flow and
processing. Furthermore, a dedicated cluster allows us to scale resources in line with the specific demands of these
event topics. Maintenance and monitoring also become more elegant with this segregation.

{/* truncate */}

### The Migration Strategy

Diagram:

<iframe width="800" height="450" src="https://whimsical.com/embed/ETDoqaDTkMQF4MYrwFVaGw"></iframe>

The migration strategy involves the following steps:

#### Step 1: Create the event topic to the New Cluster

Make sure that the target cluster is up and running and then create the event topic with the same name as the source
topic.
We use Terraform, and creating the new topic should be as easy as copying and pasting the existing topic configuration.

#### Step 2: Switching the Producer to the New Cluster

Start with redirecting the Kafka producer from the old to the new cluster by updating the producer configuration to
point to the new cluster's bootstrap servers.

#### Step 3: Consuming to the End of the Old Topic

Monitor the consumer lag on the old topic. Once it reaches zero, all messages have been consumed, indicating that the
consumer can be safely switched to the new cluster. The easiest way to track the consumer lag is to use
the [AKHQ tool](https://prod-akhq.theorchard.io).

![Consumer lag in AKHQ](./2023-11-02-kafka-topic-migration/AKHQ-lag-example.png)

#### Step 4: Switching the Consumer to the New Cluster

After consuming all messages from the old topic, switch the consumer to start reading from the new cluster's topic,
beginning with the earliest offset to ensure no message loss.

### Considerations and Best Practices

- Testing: Test the migration process in QA environment first.
- Coordination: Precisely time each step, especially in high-throughput topics, to minimize processing delay.

### Note on Alternative Approaches

It's worth mentioning Apache Kafka's MirrorMaker 2 as a potential alternative for migration. MirrorMaker 2, particularly
with its `sync.group.offsets.enabled` setting, can be a powerful tool for replicating topics and consumer groups between
clusters. However, for the current task of migrating event topics, we will focus on a more straightforward and simple
approach.

### Conclusion

Migrating a Kafka topic between clusters is a delicate process but manageable with proper planning and execution. By
carefully moving the producer, ensuring all messages are consumed from the old topic, and then transitioning the 
consumer, you can achieve a smooth, effective migration.