---
sidebar_position: 2
---

# Clusters and Brokers
Kafka runs as a cluster of one or more servers which form the data storage layer called the **brokers**. Kafka clusters are highly scalable and fault-tolerant: if any of its servers fails, the other servers will take over their work to ensure continuous operations without any data loss.

## Brokers
![Broker endpoints in MSK](./img/msk_brokers.png)

Kafka keeps more than one copy of the same `partition` across multiple brokers; this redundant copy is called a replica. If a broker fails, Kafka can still serve consumers with the replicas of the partitions in the other brokers.

## Infrastructure setup
To manage MSK infrastructure, explore current examples in [terraform-infra](https://github.com/theorchard/terraform-infra/blob/master/prod/kafka-infra/kafka-cluster).

We use the [terraform-managed-kafka](https://github.com/theorchard/terraform-managed-kafka) module to provision and manage a AWS MSK cluster's infrastructure, and there you can check all resources deployed and configurable variables.

### Custom Kafka properties
In addition to the settings exposed through variables in the MSK Terraform module, Kafka allows for extra custom configurations.

Some of this settings are set by default in the module's `kafka_properties.conf` and can be overriden:

| Property                      	| Default Value 	| Description                                                                                              	|
|-------------------------------	|---------------	|----------------------------------------------------------------------------------------------------------	|
| **auto.create.topics.enable** 	| `true`        	| Allows Producer clients to automatically create a new topic when publishing a message for the first time.<br></br> When creating a topic automatically you lose the ability to change it's configuration. We recommend [creating them beforehand through terraform](./events-and-topics.mdx#infrastructure-setup) when possible.	|
| **delete.topic.enable**         	| `true`           	| Allows to delete topics through the admin tool.                                                           |
| **num.partitions**                | `3`               | Default number of partitions for topics created automatically.                                            |
| **default.replication.factor**    | `3`               | Default replication factor for topics created automatically.                                              |
| **unclean.leader.election.enable**| `false`           | Forbids replicas not in sync from serving as leader.                                                      |

For a full list of possible settings that can be added to this file in a cluster, check the [AWS MSK Custom Configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html) and the [Apache Kafka Configuration](https://kafka.apache.org/documentation/#configuration) docs.