--- sidebar_position: 5 --- # Partitions Each Kafka topic can consist of one or several partitions. While the topic is a logical concept in Kafka, a partition is the smallest storage unit that holds a subset of records owned by a topic. Each partition is a single log file where records are written to it in an append-only fashion. ![Topic Partition](./img/partition.png) ## Offsets and ordering of messages The records in the partitions are each assigned a sequential identifier called the offset, which is unique for each record within the partition. The offset is an incremental and immutable number, maintained by Kafka. When a record is written to a partition, it is appended to the end of the log, assigning the next sequential offset. Offsets are particularly useful for consumers when reading records from a partition. The figure below shows a topic with three partitions. Records are being appended to the end of each one. Although it is guaranteed that the messages within a partition are ordered, messages across a topic are *not* guaranteed to be ordered. ![Topic with 3 partitions](./img/three_partitions.png) ## Scalability and redundancy Kafka distributes the partitions of a particular topic across multiple [brokers](./cluster-and-brokers.mdx#brokers). By doing so, we’ll get the following benefits: - If we are to put all partitions of a topic in a single broker, the scalability of that topic will be constrained by the broker’s IO throughput. A topic will never get bigger than the biggest machine in the cluster. By spreading partitions across multiple brokers, a single topic can be scaled horizontally to provide performance far beyond a single broker’s ability. - A single topic can be consumed by multiple consumers in parallel. Serving all partitions from a single broker limits the number of consumers it can support. Partitions on multiple brokers enable more consumers. To scale the reading and processing of messages from the topics add consumers to an existing consumer group, so each additional consumer in a group will only get a subset of the messages. - Multiple instances of the same consumer can connect to partitions on different brokers, allowing very high message processing throughput. Each consumer instance will be served by one partition, ensuring that each record has a clear processing owner. A copy of the same partition is stored across multiple Kafka brokers. This redundant copy is called a **replica**. If a broker fails, Kafka can still serve consumers with the replicas of partitions that failed broker owned. ## Writing records to partitions How does a producer decide to which partition a record should go? There are two ways a producer can rule on that. - **Using a message key to specify the partition**: A producer can set a message key to direct messages to a specific destination partition in a topic. A message key can be any value that can be derived from the application context. By default, the message key is passed through a hashing function, which creates the partition assignment. That assures that all **records produced with the same key will arrive at the same partition**. Specifying a message key enables keeping related events together in the same partition and in the exact order in which they were sent. ![Partition Keys](./img/partition_key.png) - **Specifying a partition number**: Alternatively, a [partition number can be set](https://kafka.apache.org/35/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#partition()) to indicate to which partition the message will be sent to. - **Allowing Kafka to decide the partition**: If a producer doesn’t specify a message key or a partition number for partitioning when producing a record, Kafka will use **a round-robin partition assignment**. Those records will be written evenly across all partitions of a particular topic. However, if no message key is used, the ordering of records can not be guaranteed within a given partition. The key takeaway is to use a message key to put related events together in the same partition in the exact order in which they were sent. ## Reading records from partitions Unlike the other pub/sub implementations, Kafka doesn’t push messages to consumers. Instead, consumers have to pull messages off Kafka topic partitions. A consumer connects to a partition in a broker and reads the messages in the order in which they were written. The offset of a message works as a consumer side cursor at this point. The consumer keeps track of which messages it has already consumed by keeping track of the offset of messages. After reading a message, the consumer advances its cursor to the next offset in the partition and continues. Advancing and remembering the last read offset within a partition is the responsibility of the consumer. Kafka has nothing to do with it. The consumer offsets are also stored inside Kafka within an internal topic used by the consumer. ![Offset topics](./img/offset_topics.png) By remembering the offset of the last consumed message for each partition, a consumer can join a partition at the point in time they choose and resume from there. That is particularly useful for a consumer to resume reading after recovering from a crash. A partition can be consumed by one or more consumers, each reading at different offsets. Several consumers are grouped into a **consumer group** to consume a given topic. Consumers in the same consumer group are assigned the same **group-id** value. The consumer group concept ensures that a message is only ever read by a single consumer in the group. ![Consumer offsets](./img/consumer_offsets.png) A new consumer group is created for each application that needs messages from one or more topics. *To scale the reading and processing of messages from the topics new consumers have to be added to an existing consumer group*, so each additional consumer in a group will only get a subset of the messages. *When a consumer group consumes the partitions of a topic, Kafka makes sure that each partition is consumed by **exactly one consumer** in the group.* ![Consumer groups](./img/consumer_groups.png) Consumer groups enable consumers to parallelise and process messages at very high throughputs. **However, the maximum parallelism of a group will be equal to the number of partitions of that topic.** For example, if you have N + 1 consumers for a topic with N partitions, then the first N consumers will be assigned a partition, and the remaining consumer will be idle. ![Consumer group](./img/consumer_group.png) :::tip[Tip] The number of consumers don’t govern the degree of parallelism of a topic — it’s the number of partitions. :::