Apache Kafka KRaft Metadata Quorums Explained

Apache Kafka KRaft metadata quorums are the control-plane consensus mechanism that lets Kafka nodes agree on topics, partitions, broker registrations, and other cluster metadata without Apache ZooKeeper. This explainer is for Kafka operators and developers who need to understand what controllers replicate, how a quorum behaves during failures, and which operational checks matter.

Kafka uses KRaft (Kafka Raft) to manage cluster metadata. The Apache Kafka project describes the broader event-streaming system; this article focuses on the smaller but critical controller quorum behind it. Kafka 4.x no longer supports ZooKeeper mode, while existing ZooKeeper clusters must follow the documented bridge-release migration path before upgrading.

Why KRaft Metadata Quorums Are Being Discussed

Many Kafka guides summarize KRaft as “Kafka without ZooKeeper.” That is true, but it hides a key operational distinction: KRaft does not put application events into a new shared controller database. It gives Kafka a built-in, replicated log for the metadata that brokers need in order to agree on the cluster’s structure and configuration.

This matters during controller failures, cluster upgrades, and changes to the controller set. A broker can continue serving some existing partition traffic while metadata updates are stalled, so a cluster that appears partly available may still be unable to create topics or apply other changes. Operators need to distinguish that control-plane condition from a failure of the user-data partitions themselves.

What Is a KRaft Metadata Quorum?

A metadata quorum is the set of Kafka controller replicas that use a Raft-based consensus protocol to elect an active controller and agree on the order of metadata changes. KRaft stores those changes in the __cluster_metadata log. Controllers replicate the log, and the active controller applies committed records to its in-memory view of the cluster.

Metadata includes items such as topic and partition definitions, broker registrations, and configuration changes. It is separate from the records producers append to application topics. Kafka brokers use the resulting metadata to understand partition assignments and cluster state, but the controller quorum is not a replica set for every business event.

In KRaft, each server can be configured as a controller, a broker, or both. Controllers participate in the metadata quorum. Brokers host user-data partitions and communicate with the controller layer; a broker is not automatically a voting controller. The official KRaft operations guide explains the roles and the supported quorum modes in more detail.

The Problem KRaft Solves

Before KRaft, Kafka relied on ZooKeeper to store cluster metadata and coordinate controller activity. That meant operating two distributed systems: Kafka brokers for event data and ZooKeeper for Kafka’s control plane. Teams had to monitor, secure, upgrade, and troubleshoot both, and changes to the Kafka cluster crossed that boundary.

KRaft moves metadata consensus into Kafka itself. This reduces the number of systems an operator must run and gives Kafka a controller log that can be replicated and inspected using Kafka’s own operational tools. It does not eliminate consensus, coordination, or failure planning: the metadata quorum still needs a reachable majority of its voting controllers to elect a leader and commit new metadata.

How the KRaft Quorum Works

The controller quorum maintains one ordered metadata log. An active controller proposes a metadata change, such as a topic creation or partition reassignment. The proposal is replicated to the other voters. Once a majority has replicated the relevant log position, it is committed and can be applied as authoritative cluster state. Followers that have not yet caught up do not make their uncommitted entries authoritative.

For a quorum of three voters, two must be available to make progress; it can tolerate one controller failure. Five voters require three for progress and can tolerate two failures. A majority rule prevents two sides of a network partition from independently committing conflicting metadata histories. It also means that adding more voters does not make the cluster immune to partitions or bad recovery procedures.

When the active controller becomes unavailable, the remaining voters elect a new leader using the replicated log and quorum state. An election can restore metadata progress only if a majority can communicate. If a majority is lost, brokers may retain previously learned metadata and continue some work, but new cluster metadata cannot safely be committed until quorum service is restored.

This is why a controller’s metadata storage is important state. Do not erase or reformat controller directories as a routine troubleshooting step: a set of empty logs must not be allowed to form a replacement quorum that forgets already committed metadata. Use the recovery and migration procedures for the deployed Kafka release, and preserve controller data until the recovery plan is clear.

The underlying consensus protocol is specific to Kafka’s metadata use. KIP-853: KRaft Controller Membership Changes describes how dynamic controller membership keeps voter changes coordinated with the metadata log rather than relying solely on a fixed list in every configuration file.

Key Components and Quorum Modes

• Controller: A Kafka process with the controller role. Controllers participate in elections and metadata replication.
• Active controller: The elected leader that handles metadata changes and replicates them to the other voters.
• Metadata log: The ordered record of cluster metadata changes. It is distinct from the logs for application topics.
• Voter: A controller replica that can vote in elections and count toward the majority needed for progress.
• Observer: A non-voting replica that can replicate metadata without counting toward a voting majority.
• Controller listener: The network endpoint controllers use to communicate. Misconfigured addresses, firewall rules, or security settings can prevent an otherwise healthy set of nodes from forming a quorum.

Kafka supports two quorum configuration models. Static quorums enumerate controller IDs and endpoints in controller.quorum.voters. Dynamic quorums persist the voter set in metadata and use controller.quorum.bootstrap.servers to help servers find the quorum leader. In dynamic mode, the bootstrap list is for discovery; it is not itself the complete voting membership. Follow the guidance for the exact Kafka version and quorum feature level rather than combining the two settings.

Dynamic membership does not remove the need to keep a majority healthy during a change. Plan controller additions and removals so the surviving voters can continue to elect a leader and replicate the metadata log.

Real-World Uses

Every Kafka cluster depends on KRaft, whether it serves event-driven microservices, telemetry, change-data-capture pipelines, or stream processing. The quorum becomes especially visible when operators create topics, change configurations, reassign partitions, register brokers, or upgrade the controller layer.

During an incident, separate controller-quorum health from data-plane health. Producers may still reach partition leaders even as a controller majority is unavailable; this does not mean metadata operations are safe or functioning. Likewise, a healthy metadata quorum cannot compensate for an unavailable partition leader or an under-replicated user-data topic.

Controller placement matters in larger clusters. Dedicated controller nodes can be isolated from broker workload and managed independently. Combined broker/controller nodes are simpler for local development and small deployments, but broker resource pressure can also affect the controllers. The current Kafka KRaft deployment documentation describes the trade-offs and provisioning process.

Practical Guide: Inspect Quorum Health

Begin with the metadata quorum tool shipped with Kafka. From a host that can reach a broker, inspect the current leader, voter set, and high watermark:

bin/kafka-metadata-quorum.sh \
  --bootstrap-server kafka-1.example.net:9092 \
  describe --status

Then inspect replication progress:

bin/kafka-metadata-quorum.sh \
  --bootstrap-server kafka-1.example.net:9092 \
  describe --replication

Use the output to answer three questions: Is there a leader? Which controllers are current voters? Are followers caught up closely enough to participate in recovery? Compare repeated observations with controller logs and network telemetry; a single snapshot does not show whether lag is growing or recovering.

For a dynamic quorum, representative controller properties for one node look like this:

process.roles=controller
node.id=1
listeners=CONTROLLER://controller-1.example.net:9093
controller.listener.names=CONTROLLER
controller.quorum.bootstrap.servers=controller-1.example.net:9093,controller-2.example.net:9093,controller-3.example.net:9093
listener.security.protocol.map=CONTROLLER:PLAINTEXT
metadata.log.dir=/var/lib/kafka/metadata

This illustrates the relevant settings, not a complete production security configuration. Give each controller a unique node ID and listener address, configure broker nodes to discover the same quorum, and replace plaintext controller communication with the authentication and encryption required by the environment. Provision and format storage using the commands for the selected Kafka version; formatting an existing data directory is not a safe way to repair a damaged quorum.

If a controller is being added or removed, first confirm the current quorum is healthy and follow the version’s documented membership procedure. For a dynamic quorum, wait for the new controller to catch up before treating it as a voter, and remove a controller from membership before shutting it down. For static quorums, keep the configured voter list consistent across nodes. Never test controller loss or quorum recovery on a production cluster as an improvised diagnostic.

For ZooKeeper-based clusters, do not convert a live deployment by only changing configuration keys or copying metadata files. Kafka’s ZooKeeper-to-KRaft migration procedure requires a bridge release; the last bridge release is Kafka 3.9. Complete the supported migration steps before moving to Kafka 4.x, which no longer supports ZooKeeper mode.

Common Misconceptions

“The metadata quorum stores application events.”

It stores cluster metadata, not the payload records in application topics. Topic partitions have their own leaders and replicas. A healthy controller quorum does not guarantee that every data partition is available.

“Three controllers means any two can be shut down.”

Two of three voters are needed for a majority. If only one remains available, it cannot safely elect a leader and commit metadata changes. Controller maintenance should be planned one node at a time with quorum health checked between steps.

“A Kafka broker is automatically a quorum voter.”

Broker and controller are separate process roles. A server can run both, but a broker-only node does not count toward the controller majority. A consumer group’s coordinator is also a different role: it tracks group membership and offsets, not the replicated cluster metadata log.

Related Articles

• Start with event streaming with Apache Kafka for topics, partitions, and broker replication.
• Learn how Kafka consumer groups and partition rebalancing coordinate readers and offsets.
• See the separate transaction boundary in Kafka transactions and exactly-once processing.
• Review distributed system failures and network partitions for the failure conditions that quorum consensus is designed to tolerate.

Changelog

• 2026-10-12: First publication.