You are viewing documentation for an older version (3.7) of Kafka. For up-to-date documentation, see the latest version.
In KRaft mode each Kafka server can be configured as a controller, a broker, or both using the process.roles
property. This property can have the following values:
process.roles
is set to broker
, the server acts as a broker.process.roles
is set to controller
, the server acts as a controller.process.roles
is set to broker,controller
, the server acts as both a broker and a controller.process.roles
is not set at all, it is assumed to be in ZooKeeper mode.Kafka servers that act as both brokers and controllers are referred to as “combined” servers. Combined servers are simpler to operate for small use cases like a development environment. The key disadvantage is that the controller will be less isolated from the rest of the system. For example, it is not possible to roll or scale the controllers separately from the brokers in combined mode. Combined mode is not recommended in critical deployment environments.
In KRaft mode, specific Kafka servers are selected to be controllers (unlike the ZooKeeper-based mode, where any server can become the Controller). The servers selected to be controllers will participate in the metadata quorum. Each controller is either an active or a hot standby for the current active controller.
A Kafka admin will typically select 3 or 5 servers for this role, depending on factors like cost and the number of concurrent failures your system should withstand without availability impact. A majority of the controllers must be alive in order to maintain availability. With 3 controllers, the cluster can tolerate 1 controller failure; with 5 controllers, the cluster can tolerate 2 controller failures.
All of the servers in a Kafka cluster discover the quorum voters using the controller.quorum.voters
property. This identifies the quorum controller servers that should be used. All the controllers must be enumerated. Each controller is identified with their id
, host
and port
information. For example:
controller.quorum.voters=id1@host1:port1,id2@host2:port2,id3@host3:port3
If a Kafka cluster has 3 controllers named controller1, controller2 and controller3, then controller1 may have the following configuration:
process.roles=controller
node.id=1
listeners=CONTROLLER://controller1.example.com:9093
controller.quorum.voters=1@controller1.example.com:9093,2@controller2.example.com:9093,3@controller3.example.com:9093
Every broker and controller must set the controller.quorum.voters
property. The node ID supplied in the controller.quorum.voters
property must match the corresponding id on the controller servers. For example, on controller1, node.id must be set to 1, and so forth. Each node ID must be unique across all the servers in a particular cluster. No two servers can have the same node ID regardless of their process.roles
values.
The kafka-storage.sh random-uuid
command can be used to generate a cluster ID for your new cluster. This cluster ID must be used when formatting each server in the cluster with the kafka-storage.sh format
command.
This is different from how Kafka has operated in the past. Previously, Kafka would format blank storage directories automatically, and also generate a new cluster ID automatically. One reason for the change is that auto-formatting can sometimes obscure an error condition. This is particularly important for the metadata log maintained by the controller and broker servers. If a majority of the controllers were able to start with an empty log directory, a leader might be able to be elected with missing committed data.
The kafka-metadata-quorum
tool can be used to describe the runtime state of the cluster metadata partition. For example, the following command displays a summary of the metadata quorum:
> bin/kafka-metadata-quorum.sh --bootstrap-server broker_host:port describe --status
ClusterId: fMCL8kv1SWm87L_Md-I2hg
LeaderId: 3002
LeaderEpoch: 2
HighWatermark: 10
MaxFollowerLag: 0
MaxFollowerLagTimeMs: -1
CurrentVoters: [3000,3001,3002]
CurrentObservers: [0,1,2]
The kafka-dump-log
tool can be used to debug the log segments and snapshots for the cluster metadata directory. The tool will scan the provided files and decode the metadata records. For example, this command decodes and prints the records in the first log segment:
> bin/kafka-dump-log.sh --cluster-metadata-decoder --files metadata_log_dir/__cluster_metadata-0/00000000000000000000.log
This command decodes and prints the records in the a cluster metadata snapshot:
> bin/kafka-dump-log.sh --cluster-metadata-decoder --files metadata_log_dir/__cluster_metadata-0/00000000000000000100-0000000001.checkpoint
The kafka-metadata-shell
tool can be used to interactively inspect the state of the cluster metadata partition:
> bin/kafka-metadata-shell.sh --snapshot metadata_log_dir/__cluster_metadata-0/00000000000000000000.log
>> ls /
brokers local metadataQuorum topicIds topics
>> ls /topics
foo
>> cat /topics/foo/0/data
{
"partitionId" : 0,
"topicId" : "5zoAlv-xEh9xRANKXt1Lbg",
"replicas" : [ 1 ],
"isr" : [ 1 ],
"removingReplicas" : null,
"addingReplicas" : null,
"leader" : 1,
"leaderEpoch" : 0,
"partitionEpoch" : 0
}
>> exit
process.role
should be set to either broker
or controller
but not both. Combined mode can be used in development environments, but it should be avoided in critical deployment environments.The following features are not fully implemented in KRaft mode:
ZooKeeper to KRaft migration is considered an Early Access feature and is not recommended for production clusters. Please report issues with ZooKeeper to KRaft migration using the project JIRA and the “kraft” component.
In general, the migration process passes through several phases.
Before beginning the migration, the Kafka brokers must be upgraded to software version 3.7.2 and have the “inter.broker.protocol.version” configuration set to “3.7”.
It is recommended to enable TRACE level logging for the migration components while the migration is active. This can be done by adding the following log4j configuration to each KRaft controller’s “log4j.properties” file.
log4j.logger.org.apache.kafka.metadata.migration=TRACE
It is generally useful to enable DEBUG logging on the KRaft controllers and the ZK brokers during the migration.
Two things are needed before the migration can begin. First, the brokers must be configured to support the migration and second, a KRaft controller quorum must be deployed. The KRaft controllers should be provisioned with the same cluster ID as the existing Kafka cluster. This can be found by examining one of the “meta.properties” files in the data directories of the brokers, or by running the following command.
./bin/zookeeper-shell.sh localhost:2181 get /cluster/id
The KRaft controller quorum should also be provisioned with the latest metadata.version
. This is done automatically when you format the node with the kafka-storage.sh
tool. For further instructions on KRaft deployment, please refer to the above documentation.
In addition to the standard KRaft configuration, the KRaft controllers will need to enable support for the migration as well as provide ZooKeeper connection configuration.
Here is a sample config for a KRaft controller that is ready for migration:
# Sample KRaft cluster controller.properties listening on 9093
process.roles=controller
node.id=3000
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
# Enable the migration
zookeeper.metadata.migration.enable=true
# ZooKeeper client configuration
zookeeper.connect=localhost:2181
# The inter broker listener in brokers to allow KRaft controller send RPCs to brokers
inter.broker.listener.name=PLAINTEXT
# Other configs ...
Note: The KRaft clusternode.id
values must be different from any existing ZK broker broker.id
. In KRaft-mode, the brokers and controllers share the same Node ID namespace.
Once the KRaft controller quorum has been started, the brokers will need to be reconfigured and restarted. Brokers may be restarted in a rolling fashion to avoid impacting cluster availability. Each broker requires the following configuration to communicate with the KRaft controllers and to enable the migration.
Here is a sample config for a broker that is ready for migration:
# Sample ZK broker server.properties listening on 9092
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://localhost:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
# Set the IBP
inter.broker.protocol.version=3.7
# Enable the migration
zookeeper.metadata.migration.enable=true
# ZooKeeper client configuration
zookeeper.connect=localhost:2181
# KRaft controller quorum configuration
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER
Note: Once the final ZK broker has been restarted with the necessary configuration, the migration will automatically begin. When the migration is complete, an INFO level log can be observed on the active controller:
Completed migration of metadata from Zookeeper to KRaft
Once the KRaft controller completes the metadata migration, the brokers will still be running in ZooKeeper mode. While the KRaft controller is in migration mode, it will continue sending controller RPCs to the ZooKeeper mode brokers. This includes RPCs like UpdateMetadata and LeaderAndIsr.
To migrate the brokers to KRaft, they simply need to be reconfigured as KRaft brokers and restarted. Using the above broker configuration as an example, we would replace the broker.id
with node.id
and add process.roles=broker
. It is important that the broker maintain the same Broker/Node ID when it is restarted. The zookeeper configurations should be removed at this point. Finally, if you have set control.plane.listener.name
. please remove it before restarting in KRaft mode.
If your broker has authorization configured via the authorizer.class.name
property using kafka.security.authorizer.AclAuthorizer
, this is also the time to change it to use org.apache.kafka.metadata.authorizer.StandardAuthorizer
instead.
# Sample KRaft broker server.properties listening on 9092
process.roles=broker
node.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://localhost:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
# Don't set the IBP, KRaft uses "metadata.version" feature flag
# inter.broker.protocol.version=3.7
# Remove the migration enabled flag
# zookeeper.metadata.migration.enable=true
# Remove ZooKeeper client configuration
# zookeeper.connect=localhost:2181
# Keep the KRaft controller quorum configuration
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER
Each broker is restarted with a KRaft configuration until the entire cluster is running in KRaft mode.
Once all brokers have been restarted in KRaft mode, the last step to finalize the migration is to take the KRaft controllers out of migration mode. This is done by removing the “zookeeper.metadata.migration.enable” property from each of their configs and restarting them one at a time.
Once the migration has been finalized, you can safely deprovision your ZooKeeper cluster, assuming you are not using it for anything else. After this point, it is no longer possible to revert to ZooKeeper mode.
# Sample KRaft cluster controller.properties listening on 9093
process.roles=controller
node.id=3000
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
# Disable the migration
# zookeeper.metadata.migration.enable=true
# Remove ZooKeeper client configuration
# zookeeper.connect=localhost:2181
# Other configs ...
While the cluster is still in migration mode, it is possible to revert to ZooKeeper mode. The process to follow depends on how far the migration has progressed. In order to find out how to revert, select the final migration step that you have completed in this table.
Note that the directions given here assume that each step was fully completed, and they were done in order. So, for example, we assume that if “Enter Migration Mode on the Brokers” was completed, “Provisioning the KRaft controller quorum” was also fully completed previously.
If you did not fully complete any step, back out whatever you have done and then follow revert directions for the last fully completed step.
Final Migration Section Completed | Directions for Reverting | Notes |
---|---|---|
Preparing for migration | The preparation section does not involve leaving ZooKeeper mode. So there is nothing to do in the case of a revert. | |
Provisioning the KRaft controller quorum |
|
Enter Migration Mode on the brokers |
zookeeper-shell.sh
, run rmr /controller
so that one of the brokers can become the new old-style controller.zookeeper.metadata.migration.enable
, controller.listener.names
, and controller.quorum.voters
configurations, and replace node.id
with broker.id
. Then perform a rolling restart of all brokers.| It is important to perform the zookeeper-shell.sh
step quickly, to minimize the amount of time that the cluster lacks a controller.
Migrating brokers to KRaft |
process.roles
configuration, and restore the zookeeper.connect
configuration to its previous value. If your cluster requires other ZooKeeper configurations for brokers, such as zookeeper.ssl.protocol
, re-add those configurations as well. Then perform a rolling restart of all brokers.zookeeper-shell.sh
, run rmr /controller
so that one of the brokers can become the new old-style controller.zookeeper.metadata.migration.enable
, controller.listener.names
, and controller.quorum.voters
configurations. Replace node.id
with broker.id
. Then perform a second rolling restart of all brokers.|
zookeeper-shell.sh
step quickly , to minimize the amount of time that the cluster lacks a controller.zookeeper.metadata.migration.enable
remains set to true
. Do not set it to false until the second cluster roll.Finalizing the migration | If you have finalized the ZK migration, then you cannot revert. | Some users prefer to wait for a week or two before finalizing the migration. While this requires you to keep the ZooKeeper cluster running for a while longer, it may be helpful in validating KRaft mode in your cluster.
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.