Apache Kafka 2.8.0 has been released by the Apache Kafka community, bringing many new features and improvements, chief of which is an early access look at a method for using Kafka without Apache ZooKeeper.

Kafka 2.8.0 is the latest release of the open source distributed event streaming platform. It has until now relied on another Apache project, ZooKeeper, for coordination of its distributed nodes.

In this release, the Apache Kafka team is trialling an alternative method that enables users to run Kafka brokers without Apache ZooKeeper, instead depending on an internal implementation of the Raft consensus algorithm. The Kafka team warned that the implementation should not be used in production as it is not yet feature complete, but it is possible to start new clusters without ZooKeeper and go through basic produce and consume use cases.

The changes are outlined in Kafka Improvement Proposal KIP-500. This is intended to work by moving topic metadata and configurations out of ZooKeeper and into a new internal topic named @metadata. This topic is managed by an internal Raft quorum of “controllers” and is replicated to all brokers in the cluster.

According to the Kafka team, the leader of the Raft quorum serves the same role as the controller in clusters today, while a node in this scenario can serve as a controller, a broker, or both, depending on the new process.roles configuration. The team said it expects to see progress in feature completeness and hardening by the mid- and end of year Kafka releases.

Other changes in the nodes include KIP-700, which decouples the Kafka AdminClient from the Metadata API by adding a new API to directly query the brokers for information about the cluster. This change will allow the addition of new admin features in the future without disruption to the producer and consumer.

In Kafka Connect, KIP-661 adds a new API endpoint and method in the Connect REST API to allow callers to retrieve the actual runtime configuration of a connector’s tasks. This can be used for debugging, but also for understanding the impact of failures.

In Kafka Streams, KIP-663 adds new methods to the KafkaStreams interface, which allows users to add and remove individual processing threads without disrupting the other threads running on the same instance. This is to allow the number of threads to be adjusted where a cluster of instances may be experiencing a heavy load.

KIP-671 adds a new handler (StreamsUncaughtExceptionHandler), which offers greater flexibility in handling exceptions, providing a mechanism to replace a dead thread, for example, allowing different actions depending on the actual exception.

These are just some of the fixes and improvements in Apache Kafka 2.8.0. For the full list of changes, see the release notes.