Developers of the Apache Kafka distributed event streaming platform have now belatedly pushed version 2.7 out of the door, rewarding users’ patience with various features to make the project more reliable.

In version 2.7, the Kafka team tackled the issue of various create and delete operations impacting the overall load on the controller by introducing a resource level quota, letting the system reject all operations that would overload it. Another new protective mechanism includes broker-wide and per-listener connection creation rates.

To ensure users are aware they have deleted a source topic from a running application using Kafka Streams, the app now throws a MissingSourceTopicException instead of shutting it down without any indication. Streams has also seen the addition of reverse state store iteration to make retrieving the latest records a tad more efficient.

After not making it into the 2.6 release, Kafka Streams 2.7 offers developers trace level end-to-end latency metrics such as avg, min, and max to help with architectural choices. A couple of security concerns were also addressed by adding support for the PEM format for storing SSL certificates and private keys, which is meant to open the platform to cooperation with third party providers who rely on the format. 

The Kafka project is working towards getting rid of its ZooKeeper dependencies to make managing metadata more scalable. As such, the development team has added an AlterIsr API. 

The new interface grants the controller – a broker responsible for state management of partitions and replicas – the exclusive right to update the state of partition leaders and in-sync replica (ISR) sets. In earlier versions, this ability was shared between the controller and the partition leader, which led to delays and an incorrect reflection of the ISR state. 

A newly added broker-side API to allow users to change their SCRAM settings also emerged in 2.7, as the old approach made use of ZooKeeper as well. In the long run, ZooKeeper is supposed to be replaced by a self-managed metadata quorum. A core raft protocol implementation for that is now available and can be tested via a stand-alone server until it can be integrated with the controller.

A full list of changes can be found in the Kafka release notes.