The team behind distributed event streaming platform Apache Kafka is currently putting a lot of effort into following through with larger projects such the replacement of cluster management tool ZooKeeper and the introduction of topic identifiers. To avoid leaving devs without a release for too long though, version 3.1 has now been pushed into the open and provides some useful security and usability additions.

To make sure users have a chance of finding brokers that for some reason are alive but can’t establish a connection, Kafka was fitted with new metrics FencedBrokerCount and ActiveBrokerCount. With these, KRaft and ZooKeeper controllers are able to share the number of active and fenced brokers in a cluster they have registered or know about.

Kafka 3.1 also promises production ready support for OIDC, meaning users should be easily able to connect to external identity providers such as Auth0 or Octa, and should work with Java 17. The project’s endeavours to introduce topic identifiers to reduce confusion around topic names progressed as well, as FetchRequests now know how to handle such IDs.

Teams working with latency metrics bufferpool-wait-time-total, io-waittime-total, and iotime-total might run into warnings after updating their system. This is down to a renaming that looks to improve naming consistency and ensure unit clarity – the metrics are now available as bufferpool-wait-time-ns-total, io-waittime-ns-total, and iotime-ns-total. Updating code accordingly is advised, as a removal of the old metrics is planned for the next release.

As usual, the Kafka update includes some enhancements for client library Kafka Streams as well, one of the more helpful being the option to raise range queries for ReadOnlyKeyValueStore, ReadOnlySessionStore and ReadOnlyWindowStore without the need to specify both start and end key. Starting with the release, uncaught exceptions are wrapped as StreamsExceptions with a TaskId field to help identify the source of an exception.

Streams now also offers the option to realise foreign key joins on tables using custom instead of the default partitioners. The custom partitioners will be passed into a new TableJoined object that extends the NamedOperation. Join methods accepting the latter will be replaced with alternatives working with TableJoined, and deprecated. The join implementation for stream-stream joins also saw some reworking and should no longer emit eager left/outer join results. Instead the function will now wait until the end of the window grace period before putting anything out. 

It’s also worth noting that the eager rebalance protocol has been marked for removal with the release which is why users should slowly but surely upgrade to the cooperative protocol instead.

Developers who are waiting for KRaft to finally be production ready will need to be patient for just a bit longer. While getting ready to lose its predecessor ZooKeeper was a big topic in the 3.0 release, and a KRaft mode is already available for testing, important security features are still in development and might need another couple of months until they are where they need to be. Details on the current state of the efforts are available in the project’s repository.