Apache Kafka 3.9.0 Release Summary

History has been made.

This week the final 3.x Kafka release went out. 🔥

It will be the LAST release with ZooKeeper.

Here are the top features you should know about:

Tiered Storage GA 🥳

Tiered Storage finally...

...FINALLY...

turns GA!

It is now 100% production ready. (and honestly, has been running in production for a while in hosted providers like Aiven and AWS)

Final touches that were added with this release were:

• the ability to disable per-topic
• exposing the latest offsets in tiered stores
• quotas
• debug functionality

Tiered Storage is a must-have in Kafka. It can result in:

• better performance via SSDs
• 10-100x faster rebalances
• 10-100x faster recovery times
• the ability to store effectively infinite data in Kafka

I've covered it before, many times:

• https://2minutestreaming.beehiiv.com/p/apache-kafka-kip-405-tiered-storage
• https://www.linkedin.com/posts/stanislavkozlovski_kafka-apachekafka-activity-7072510094537576449-vw6b?utm_source=share&utm_medium=member_desktop
• https://www.linkedin.com/posts/stanislavkozlovski_everyone-is-using-kafka-but-almost-no-activity-7221576100596858882-6eeT

Use Tiered Storage!

Improved ZK Migration

Confluent , as the good steward they are, have migrated thousands of clusters (big and small) from ZK to KRaft. 😰

They’ve found and fixed many bugs throughout the process.

This latest Kafka version contains the most bulletproof migration code that could exist today. 👌

Note that this is the LAST release that'll contain the migration code.

With 4.0 and beyond, there will be no ability to run ZK nor migrate back to ZK.

Before going to 4.0, you'll always need to stop at a bridge release. That is, the release you'll use to migrate from ZK to KRaft.

The best one for that?

3.9.0 👌

Dynamic Controller Quotas

With KRaft, you have a quorum of Kafka brokers that act as controllers.

Previously, you had to:

• statically configure the address for all of the controllers in your cluster (via the properties file)
• change the file and restart the cluster (when you wanted to add/remove controllers)

Now, you can:

• configure a general bootstrap.servers config consisting of one (or a few)
• add and remove controllers as you please via kafka-metadata-quorum.sh while the cluster is running

This flexibilty lets you dynamically modify controllers on the go.

It's simpler. It's safer. 👌

Kafka Streams

• less log spam (KIP-1049)

• better exception handling (KIP-1033)

KIP-1033 is interesting. It allows you to define exception handlers for your business logic in a single class.

I ran through some pseudo-code to see how it compared before and after.

Here it is 👇

It's much nicer! 🔥

Kafka Connect

Last but not least - Connect. Three things:

• New /health endpoint in Kafka Connect for better monitoring (no more hacky connector checks!)

• You can now disable offset syncs in MirrorMaker2

• Better handling of null values in transformations

Looking Back 👀

It's funny, looking back now I realize this release was expected to come a lot sooner.

The 4.0 release has been getting pushed back quite frequently.

But that's how all good (software) things are... usually.

Better late than never! 😉

Looking Forward 🤩

As we end this article, all eyes are on the next MAJOR Kafka release:

4.0.0 👑

Expected end of January, 2025.

fun fact?

The last time Kafka had a major release (3.0) was more than 4 years ago!

September 2021.

Time flies 🥲

Found this useful?

1. Reshare with your network. 🙏
2. Follow me here: ✅ Stanislav Kozlovski
3. Say "Hi" in the comments (I'll say hi in a random language back)
4. For more Kafka & Big Data -> https://2minutestreaming.beehiiv.com/