Alpakka Kafka 1.0
Final release of Alpakka Kafka 1.0
Theses release notes contain the need-to-know features and changes for Alpakka Kafka since release 0.22.
A bit of “history”
Alpakka Kafka has made a long journey. It started off as reactive kafka with a first release in January 2015 when Akka Streams were still experimental and created by Krzysiek Ciesielski, @kciesielski. The team at SoftwareMill were the first stewards of the project.
In 2016 the Akka team at Lightbend (at that time still called Typesafe) took over the responsibility to take care of it and started offering commercial support from version 0.16 (released May 2017). They were assisted by Alexey Romanchuk, @13h3r during that time. The Akka team has participated significantly in the creation of the Reactive Streams industry standard and Akka Streams has been one of its first implementations.
In May 2018 Lightbend started the Alpakka team to steward the Alpakka project and Alpakka Kafka. When the work of improving the tests and the code for Alpakka Kafka kicked off, we dropped the name it had for a while Akka Streams Kafka.
Apache Kafka has made an incredible journey during this period. From in-house project at LinkedIn via being open-sourced to becoming the integration backbone of many micro-service architectures.
The need for streaming data has been the key driver for all these endeavours.
Features
Alpakka Kafka provides Apache Kafka connectivity for Akka Streams. It supports consuming messages from Kafka into Akka Streams with at-most-once, at-least-once and transactional semantics, and supports producing messages to Kafka.
Once consumed messages are in the Akka Stream, the whole flexibility of all Akka Stream operators becomes available.
Alpakka Kafka achieves back-pressure for consuming by automatically pausing and resuming its Kafka subscriptions. When there is no downstream demand for more data, the consumer will not read any data. Any other communication with the Kafka broker (heartbeats, committing, rebalancing) will still continue.
Alpakka Kafka 1.0 uses the Apache Kafka Java client 2.1.0 internally.
Most important changes since 0.22
-
Upgrade to Kafka client 2.1.0 #660. This upgrade makes it possible to use of the zstandard compression (with Kafka 2.1 brokers). Use Kafka client 2.x poll API #614.
-
No more
WakeupException! The Kafka client API 2.x allows for specifying a timeout when polling the Kafka broker, thus we do not need to use the cranky tool of Kafka’sWakeupExceptions to be sure not to block a precious thread. The settings to configure wake-ups are not used anymore. -
Alpakka Kafka consumers don’t fail for non-responding Kafka brokers anymore (as they used to to after a number of
WakeupExceptions). -
New
Committer.sinkandCommitter.flowfor standardised committing #622 and #644 -
Java APIs for all settings classes #616
For more detailed information on all changes, please refer to the release notes of 1.0-M1, 1.0-RC1, 1.0-RC2.