Alpakka Kafka 1.0.x

1.0.1

Released: 2019-03-07

Alpakka Kafka 1.0.1 upgrades the Apache Kafka client to 2.1.1, as version 2.1.0 contained a few annoying bugs which disturb proper operations.

Most notably: Kubernetes - Kafka clients are resolving DNS entries only one time (KAFKA-7755)

Full Apache Kafka 2.1.1 release notes.

The upgrade to the Apache Kafka client 2.1.1 is the only change compared to release 1.0.

General information

This release is compiled and tested against Akka 2.5 and Scala 2.11 and 2.12.

This release was made possible by:

Author Commits Lines added Lines removed
Philippus Philippus 1 1 1
raboof raboof 1 1 1

1.0

Released: 2019-02-28

Final release of Alpakka Kafka 1.0! Theses release notes contain the need-to-know features and changes for Alpakka Kafka since release 0.22.

A bit of “history”

Alpakka Kafka has made a long journey. It started off as reactive kafka with a first release in January 2015 when Akka Streams were still experimental and created by Krzysiek Ciesielski, @kciesielski. The team at SoftwareMill were the first stewards of the project.

In 2016 the Akka team at Lightbend (at that time still called Typesafe) took over the responsibility to take care of it and started offering commercial support from version 0.16 (released May 2017). They were assisted by Alexey Romanchuk, @13h3r during that time. The Akka team has participated significantly in the creation of the Reactive Streams industry standard and Akka Streams has been one of its first implementations.

In May 2018 Lightbend started the Alpakka team to steward the Alpakka project and Alpakka Kafka. When the work of improving the tests and the code for Alpakka Kafka kicked off, we dropped the name it had for a while Akka Streams Kafka.

Apache Kafka has made an incredible journey during this period. From in-house project at LinkedIn via being open-sourced to becoming the integration backbone of many micro-service architectures.

The need for streaming data has been the key driver for all these endeavours.

Features

Alpakka Kafka provides Apache Kafka connectivity for Akka Streams. It supports consuming messages from Kafka into Akka Streams with at-most-once, at-least-once and transactional semantics, and supports producing messages to Kafka.

Once consumed messages are in the Akka Stream, the whole flexibility of all Akka Stream operators becomes available.

Alpakka Kafka achieves back-pressure for consuming by automatically pausing and resuming its Kafka subscriptions. When there is no downstream demand for more data, the consumer will not read any data. Any other communication with the Kafka broker (heartbeats, committing, rebalancing) will still continue.

Alpakka Kafka 1.0 uses the Apache Kafka Java client 2.1.0 internally.

Most important changes since 0.22

  • Upgrade to Kafka client 2.1.0 #660. This upgrade makes it possible to use of the zstandard compression (with Kafka 2.1 brokers). Use Kafka client 2.x poll API #614.

  • No more WakeupException! The Kafka client API 2.x allows for specifying a timeout when polling the Kafka broker, thus we do not need to use the cranky tool of Kafka’s WakeupExceptions to be sure not to block a precious thread. The settings to configure wake-ups are not used anymore.

  • Alpakka Kafka consumers don’t fail for non-responding Kafka brokers anymore (as they used to to after a number of WakeupExceptions).

  • New Committer.sink and Committer.flow for standardised committing #622 and #644

  • Commit with metadata #563 and #579

  • Java APIs for all settings classes #616

  • Alpakka Kafka testkit

For more detailed information on all changes, please refer to the release notes of 1.0-M1, 1.0-RC1, 1.0-RC2.

Found an error in this documentation? The source code for this page can be found here. Please feel free to edit and contribute a pull request.