Alpakka Kafka 1.0.x

Note

In case you are browsing a specific version’s documentation: check out the latest release notes

1.0.5

Released: 2019-07-25

Notable changes since 1.0.4

  • FlowWithContext support #780
  • Added connection to broker status checker #674

This release is compiled and tested against Akka 2.5, 2.6 and Scala 2.11, 2.12, 2.13.0 on Adopt OpenJDK 8 and 11.

This release was made possible by:

Author Commits Lines added Lines removed
2m 2m 13 629 213
ennru ennru 7 1327 203
tayvs tayvs 1 397 9
lomigmegard lomigmegard 1 1 1

1.0.4

Released: 2019-06-11

Alpakka Kafka 1.0.4 is released for Scala 2.13, 2.12 and 2.11.

Notable changes since 1.0.3

  • Compile with Scala 2.13 #817
  • Add Committer.batchFlow that emits CommittableOffsetBatch for every committed batch. #799

The detailed list of changes is found in the milestone.

This release is compiled and tested against Akka 2.5 and Scala 2.11, 2.12, 2.13.0 on Adopt OpenJDK 8 and 11.

This release was made possible by:

Author Commits Lines added Lines removed
2m 2m 30 2947 2735
ennru ennru 3 966 301
mowczare mowczare 2 40 32
mowczare Jimmycheong 1 3 3

1.0.3

Released: 2019-05-09

Alpakka Kafka 1.0.3 fixes an important bug in the transactional support and improves the testkit for Java users.

Notable changes since 1.0.2

  • Transactions: Avoid a timeout during draining #787 by Szymon Matejczyk, @szymonm
  • Error handling: Signal exceptions from subscribe to stream, part of #772
  • Testkit: Pull Embedded Kafka from new organisation id, part of #772
  • Testkit: Improve Java API and make timeouts configurable via config #786
  • Testkit: Specify the Docker image version for Testcontainers Kafka #783 as suggested by @miguelpuyol

The detailed list of changes is found in the milestone.

This release is compiled and tested against Akka 2.5 and Scala 2.11 and 2.12 on Adopt OpenJDK 1.8.

This release was made possible by:

Author Commits Lines added Lines removed
2m 2m 17 957 517
szymonm szymonm 2 29 17
ennru ennru 2 15 6

1.0.2

Released: 2019-05-02

Alpakka Kafka 1.0.2 makes the transactional support more reliable.

Notable changes since 1.0.1

The detailed list of changes is found in the milestone.

This release is compiled and tested against Akka 2.5 and Scala 2.11 and 2.12 on Adopt OpenJDK 1.8.

This release was made possible by:

Author Commits Lines added Lines removed
ennru ennru 23 2165 1453
2m 2m 4 147 59
szymonm szymonm 3 669 192
fancywriter fancywriter 1 5 3

1.0.1

Released: 2019-03-07

Alpakka Kafka 1.0.1 upgrades the Apache Kafka client to 2.1.1, as version 2.1.0 contained a few annoying bugs which disturb proper operations.

Most notably: Kubernetes - Kafka clients are resolving DNS entries only one time (KAFKA-7755)

Full Apache Kafka 2.1.1 release notes.

The upgrade to the Apache Kafka client 2.1.1 is the only change compared to release 1.0.

This release is compiled and tested against Akka 2.5 and Scala 2.11 and 2.12.

This release was made possible by:

Author Commits Lines added Lines removed
Philippus Philippus 1 1 1
raboof raboof 1 1 1

1.0

Released: 2019-02-28

Final release of Alpakka Kafka 1.0! Theses release notes contain the need-to-know features and changes for Alpakka Kafka since release 0.22.

A bit of “history”

Alpakka Kafka has made a long journey. It started off as reactive kafka with a first release in January 2015 when Akka Streams were still experimental and created by Krzysiek Ciesielski, @kciesielski. The team at SoftwareMill were the first stewards of the project.

In 2016 the Akka team at Lightbend (at that time still called Typesafe) took over the responsibility to take care of it and started offering commercial support from version 0.16 (released May 2017). They were assisted by Alexey Romanchuk, @13h3r during that time. The Akka team has participated significantly in the creation of the Reactive Streams industry standard and Akka Streams has been one of its first implementations.

In May 2018 Lightbend started the Alpakka team to steward the Alpakka project and Alpakka Kafka. When the work of improving the tests and the code for Alpakka Kafka kicked off, we dropped the name it had for a while Akka Streams Kafka.

Apache Kafka has made an incredible journey during this period. From in-house project at LinkedIn via being open-sourced to becoming the integration backbone of many micro-service architectures.

The need for streaming data has been the key driver for all these endeavours.

Features

Alpakka Kafka provides Apache Kafka connectivity for Akka Streams. It supports consuming messages from Kafka into Akka Streams with at-most-once, at-least-once and transactional semantics, and supports producing messages to Kafka.

Once consumed messages are in the Akka Stream, the whole flexibility of all Akka Stream operators becomes available.

Alpakka Kafka achieves back-pressure for consuming by automatically pausing and resuming its Kafka subscriptions. When there is no downstream demand for more data, the consumer will not read any data. Any other communication with the Kafka broker (heartbeats, committing, rebalancing) will still continue.

Alpakka Kafka 1.0 uses the Apache Kafka Java client 2.1.0 internally.

Most important changes since 0.22

  • Upgrade to Kafka client 2.1.0 #660. This upgrade makes it possible to use of the zstandard compression (with Kafka 2.1 brokers). Use Kafka client 2.x poll API #614.

  • No more WakeupException! The Kafka client API 2.x allows for specifying a timeout when polling the Kafka broker, thus we do not need to use the cranky tool of Kafka’s WakeupExceptions to be sure not to block a precious thread. The settings to configure wake-ups are not used anymore.

  • Alpakka Kafka consumers don’t fail for non-responding Kafka brokers anymore (as they used to to after a number of WakeupExceptions).

  • New Committer.sink and Committer.flow for standardised committing #622 and #644

  • Commit with metadata #563 and #579

  • Java APIs for all settings classes #616

  • Alpakka Kafka testkit

Found an error in this documentation? The source code for this page can be found here. Please feel free to edit and contribute a pull request.