Alpakka Kafka 2.0.x

Note

In case you are browsing a specific version’s documentation: check out the latest release notes

The Alpakka Kafka 2.0 series features

  • upgrade to the Apache Kafka client 2.4.0
  • Producer.committableSink behaves as a combination of Producer.flexiFlow and Committer.sink, but has a special-purpose implementation
  • built-in Akka Discovery support
  • More versatile use of Testcontainers in Alpakka Kafka Testkit #939.
  • Removal of some API which was deprecated since 1.0.0 or earlier.
  • Internal filter to avoid emitting records of revoked partitions #946 and #992

2.0.7

Released: 2021-02-03

Notable changes since 2.0.6

This release only affects the Alpakka Kafka TestKit

Alpakka Kafka Testkit

  • Increase default cluster start timeouts and make configurable (backport) #1311 by @seglo
  • Use restart script #1310 by @seglo

The milestone contains everything closed in 2.0.7.

Since 2.0.6 Alpakka Kafka has received contributions by:

Author Commits Lines added Lines removed
seglo seglo 2 89 12

2.0.6

Released: 2020-12-22

Notable changes since 2.0.5

The most relevant change in this release is the update of TestContainers to 1.15.1 (see below)

  • Handle any combination of Committables in CommitObservationLogic #1262 by @seglo
  • Fix message-batch loss when rebalancing partitioned sources #1263 by @jhooda
  • Producer.committableSink: offsets not committed, when MultiMessage is empty #1200 by @herzrasen

Alpakka Kafka TestKit

  • Update default CP version and make docker image/tag configurable #1287 by @seglo
  • Testkit: Make Testcontainer logs visible #1281 by @seglo
  • Testcontainers Kafka 1.15.1 (was 1.14.3) #1257 by @scala-steward
  • TestKit: make consumerDefaults parameterless again #1292

Dependencies

2.0.5

Released: 2020-09-09

Notable changes since 2.0.4

  • Allow typed Actor Systems in SendProducer and DiscoverySupport #1192 by @ennru
  • Generate version information when publishing artifacts #1188 by @marcospereira

Alpakka Kafka core

  • Deprecate RestrictedConsumer.committed(TopicPartition) #1178 by @seglo
  • Cleanup build warnings #1177 by @seglo
  • Order logging params for RetriableException consumer failures #1168 by @seglo
  • Generate version information when publishing artifacts #1188 by @marcospereira

Alpakka Kafka Testkit

Documentation

  • Run & materialize stream in Akka Cluster Sharding Example #1190 by @seglo
  • Cleanup testcontainers docs snippets #1180 by @seglo

Tests

  • Reduce bloat of log when testsuite log reported in failed travis build #1182 by @seglo
  • Simplify consuming stream for AssignmentSpec “consume from the specified topic pattern” test #1181 by @seglo
  • Cleanup build warnings #1177 by @seglo

Updates

The milestone contains everything closed in 2.0.5.

Since 2.0.4 Alpakka Kafka has received contributions by:

Author Commits Lines added Lines removed
seglo seglo 7 104 94
scala-steward scala-steward 7 7 7
ennru ennru 1 75 23
marcospereira marcospereira 1 36 0
johanandren johanandren 1 1 1

2.0.4

Released: 2020-07-22

Notable changes since 2.0.3

Alpakka Kafka core

Alpakka Kafka Akka Cluster Sharding

Alpakka Kafka Testkit

  • Deprecate embedded kafka support #1136 by @seglo
  • Replace embedded kafka with schema registry with testcontainers #1135 by @seglo
  • Add produce consume check to KafkaContainerCluster #1131 by @seglo

Updates

The milestone contains everything closed in 2.0.4.

Since 2.0.3 Alpakka Kafka has received contributions by:

Author Commits Lines added Lines removed
ennru ennru 5 1412 354
seglo seglo 4 417 210
jyates jyates 2 157 6
chbatey chbatey 2 45 31
MiLk MiLk 2 3 3
Zhen-hao Zhen-hao 1 3 7
raboof raboof 1 2 0

2.0.3

Released: 2020-05-14

Notable changes since 2.0.2

Alpakka Kafka core

  • Committing: retry commits marked as retriable #1111 by @ennru
  • Build: Jackson 2.10.4 and sbt plugin updates #1112 by @ennru
  • Security: mask passwords in settings’ toString #1110 by @ennru
  • Commit when next offset is observed #1093 by @seglo
  • Non-streaming producer wrapper #1085 by @ennru
  • Simplify commit replying #1095 by @ennru
  • DrainingControl: creation in toMat #1084 by @ennru
  • Simpler use with the new actors API #1088 by @ennru
  • Cluster-sharding: single listener per entity type key + Java DSL #1080 by @chbatey
  • Drain partial offset commit batches on upstream failure #1058 by @sdudzin
  • Kafka cluster sharding support with external sharding allocation #1067 by @seglo
  • Add MetadataClient getCommittedOffsets #1073 by @epalace

Alpakka Kafka Testkit

The milestone contains everything closed in 2.0.3.

Since 2.0.2 Alpakka Kafka has received contributions by:

Author Commits Lines added Lines removed
ennru ennru 14 1331 560
seglo seglo 6 876 151
claudio-scandura claudio-scandura 2 54 17
sdudzin sdudzin 1 554 46
chbatey chbatey 1 224 21
sebarys sebarys 1 4 0

2.0.2

Released: 2020-02-20

This release fixes

Alpakka Kafka core

Alpakka Kafka Testkit

The milestone contains everything closed in 2.0.2.

Since 2.0.1 Alpakka Kafka has received contributions by:

Author Commits Lines added Lines removed
gabrielreid gabrielreid 5 328 14
seglo seglo 5 151 84
ennru ennru 2 18 21
jewertow jewertow 1 44 1
ignasi35 ignasi35 1 16 3
Sebruck Sebruck 1 1 1

2.0.1

Released: 2020-01-23

This release fixes

  • CommittableProducer: Record outstanding commits on pass-through #1022 by @gabrielreid
  • Producer: avoid race between failure and upstream finish #1025

The detailed list of changes is found in the 2.0.1 Github release listing.

Since 2.0.0 Alpakka Kafka has received contributions by:

Author Commits Lines added Lines removed
ennru ennru 2 60 44
gabrielreid gabrielreid 1 78 0

2.0.0

Released: 2020-01-15

This release doesn’t contain any relevant changes over 2.0.0-RC1.

The detailed list of changes is found in the 2.0.0 Github release listing.

Since 1.1.0 Alpakka Kafka has received contributions by:

Author Commits Lines added Lines removed
ennru ennru 36 4358 1566
seglo seglo 11 2692 1122
charlibot charlibot 1 1901 664
jewertow jewertow 1 668 12
def1ne def1ne 1 6 0
shobull shobull 1 1 1

2.0.0-RC1

Released: 2019-12-17

Notable changes since 2.0.0-M2

  • Kafka 2.4.0 Final

    • Update to Kafka 2.4.0 final release #971 by @ennru
  • Bugs

    • Rebalance: filter messages of revoked partitions in partitioned sources #992 by @seglo
  • Alpakka Kafka API

This release contains contributions by:

Author Commits Lines added Lines removed
ennru ennru 5 75 85
seglo seglo 2 494 186

2.0.0-M2

Released: 2019-12-03

Notable changes since 1.1.0

  • Kafka 2.4.0-RC1

  • Producer.committableSink behaves as a combination of Producer.flexiFlow and Committer.sink, but has a special-purpose implementation

    • Committing producer sink: producer and committer in a sink stage #963 by @ennru
    • Create committableSink as combination of Producer.flexiFlow an… #932 by @ennru
    • Committer: change type bound to allow offset batches #931 by @ennru
  • Akka Discovery support #836 by @ennru

  • Add wrapper for Metadata requests #497 #900 by @jewertow

  • Offset batches allow aggregation from multiple consumers #953 by @ennru

  • Performance

    • As part of Apache Kafka: KafkaConsumer should not throw away already fetched data for paused partitions apache/kafka#6988 by @seglo
    • Producer: change default parallelism #944 by @ennru
    • Rebalance: filter messages of revoked partitions #946 by @ennru
  • Testkit

    • Use testcontainers for multi-broker integration tests and benchmarks #939 by @seglo
    • Make EmbeddedKafka a provided dependency #954 by @ennru
  • Alpakka Kafka API

    • Committable: deprecate commitJavadsl and commitScaladsl #959 by @ennru
    • ConsumerSetting: Add group instance id setter #861 by @def1ne
  • Bugs

    • Commit refreshing: fix bug from aggregating offsets #975 by @ennru
    • Consumer drops records after rebalance #978 band-aided with #997

The detailed list of changes is found in the 2.0.0-M2 Github release listing.

This release is compiled and tested against Akka 2.5.23, 2.6.0 and Scala 2.11, 2.12, 2.13 on Adopt OpenJDK 8 and 11.

This release contains contributions by:

Author Commits Lines added Lines removed
ennru ennru 26 4167 1434
seglo seglo 6 1521 882
charlibot charlibot 1 1901 664
jewertow jewertow 1 668 12
def1ne def1ne 1 6 0
shobull shobull 1 1 1

2.0.0-M1

The release process failed.

Found an error in this documentation? The source code for this page can be found here. Please feel free to edit and contribute a pull request.