Alpakka Kafka 2.0.x

Note

In case you are browsing a specific version’s documentation: check out the latest release notes

The Alpakka Kafka 2.0 series features

upgrade to the Apache Kafka client 2.4.0
Producer.committableSink behaves as a combination of Producer.flexiFlow and Committer.sink, but has a special-purpose implementation
built-in Akka Discovery support
More versatile use of Testcontainers in Alpakka Kafka Testkit #939.
Removal of some API which was deprecated since 1.0.0 or earlier.
Internal filter to avoid emitting records of revoked partitions #946 and #992

2.0.7

Released: 2021-02-03

Notable changes since 2.0.6

This release only affects the Alpakka Kafka TestKit

Alpakka Kafka Testkit

Increase default cluster start timeouts and make configurable (backport) #1311 by @seglo
Use restart script #1310 by @seglo

The milestone contains everything closed in 2.0.7.

Since 2.0.6 Alpakka Kafka has received contributions by:

Author	Commits	Lines added	Lines removed
seglo	2	89	12

2.0.6

Released: 2020-12-22

Notable changes since 2.0.5

The most relevant change in this release is the update of TestContainers to 1.15.1 (see below)

Handle any combination of Committables in CommitObservationLogic #1262 by @seglo
Fix message-batch loss when rebalancing partitioned sources #1263 by @jhooda
Producer.committableSink: offsets not committed, when MultiMessage is empty #1200 by @herzrasen

Alpakka Kafka TestKit

Update default CP version and make docker image/tag configurable #1287 by @seglo
Testkit: Make Testcontainer logs visible #1281 by @seglo
Testcontainers Kafka 1.15.1 (was 1.14.3) #1257 by @scala-steward
TestKit: make consumerDefaults parameterless again #1292

Dependencies

jackson-databind 2.10.5.1 (was 2.10.5) #1270 by @scala-steward
Add note about provided jackson dependency #1245 by @seglo

2.0.5

Released: 2020-09-09

Notable changes since 2.0.4

Allow typed Actor Systems in SendProducer and DiscoverySupport #1192 by @ennru
Generate version information when publishing artifacts #1188 by @marcospereira

Alpakka Kafka core

Deprecate RestrictedConsumer.committed(TopicPartition) #1178 by @seglo
Cleanup build warnings #1177 by @seglo
Order logging params for RetriableException consumer failures #1168 by @seglo
Generate version information when publishing artifacts #1188 by @marcospereira

Alpakka Kafka Testkit

Cleanup build warnings #1177 by @seglo

Documentation

Run & materialize stream in Akka Cluster Sharding Example #1190 by @seglo
Cleanup testcontainers docs snippets #1180 by @seglo

Tests

Reduce bloat of log when testsuite log reported in failed travis build #1182 by @seglo
Simplify consuming stream for AssignmentSpec “consume from the specified topic pattern” test #1181 by @seglo
Cleanup build warnings #1177 by @seglo

Updates

scalapb-runtime 0.10.8 (was 0.10.7) #1171 by @scala-steward
metrics-core 4.1.12.1 (was 4.1.11) #1194 by @scala-steward
sbt-java-formatter 0.6.0 (was 0.5.1) #1193 by @scala-steward
jackson-databind 2.10.5 (was 2.10.4) #1169 by @scala-steward
mockito-core 3.4.6 (was 2.28.2) #1173 by @scala-steward
sbt-scalafmt 2.4.2 (was 2.4.0) #1174 by @scala-steward
Update paradox akka to 0.35, new links to LB.com #1167 by @johanandren

The milestone contains everything closed in 2.0.5.

Since 2.0.4 Alpakka Kafka has received contributions by:

Author	Commits	Lines added	Lines removed
seglo	7	104	94
scala-steward	7	7	7
ennru	1	75	23
marcospereira	1	36	0
johanandren	1	1	1

2.0.4

Released: 2020-07-22

Notable changes since 2.0.3

Emit offset batches on timer #1160 by @jyates
Do not commit offsets for unassigned partitions #1123 by @jyates
Use bulk update API for shard allocation #1139 by @chbatey
Add produce consume check to KafkaContainerCluster #1131 by @seglo
Deprecate embedded kafka support #1136 by @seglo

Alpakka Kafka core

Emit offset batches on timer #1160 by @jyates
Do not commit offsets for unassigned partitions #1123 by @jyates

Alpakka Kafka Akka Cluster Sharding

Use bulk update API for shard allocation #1139 by @chbatey
Don’t use ctx.log in KafkaClusterSharding #1138 by @chbatey

Alpakka Kafka Testkit

Deprecate embedded kafka support #1136 by @seglo
Replace embedded kafka with schema registry with testcontainers #1135 by @seglo
Add produce consume check to KafkaContainerCluster #1131 by @seglo

Updates

mockito-core 2.28.2 (was 2.24.5) #1151 by @scala-steward
scala-collection-compat 2.1.6 (was 2.1.2) #1152 by @scala-steward
jul-to-slf4j, log4j-over-slf4j 1.7.30 (was 1.7.26) #1155 by @scala-steward
embedded-kafka 2.4.1.1 (was 2.4.1) #1147 by @scala-steward
protobuf-java 3.12.2 (was 3.11.4) #1142 by @scala-steward
sbt 1.3.13 (was 1.3.9) #1153 by @scala-steward
akka-stream-alpakka-csv 2.0.1 (was 2.0.0) #1143 by @scala-steward

The milestone contains everything closed in 2.0.4.

Since 2.0.3 Alpakka Kafka has received contributions by:

Author	Commits	Lines added	Lines removed
ennru	5	1412	354
seglo	4	417	210
jyates	2	157	6
chbatey	2	45	31
MiLk	2	3	3
Zhen-hao	1	3	7
raboof	1	2	0

2.0.3

Released: 2020-05-14

Notable changes since 2.0.2

Akka Cluster Sharding integration to co-locate Kafka partition reading with Akka Cluster shards by @seglo and @chbatey
Retry committing of offsets when failure is marked as retriable #1111 by @ennru
Delay commits until the next offset is observed #1093 by @seglo
Non-streaming producer wrapper #1085 by @ennru
Drain partial offset commit batches on upstream failure #1058 by @sdudzin
Kafka client 2.4.1 #1103 by @ennru

Alpakka Kafka core

Committing: retry commits marked as retriable #1111 by @ennru
Build: Jackson 2.10.4 and sbt plugin updates #1112 by @ennru
Security: mask passwords in settings’ toString #1110 by @ennru
Commit when next offset is observed #1093 by @seglo
Non-streaming producer wrapper #1085 by @ennru
Simplify commit replying #1095 by @ennru
DrainingControl: creation in toMat #1084 by @ennru
Simpler use with the new actors API #1088 by @ennru
Cluster-sharding: single listener per entity type key + Java DSL #1080 by @chbatey
Drain partial offset commit batches on upstream failure #1058 by @sdudzin
Kafka cluster sharding support with external sharding allocation #1067 by @seglo
Add MetadataClient getCommittedOffsets #1073 by @epalace

Alpakka Kafka Testkit

Testkit: Generify test producer consumer #1099 by @claudio-scandura
Testkit: create ProducerMessage.PassThroughResult #1096 by @sebarys
Set stopTimeout to zero in BaseKafkaTest.consumeString. #1094 by @claudio-scandura

The milestone contains everything closed in 2.0.3.

Since 2.0.2 Alpakka Kafka has received contributions by:

Author	Commits	Lines added	Lines removed
ennru	14	1331	560
seglo	6	876	151
claudio-scandura	2	54	17
sdudzin	1	554	46
chbatey	1	224	21
sebarys	1	4	0

2.0.2

Released: 2020-02-20

This release fixes

Alpakka Kafka core

CommittingProducerSink: outstanding commits on multi-msg #1041 by @gabrielreid
CommittingProducerSink: Fix count on failure #1043 by @gabrielreid
Don’t close shared Producer on fail #1046 by @gabrielreid
Java API createCommittableOffsetBatch accepts Committable #1033 by @jewertow
Discard of external offsets on rebalance #1037 by @gabrielreid
Fix slow-loading offsets during rebalance #1029 by @gabrielreid

Alpakka Kafka Testkit

Upgrade to Confluent Platform 5.4.0 #1034 by @seglo

The milestone contains everything closed in 2.0.2.

Since 2.0.1 Alpakka Kafka has received contributions by:

Author	Commits	Lines added	Lines removed
gabrielreid	5	328	14
seglo	5	151	84
ennru	2	18	21
jewertow	1	44	1
ignasi35	1	16	3
Sebruck	1	1	1

2.0.1

Released: 2020-01-23

This release fixes

CommittableProducer: Record outstanding commits on pass-through #1022 by @gabrielreid
Producer: avoid race between failure and upstream finish #1025

The detailed list of changes is found in the 2.0.1 Github release listing.

Since 2.0.0 Alpakka Kafka has received contributions by:

Author	Commits	Lines added	Lines removed
ennru	2	60	44
gabrielreid	1	78	0

2.0.0

Released: 2020-01-15

This release doesn’t contain any relevant changes over 2.0.0-RC1.

The detailed list of changes is found in the 2.0.0 Github release listing.

Since 1.1.0 Alpakka Kafka has received contributions by:

Author	Commits	Lines added	Lines removed
ennru	36	4358	1566
seglo	11	2692	1122
charlibot	1	1901	664
jewertow	1	668	12
def1ne	1	6	0
shobull	1	1	1

2.0.0-RC1

Released: 2019-12-17

Notable changes since 2.0.0-M2

Kafka 2.4.0 Final
- Update to Kafka 2.4.0 final release #971 by @ennru
Bugs
- Rebalance: filter messages of revoked partitions in partitioned sources #992 by @seglo
Alpakka Kafka API
- Stage ID traceability #995 by @seglo

This release contains contributions by:

Author	Commits	Lines added	Lines removed
ennru	5	75	85
seglo	2	494	186

2.0.0-M2

Released: 2019-12-03

Notable changes since 1.1.0

Kafka 2.4.0-RC1
- Kafka 2.4.0-RC1 #971 by @seglo
Producer.committableSink behaves as a combination of Producer.flexiFlow and Committer.sink, but has a special-purpose implementation
- Committing producer sink: producer and committer in a sink stage #963 by @ennru
- Create committableSink as combination of Producer.flexiFlow an… #932 by @ennru
- Committer: change type bound to allow offset batches #931 by @ennru
Akka Discovery support #836 by @ennru
Add wrapper for Metadata requests #497 #900 by @jewertow
Offset batches allow aggregation from multiple consumers #953 by @ennru
Performance
- As part of Apache Kafka: KafkaConsumer should not throw away already fetched data for paused partitions apache/kafka#6988 by @seglo
- Producer: change default parallelism #944 by @ennru
- Rebalance: filter messages of revoked partitions #946 by @ennru
Testkit
- Use testcontainers for multi-broker integration tests and benchmarks #939 by @seglo
- Make EmbeddedKafka a provided dependency #954 by @ennru
Alpakka Kafka API
- Committable: deprecate commitJavadsl and commitScaladsl #959 by @ennru
- ConsumerSetting: Add group instance id setter #861 by @def1ne
Bugs
- Commit refreshing: fix bug from aggregating offsets #975 by @ennru
- Consumer drops records after rebalance #978 band-aided with #997

The detailed list of changes is found in the 2.0.0-M2 Github release listing.

This release is compiled and tested against Akka 2.5.23, 2.6.0 and Scala 2.11, 2.12, 2.13 on Adopt OpenJDK 8 and 11.

This release contains contributions by:

Author	Commits	Lines added	Lines removed
ennru	26	4167	1434
seglo	6	1521	882
charlibot	1	1901	664
jewertow	1	668	12
def1ne	1	6	0
shobull	1	1	1

2.0.0-M1

The release process failed.