Overview

The Alpakka project is an open source initiative to implement stream-aware and reactive integration pipelines for Java and Scala. It is built on top of Akka Streams, and has been designed from the ground up to understand streaming natively and provide a DSL for reactive and stream-oriented programming, with built-in support for backpressure. Akka Streams is a Reactive Streams and JDK 9+ java.util.concurrent.Flow-compliant implementation and therefore fully interoperable with other implementations.

This Alpakka Kafka connector lets you connect Apache Kafka to Akka Streams. It was formerly known as Akka Streams Kafka and even Reactive Kafka.

Project Info

Project Info: Alpakka Kafka
Artifact
com.typesafe.akka
akka-stream-kafka
5.0.0
JDK versions
Eclipse Temurin JDK 11
Eclipse Temurin JDK 17
Scala versions2.13.12, 3.3.1
JPMS module nameakka.stream.alpakka.kafka
License
Readiness level
Since 0.16, 2017-05-02
Home pagehttps://doc.akka.io/docs/alpakka-kafka/current
API documentation
Forums
Release notesIn the documentation
IssuesGithub issues
Sourceshttps://github.com/akka/alpakka-kafka

Matching Kafka Versions

Kafka client Scala Versions Akka version Alpakka Kafka Connector
3.3.1 2.13, 2.12 2.7.0+ release 4.0.0
3.0.1 2.13 2.6.18+ release 3.0.1
3.0.0 2.13 2.6.18+ release 3.0.0 RC1
2.7.0 2.13, 2.12 2.6.14+ release 2.1.0
2.4.1 2.13, 2.12, 2.11 2.5.31+, 2.6.6+ release 2.0.5
2.4.1 2.13, 2.12, 2.11 2.5.30+, 2.6.6+ release 2.0.4
2.4.1 2.13, 2.12, 2.11 2.5.30+, 2.6.3+ release 2.0.3
2.4.0 2.13, 2.12, 2.11 2.5.23+, 2.6.x release 2.0.0
2.1.1 2.13, 2.12, 2.11 2.5.x release 1.0.4
2.1.1 2.12, 2.11 2.5.x release 1.0.1
2.1.0 2.12, 2.11 2.5.x release 1.0
1.1.x 2.12, 2.11 2.5.x release 0.20+
1.0.x 2.12, 2.11 2.5.x release 0.20+
0.11.x 2.12, 2.11 2.5.x release 0.19
Note

As Kafka’s client protocol negotiates the version to use with the Kafka broker, you may use a Kafka client version that is different than the Kafka broker’s version.

These clients can communicate with brokers that are version 2.1.0 or newer. Older or newer brokers may not support certain features. You will receive an UnsupportedVersionException when invoking an API that is not available on the running broker version.

Check even Confluent’s Versions and Interoperability page for more details.

Javadoc for KafkaConsumer

Dependencies

The Akka dependencies are available from Akka’s library repository. To access them there, you need to configure the URL for this repository.

Maven
<project>
  ...
  <repositories>
    <repository>
      <id>akka-repository</id>
      <name>Akka library repository</name>
      <url>https://repo.akka.io/maven</url>
    </repository>
  </repositories>
</project>
sbt
resolvers += "Akka library repository".at("https://repo.akka.io/maven")
Gradle
repositories {
    mavenCentral()
    maven {
        url "https://repo.akka.io/maven"
    }
}

Additionally, add the dependencies as below.

Maven
<properties>
  <akka.version>2.9.0</akka.version>
  <scala.binary.version>2.13</scala.binary.version>
</properties>
<dependencies>
  <dependency>
    <groupId>com.typesafe.akka</groupId>
    <artifactId>akka-stream-kafka_${scala.binary.version}</artifactId>
    <version>5.0.0</version>
  </dependency>
  <dependency>
    <groupId>com.typesafe.akka</groupId>
    <artifactId>akka-stream_${scala.binary.version}</artifactId>
    <version>${akka.version}</version>
  </dependency>
</dependencies>
sbt
val AkkaVersion = "2.9.0"
libraryDependencies ++= Seq(
  "com.typesafe.akka" %% "akka-stream-kafka" % "5.0.0",
  "com.typesafe.akka" %% "akka-stream" % AkkaVersion
)
Gradle
def versions = [
  AkkaVersion: "2.9.0",
  ScalaBinary: "2.13"
]
dependencies {
  implementation "com.typesafe.akka:akka-stream-kafka_${versions.ScalaBinary}:5.0.0"
  implementation "com.typesafe.akka:akka-stream_${versions.ScalaBinary}:${versions.AkkaVersion}"
}

This connector depends on Akka 2.9.0 and note that it is important that all akka-* dependencies are in the same version, so it is recommended to depend on them explicitly to avoid problems with transient dependencies causing an unlucky mix of versions.

Alpakka Kafka APIs accept a typed ActorSystemActorSystem or a classic ActorSystemActorSystem because both implement the ClassicActorSystemProviderClassicActorSystemProvider traitinterface. There are some Alpakka Kafka APIs that only accept classic a ActorRefActorRef, such as the rebalance listener API, but otherwise there is no difference between running Alpakka Kafka and any other Akka Streams implementation with a typed ActorSystemActorSystem. For more information on Akka classic and typed interoperability read the Coexistence page of the Akka Documentation.

The table below shows Alpakka Kafka’s direct dependencies and the second tab shows all libraries it depends on transitively.

Direct dependencies
OrganizationArtifactVersion
com.typesafe.akkaakka-stream_2.132.9.0
org.apache.kafkakafka-clients3.5.1
org.scala-langscala-library2.13.12
Dependency tree
com.typesafe.akka    akka-stream_2.13    2.9.0    BUSL-1.1
    com.typesafe.akka    akka-actor_2.13    2.9.0    BUSL-1.1
        com.typesafe    config    1.4.3    Apache-2.0
        org.scala-lang.modules    scala-java8-compat_2.13    1.0.2    Apache-2.0
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    com.typesafe.akka    akka-protobuf-v3_2.13    2.9.0    BUSL-1.1
    org.reactivestreams    reactive-streams    1.0.4    MIT-0
    org.scala-lang    scala-library    2.13.12    Apache-2.0
org.apache.kafka    kafka-clients    3.5.1    The Apache License, Version 2.0
    com.github.luben    zstd-jni    1.5.5-1    BSD 2-Clause License
    org.lz4    lz4-java    1.8.0    The Apache Software License, Version 2.0
    org.slf4j    slf4j-api    1.7.36
    org.xerial.snappy    snappy-java    1.1.10.1    Apache-2.0
org.scala-lang    scala-library    2.13.12    Apache-2.0

Scala and Java APIs

Following Akka’s conventions there are two separate packages named akka.kafka.scaladsl and akka.kafka.javadsl with the API for Scala and Java. These packages contain Producer and Consumer classes with factory methods for the various Akka Streams Flow, Sink and Source that are producing or consuming messages to/from Kafka.

Examples

A few self-contained examples using Alpakka are available as Alpakka Samples.

To read and see how others use Alpakka see the Alpakka documentation’s Webinars, Presentations and Articles listing.

Contributing

Please feel free to contribute to Alpakka and the Alpakka Kafka connector by reporting issues you identify, or by suggesting changes to the code. Please refer to our contributing instructions to learn how it can be done.

We want Akka and Alpakka to strive in a welcoming and open atmosphere and expect all contributors to respect our code of conduct.

Found an error in this documentation? The source code for this page can be found here. Please feel free to edit and contribute a pull request.