HBase

The connector provides sources, flows and sinks to interact with HBase database.

HBase is a column family NoSQL Database backed by HDFS. For more information about HBase, please visit the HBase documentation.

Project Info: Alpakka HBase
Artifact
com.lightbend.akka
akka-stream-alpakka-hbase
2.0.1
JDK versions
Adopt OpenJDK 8
Adopt OpenJDK 11
Scala versions2.12.10, 2.11.12, 2.13.1
JPMS module nameakka.stream.alpakka.hbase
License
Readiness level
Since 0.4, 2016-12-22
Home pagehttps://doc.akka.io/docs/alpakka/current
API documentation
Forums
Release notesIn the documentation
IssuesGithub issues
Sourceshttps://github.com/akka/alpakka

Artifacts

sbt
val AkkaVersion = "2.5.31"
libraryDependencies ++= Seq(
  "com.lightbend.akka" %% "akka-stream-alpakka-hbase" % "2.0.1",
  "com.typesafe.akka" %% "akka-stream" % AkkaVersion
)
Maven
<properties>
  <akka.version>2.5.31</akka.version>
  <scala.binary.version>2.12</scala.binary.version>
</properties>
<dependency>
  <groupId>com.lightbend.akka</groupId>
  <artifactId>akka-stream-alpakka-hbase_${scala.binary.version}</artifactId>
  <version>2.0.1</version>
</dependency>
<dependency>
  <groupId>com.typesafe.akka</groupId>
  <artifactId>akka-stream_${scala.binary.version}</artifactId>
  <version>${akka.version}</version>
</dependency>
Gradle
versions += [
  AkkaVersion: "2.5.31",
  ScalaBinary: "2.12"
]
dependencies {
  compile group: 'com.lightbend.akka', name: "akka-stream-alpakka-hbase_${versions.ScalaBinary}", version: '2.0.1',
  compile group: 'com.typesafe.akka', name: "akka-stream_${versions.ScalaBinary}", version: versions.AkkaVersion
}

The table below shows direct dependencies of this module and the second tab shows all libraries it depends on transitively.

Direct dependencies
OrganizationArtifactVersion
com.typesafe.akkaakka-stream_2.122.5.31
org.apache.hadoophadoop-common2.7.4
org.apache.hadoophadoop-mapreduce-client-core2.7.4
org.apache.hbasehbase-common1.4.9
org.apache.hbasehbase-shaded-client1.4.9
org.scala-langscala-library2.12.10
Dependency tree
com.typesafe.akka    akka-stream_2.12    2.5.31
    com.typesafe.akka    akka-actor_2.12    2.5.31
        com.typesafe    config    1.3.3
        org.scala-lang.modules    scala-java8-compat_2.12    0.8.0
    com.typesafe.akka    akka-protobuf_2.12    2.5.31
    com.typesafe    ssl-config-core_2.12    0.3.8
        com.typesafe    config    1.3.3
        org.scala-lang.modules    scala-parser-combinators_2.12    1.1.2
    org.reactivestreams    reactive-streams    1.0.2
org.apache.hadoop    hadoop-common    2.7.4
    com.google.code.findbugs    jsr305    3.0.0
    com.google.code.gson    gson    2.2.4
    com.google.guava    guava    16.0.1
    com.google.protobuf    protobuf-java    2.5.0
    com.jcraft    jsch    0.1.54
    com.sun.jersey    jersey-core    1.9
    com.sun.jersey    jersey-json    1.9
        com.sun.jersey    jersey-core    1.9
        com.sun.xml.bind    jaxb-impl    2.2.3-1
            javax.xml.bind    jaxb-api    2.2.2
                javax.activation    activation    1.1
                javax.xml.stream    stax-api    1.0-2
        org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-jaxrs    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
            org.codehaus.jackson    jackson-mapper-asl    1.9.13
                org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-mapper-asl    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-xc    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
            org.codehaus.jackson    jackson-mapper-asl    1.9.13
                org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jettison    jettison    1.1
    com.sun.jersey    jersey-server    1.9
        asm    asm    3.1
        com.sun.jersey    jersey-core    1.9
    commons-cli    commons-cli    1.2
    commons-codec    commons-codec    1.9
    commons-collections    commons-collections    3.2.2
    commons-configuration    commons-configuration    1.6
        commons-beanutils    commons-beanutils-core    1.8.0
            commons-logging    commons-logging    1.2
        commons-collections    commons-collections    3.2.2
        commons-digester    commons-digester    1.8
            commons-beanutils    commons-beanutils    1.7.0
                commons-logging    commons-logging    1.2
            commons-logging    commons-logging    1.2
        commons-lang    commons-lang    2.6
        commons-logging    commons-logging    1.2
    commons-httpclient    commons-httpclient    3.1
        commons-codec    commons-codec    1.9
        commons-logging    commons-logging    1.2
    commons-io    commons-io    2.4
    commons-lang    commons-lang    2.6
    commons-logging    commons-logging    1.2
    commons-net    commons-net    3.1
    javax.servlet.jsp    jsp-api    2.1
    javax.servlet    servlet-api    2.5
    net.java.dev.jets3t    jets3t    0.9.0
        com.jamesmurty.utils    java-xmlbuilder    0.4
        commons-codec    commons-codec    1.9
        commons-logging    commons-logging    1.2
        org.apache.httpcomponents    httpclient    4.2.5
            commons-codec    commons-codec    1.9
            commons-logging    commons-logging    1.2
            org.apache.httpcomponents    httpcore    4.2.4
        org.apache.httpcomponents    httpcore    4.2.4
    org.apache.avro    avro    1.7.7
        com.thoughtworks.paranamer    paranamer    2.3
        org.apache.commons    commons-compress    1.4.1
            org.tukaani    xz    1.0
        org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-mapper-asl    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
        org.slf4j    slf4j-api    1.7.30
        org.xerial.snappy    snappy-java    1.0.5
    org.apache.commons    commons-compress    1.4.1
        org.tukaani    xz    1.0
    org.apache.commons    commons-math3    3.1.1
    org.apache.curator    curator-client    2.7.1
        com.google.guava    guava    16.0.1
        org.apache.zookeeper    zookeeper    3.4.6
            io.netty    netty    3.7.0.Final
            jline    jline    0.9.94
            org.slf4j    slf4j-api    1.7.30
        org.slf4j    slf4j-api    1.7.30
    org.apache.curator    curator-recipes    2.7.1
        com.google.guava    guava    16.0.1
        org.apache.curator    curator-framework    2.7.1
            com.google.guava    guava    16.0.1
            org.apache.curator    curator-client    2.7.1
                com.google.guava    guava    16.0.1
                org.apache.zookeeper    zookeeper    3.4.6
                    io.netty    netty    3.7.0.Final
                    jline    jline    0.9.94
                    org.slf4j    slf4j-api    1.7.30
                org.slf4j    slf4j-api    1.7.30
            org.apache.zookeeper    zookeeper    3.4.6
                io.netty    netty    3.7.0.Final
                jline    jline    0.9.94
                org.slf4j    slf4j-api    1.7.30
        org.apache.zookeeper    zookeeper    3.4.6
            io.netty    netty    3.7.0.Final
            jline    jline    0.9.94
            org.slf4j    slf4j-api    1.7.30
    org.apache.hadoop    hadoop-annotations    2.7.4
    org.apache.hadoop    hadoop-auth    2.7.4
        commons-codec    commons-codec    1.9
        org.apache.curator    curator-framework    2.7.1
            com.google.guava    guava    16.0.1
            org.apache.curator    curator-client    2.7.1
                com.google.guava    guava    16.0.1
                org.apache.zookeeper    zookeeper    3.4.6
                    io.netty    netty    3.7.0.Final
                    jline    jline    0.9.94
                    org.slf4j    slf4j-api    1.7.30
                org.slf4j    slf4j-api    1.7.30
            org.apache.zookeeper    zookeeper    3.4.6
                io.netty    netty    3.7.0.Final
                jline    jline    0.9.94
                org.slf4j    slf4j-api    1.7.30
        org.apache.directory.server    apacheds-kerberos-codec    2.0.0-M15
            org.apache.directory.api    api-asn1-api    1.0.0-M20
                org.slf4j    slf4j-api    1.7.30
            org.apache.directory.api    api-util    1.0.0-M20
                org.slf4j    slf4j-api    1.7.30
            org.apache.directory.server    apacheds-i18n    2.0.0-M15
                org.slf4j    slf4j-api    1.7.30
            org.slf4j    slf4j-api    1.7.30
        org.apache.httpcomponents    httpclient    4.2.5
            commons-codec    commons-codec    1.9
            commons-logging    commons-logging    1.2
            org.apache.httpcomponents    httpcore    4.2.4
        org.apache.zookeeper    zookeeper    3.4.6
            io.netty    netty    3.7.0.Final
            jline    jline    0.9.94
            org.slf4j    slf4j-api    1.7.30
        org.slf4j    slf4j-api    1.7.30
    org.apache.htrace    htrace-core    3.1.0-incubating
    org.apache.zookeeper    zookeeper    3.4.6
        io.netty    netty    3.7.0.Final
        jline    jline    0.9.94
        org.slf4j    slf4j-api    1.7.30
    org.codehaus.jackson    jackson-core-asl    1.9.13
    org.codehaus.jackson    jackson-mapper-asl    1.9.13
        org.codehaus.jackson    jackson-core-asl    1.9.13
    org.mortbay.jetty    jetty-sslengine    6.1.26
        org.mortbay.jetty    jetty    6.1.26
            org.mortbay.jetty    jetty-util    6.1.26
            org.mortbay.jetty    servlet-api    2.5-20081211
    org.mortbay.jetty    jetty-util    6.1.26
    org.mortbay.jetty    jetty    6.1.26
        org.mortbay.jetty    jetty-util    6.1.26
        org.mortbay.jetty    servlet-api    2.5-20081211
    org.slf4j    slf4j-api    1.7.30
    xmlenc    xmlenc    0.52
org.apache.hadoop    hadoop-mapreduce-client-core    2.7.4
    com.google.inject.extensions    guice-servlet    3.0
        com.google.inject    guice    3.0
            aopalliance    aopalliance    1.0
            javax.inject    javax.inject    1
    com.google.protobuf    protobuf-java    2.5.0
    io.netty    netty    3.7.0.Final
    org.apache.avro    avro    1.7.7
        com.thoughtworks.paranamer    paranamer    2.3
        org.apache.commons    commons-compress    1.4.1
            org.tukaani    xz    1.0
        org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-mapper-asl    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
        org.slf4j    slf4j-api    1.7.30
        org.xerial.snappy    snappy-java    1.0.5
    org.apache.hadoop    hadoop-annotations    2.7.4
    org.apache.hadoop    hadoop-yarn-common    2.7.4
        com.google.guava    guava    16.0.1
        com.google.inject.extensions    guice-servlet    3.0
            com.google.inject    guice    3.0
                aopalliance    aopalliance    1.0
                javax.inject    javax.inject    1
        com.google.inject    guice    3.0
            aopalliance    aopalliance    1.0
            javax.inject    javax.inject    1
        com.google.protobuf    protobuf-java    2.5.0
        com.sun.jersey.contribs    jersey-guice    1.9
            com.google.inject.extensions    guice-servlet    3.0
                com.google.inject    guice    3.0
                    aopalliance    aopalliance    1.0
                    javax.inject    javax.inject    1
            com.google.inject    guice    3.0
                aopalliance    aopalliance    1.0
                javax.inject    javax.inject    1
            com.sun.jersey    jersey-server    1.9
                asm    asm    3.1
                com.sun.jersey    jersey-core    1.9
            javax.inject    javax.inject    1
        com.sun.jersey    jersey-client    1.9
            com.sun.jersey    jersey-core    1.9
        com.sun.jersey    jersey-core    1.9
        com.sun.jersey    jersey-json    1.9
            com.sun.jersey    jersey-core    1.9
            com.sun.xml.bind    jaxb-impl    2.2.3-1
                javax.xml.bind    jaxb-api    2.2.2
                    javax.activation    activation    1.1
                    javax.xml.stream    stax-api    1.0-2
            org.codehaus.jackson    jackson-core-asl    1.9.13
            org.codehaus.jackson    jackson-jaxrs    1.9.13
                org.codehaus.jackson    jackson-core-asl    1.9.13
                org.codehaus.jackson    jackson-mapper-asl    1.9.13
                    org.codehaus.jackson    jackson-core-asl    1.9.13
            org.codehaus.jackson    jackson-mapper-asl    1.9.13
                org.codehaus.jackson    jackson-core-asl    1.9.13
            org.codehaus.jackson    jackson-xc    1.9.13
                org.codehaus.jackson    jackson-core-asl    1.9.13
                org.codehaus.jackson    jackson-mapper-asl    1.9.13
                    org.codehaus.jackson    jackson-core-asl    1.9.13
            org.codehaus.jettison    jettison    1.1
        com.sun.jersey    jersey-server    1.9
            asm    asm    3.1
            com.sun.jersey    jersey-core    1.9
        commons-cli    commons-cli    1.2
        commons-codec    commons-codec    1.9
        commons-io    commons-io    2.4
        commons-lang    commons-lang    2.6
        commons-logging    commons-logging    1.2
        javax.servlet    servlet-api    2.5
        javax.xml.bind    jaxb-api    2.2.2
            javax.activation    activation    1.1
            javax.xml.stream    stax-api    1.0-2
        org.apache.commons    commons-compress    1.4.1
            org.tukaani    xz    1.0
        org.apache.hadoop    hadoop-annotations    2.7.4
        org.apache.hadoop    hadoop-yarn-api    2.7.4
            com.google.guava    guava    16.0.1
            com.google.protobuf    protobuf-java    2.5.0
            commons-lang    commons-lang    2.6
            commons-logging    commons-logging    1.2
            org.apache.hadoop    hadoop-annotations    2.7.4
        org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-jaxrs    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
            org.codehaus.jackson    jackson-mapper-asl    1.9.13
                org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-mapper-asl    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-xc    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
            org.codehaus.jackson    jackson-mapper-asl    1.9.13
                org.codehaus.jackson    jackson-core-asl    1.9.13
        org.mortbay.jetty    jetty-util    6.1.26
        org.slf4j    slf4j-api    1.7.30
    org.slf4j    slf4j-api    1.7.30
org.apache.hbase    hbase-common    1.4.9
    com.github.stephenc.findbugs    findbugs-annotations    1.3.9-1
    com.google.guava    guava    16.0.1
    com.google.protobuf    protobuf-java    2.5.0
    commons-codec    commons-codec    1.9
    commons-collections    commons-collections    3.2.2
    commons-io    commons-io    2.4
    commons-lang    commons-lang    2.6
    commons-logging    commons-logging    1.2
    junit    junit    4.13
        org.hamcrest    hamcrest-core    1.3
    org.apache.avro    avro    1.7.7
        com.thoughtworks.paranamer    paranamer    2.3
        org.apache.commons    commons-compress    1.4.1
            org.tukaani    xz    1.0
        org.codehaus.jackson    jackson-core-asl    1.9.13
        org.codehaus.jackson    jackson-mapper-asl    1.9.13
            org.codehaus.jackson    jackson-core-asl    1.9.13
        org.slf4j    slf4j-api    1.7.30
        org.xerial.snappy    snappy-java    1.0.5
    org.apache.hbase    hbase-annotations    1.4.9
        com.github.stephenc.findbugs    findbugs-annotations    1.3.9-1
        junit    junit    4.13
            org.hamcrest    hamcrest-core    1.3
    org.apache.hbase    hbase-protocol    1.4.9
        com.github.stephenc.findbugs    findbugs-annotations    1.3.9-1
        com.google.protobuf    protobuf-java    2.5.0
        commons-logging    commons-logging    1.2
        junit    junit    4.13
            org.hamcrest    hamcrest-core    1.3
        org.apache.hbase    hbase-annotations    1.4.9
            com.github.stephenc.findbugs    findbugs-annotations    1.3.9-1
            junit    junit    4.13
                org.hamcrest    hamcrest-core    1.3
    org.apache.htrace    htrace-core    3.1.0-incubating
    org.mortbay.jetty    jetty-util    6.1.26
org.apache.hbase    hbase-shaded-client    1.4.9
    com.github.stephenc.findbugs    findbugs-annotations    1.3.9-1
    commons-logging    commons-logging    1.2
    junit    junit    4.13
        org.hamcrest    hamcrest-core    1.3
    org.apache.htrace    htrace-core    3.1.0-incubating
    org.slf4j    slf4j-api    1.7.30
org.scala-lang    scala-library    2.12.10

Converters

Converters map the domain object to a list of HBase mutations (Append, Delete, Increment, Put).

Put

Scala
implicit def toBytes(string: String): Array[Byte] = Bytes.toBytes(string)
case class Person(id: Int, name: String)

val hBaseConverter: Person => immutable.Seq[Mutation] = { person =>
  val put = new Put(s"id_${person.id}")
  put.addColumn("info", "name", person.name)
  List(put)
}
Java
Function<Person, List<Mutation>> hBaseConverter =
    person -> {
      try {
        Put put = new Put(String.format("id_%d", person.id).getBytes("UTF-8"));
        put.addColumn(
            "info".getBytes("UTF-8"), "name".getBytes("UTF-8"), person.name.getBytes("UTF-8"));

        return Collections.singletonList(put);
      } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
        return Collections.emptyList();
      }
    };

Append

Scala
val appendHBaseConverter: Person => immutable.Seq[Mutation] = { person =>
  // Append to a cell
  val append = new Append(s"id_${person.id}")
  append.add("info", "aliases", person.name)
  List(append)
}
Java
Function<Person, List<Mutation>> appendHBaseConverter =
    person -> {
      try {
        Append append = new Append(String.format("id_%d", person.id).getBytes("UTF-8"));
        append.add(
            "info".getBytes("UTF-8"), "aliases".getBytes("UTF-8"), person.name.getBytes("UTF-8"));

        return Collections.singletonList(append);
      } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
        return Collections.emptyList();
      }
    };

Delete

Scala
val deleteHBaseConverter: Person => immutable.Seq[Mutation] = { person =>
  // Delete the specified row
  val delete = new Delete(s"id_${person.id}")
  List(delete)
}
Java
Function<Person, List<Mutation>> deleteHBaseConverter =
    person -> {
      try {
        Delete delete = new Delete(String.format("id_%d", person.id).getBytes("UTF-8"));

        return Collections.singletonList(delete);
      } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
        return Collections.emptyList();
      }
    };

Increment

Scala
val incrementHBaseConverter: Person => immutable.Seq[Mutation] = { person =>
  // Increment a cell value
  val increment = new Increment(s"id_${person.id}")
  increment.addColumn("info", "numberOfChanges", 1)
  List(increment)
}
Java
Function<Person, List<Mutation>> incrementHBaseConverter =
    person -> {
      try {
        Increment increment = new Increment(String.format("id_%d", person.id).getBytes("UTF-8"));
        increment.addColumn("info".getBytes("UTF-8"), "numberOfChanges".getBytes("UTF-8"), 1);

        return Collections.singletonList(increment);
      } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
        return Collections.emptyList();
      }
    };

Complex and noop mutations

To ignore an object return an empty List - this will have no effect on HBase. You can also combine mutations to perform complex business logic:

Scala
val mutationsHBaseConverter: Person => immutable.Seq[Mutation] = { person =>
  if (person.id != 0) {
    if (person.name.isEmpty) {
      // Delete the specified row
      val delete = new Delete(s"id_${person.id}")
      List(delete)
    } else {
      // Insert or update a row
      val put = new Put(s"id_${person.id}")
      put.addColumn("info", "name", person.name)

      val increment = new Increment(s"id_${person.id}")
      increment.addColumn("info", "numberOfChanges", 1)

      List(put, increment)
    }
  } else {
    List.empty
  }
}
Java
Function<Person, List<Mutation>> complexHBaseConverter =
    person -> {
      try {
        byte[] id = String.format("id_%d", person.id).getBytes("UTF-8");
        byte[] infoFamily = "info".getBytes("UTF-8");

        if (person.id != 0 && person.name.isEmpty()) {
          Delete delete = new Delete(id);
          return Collections.singletonList(delete);
        } else if (person.id != 0) {
          Put put = new Put(id);
          put.addColumn(infoFamily, "name".getBytes("UTF-8"), person.name.getBytes("UTF-8"));

          Increment increment = new Increment(id);
          increment.addColumn(infoFamily, "numberOfChanges".getBytes("UTF-8"), 1);

          return Arrays.asList(put, increment);
        } else {
          return Collections.emptyList();
        }
      } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
        return Collections.emptyList();
      }
    };

If you return a list of mutations they will be applied in the same order. The list of mutations are not applied in an transaction, each mutation is independent.

Settings

HBase combinators require HTableSettings. If the table referenced in the settings does not exist, it will be created on demand.

Scala
val tableSettings =
  HTableSettings(HBaseConfiguration.create(), TableName.valueOf("person"), immutable.Seq("info"), hBaseConverter)
Java
HTableSettings<Person> tableSettings =
    HTableSettings.create(
        HBaseConfiguration.create(),
        TableName.valueOf("person1"),
        Collections.singletonList("info"),
        hBaseConverter);

Source

Scala
val scan = new Scan(new Get(Bytes.toBytes("id_100")))

val f = HTableStage
  .source(scan, tableSettings)
  .runWith(Sink.seq)
Java
Scan scan = new Scan(new Get("id_300".getBytes("UTF-8")));

CompletionStage<List<Result>> f =
    HTableStage.source(scan, tableSettings).runWith(Sink.seq(), materializer);

Flow

Scala
val flow = HTableStage.flow[Person](tableSettings)

val f = Source(11 to 20).map(i => Person(i, s"zozo_$i")).via(flow).runWith(Sink.fold(0)((a, d) => a + d.id))
Java
Flow<Person, Person, NotUsed> flow = HTableStage.flow(tableSettings);
Pair<NotUsed, CompletionStage<List<Person>>> run =
    Source.from(Arrays.asList(200, 201, 202, 203, 204))
        .map((i) -> new Person(i, String.format("name_%d", i)))
        .via(flow)
        .toMat(Sink.seq(), Keep.both())
        .run(materializer);

Sink

Scala
val sink = HTableStage.sink[Person](tableSettings)

val f = Source(1 to 10).map(i => Person(i, s"zozo_$i")).runWith(sink)
Java
final Sink<Person, CompletionStage<Done>> sink = HTableStage.sink(tableSettings);
CompletionStage<Done> o =
    Source.from(Arrays.asList(100, 101, 102, 103, 104))
        .map((i) -> new Person(i, String.format("name %d", i)))
        .runWith(sink, materializer);

HBase administration commands

To manage HBase database, startup HBase shell ($HBASE_HOME/bin/shell), and run following commands:

list // list tables
scan "person" // select * from person
disable "person" // Disable table "person", before drop
drop "person"
Found an error in this documentation? The source code for this page can be found here. Please feel free to edit and contribute a pull request.