Apache Geode

Apache Geode is a distributed datagrid (ex Gemfire).

This connector provides flow and a sink to put element in and source to retrieve element from geode.

Basically it can store data as key, value. Key and value must be serialized, more on this later.

Project Info: Alpakka Geode
Artifact
com.lightbend.akka
akka-stream-alpakka-geode
1.0-M2
JDK versions
OpenJDK 8
Scala versions2.12.7, 2.11.12
JPMS module nameakka.stream.alpakka.geode
License
Readiness level
Community-driven
Since 0.10, 2017-06-30
Home pagehttps://doc.akka.io/docs/alpakka/current/
API documentation
Forums
Release notesIn the documentation
IssuesGithub issues
Sourceshttps://github.com/akka/alpakka

Artifacts

sbt
libraryDependencies += "com.lightbend.akka" %% "akka-stream-alpakka-geode" % "1.0-M2"
Maven
<dependency>
  <groupId>com.lightbend.akka</groupId>
  <artifactId>akka-stream-alpakka-geode_2.12</artifactId>
  <version>1.0-M2</version>
</dependency>
Gradle
dependencies {
  compile group: 'com.lightbend.akka', name: 'akka-stream-alpakka-geode_2.12', version: '1.0-M2'
}

The table below shows direct dependencies of this module and the second tab shows all libraries it depends on transitively.

Direct dependencies
OrganizationArtifactVersionLicense
com.chuusaishapeless_2.122.3.3Apache 2
com.typesafe.akkaakka-stream_2.122.5.19Apache License, Version 2.0
org.apache.geodegeode-core1.8.0The Apache Software License, Version 2.0
org.apache.geodegeode-cq1.8.0The Apache Software License, Version 2.0
org.scala-langscala-library2.12.7BSD 3-Clause
Dependency tree
com.chuusai    shapeless_2.12    2.3.3    Apache 2
    org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    org.typelevel    macro-compat_2.12    1.1.1    Apache 2
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
com.typesafe.akka    akka-stream_2.12    2.5.19    Apache License, Version 2.0
    com.typesafe.akka    akka-actor_2.12    2.5.19    Apache License, Version 2.0
        com.typesafe    config    1.3.3    Apache License, Version 2.0
        org.scala-lang.modules    scala-java8-compat_2.12    0.8.0    BSD 3-clause
            org.scala-lang    scala-library    2.12.7    BSD 3-Clause
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    com.typesafe.akka    akka-protobuf_2.12    2.5.19    Apache License, Version 2.0
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    com.typesafe    ssl-config-core_2.12    0.3.6    Apache-2.0
        com.typesafe    config    1.3.3    Apache License, Version 2.0
        org.scala-lang.modules    scala-parser-combinators_2.12    1.1.1    BSD 3-clause
            org.scala-lang    scala-library    2.12.7    BSD 3-Clause
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    org.reactivestreams    reactive-streams    1.0.2    CC0
    org.scala-lang    scala-library    2.12.7    BSD 3-Clause
org.apache.geode    geode-core    1.8.0    The Apache Software License, Version 2.0
    antlr    antlr    2.7.7    BSD License
    com.fasterxml.jackson.core    jackson-annotations    2.9.6    The Apache Software License, Version 2.0
    com.fasterxml.jackson.core    jackson-databind    2.9.6    The Apache Software License, Version 2.0
        com.fasterxml.jackson.core    jackson-annotations    2.9.6    The Apache Software License, Version 2.0
        com.fasterxml.jackson.core    jackson-core    2.9.6    The Apache Software License, Version 2.0
    com.github.stephenc.findbugs    findbugs-annotations    1.3.9-1    Apache License, Version 2.0
    com.healthmarketscience.rmiio    rmiio    2.1.2    Apache License, Version 2.0
        commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
    com.sun.xml.bind    jaxb-core    2.2.11    CDDL+GPL License
    com.sun.xml.bind    jaxb-impl    2.2.11    CDDL+GPL License
    commons-digester    commons-digester    2.1    The Apache Software License, Version 2.0
        commons-beanutils    commons-beanutils    1.9.2    The Apache Software License, Version 2.0
            commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
            commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
        commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
    commons-io    commons-io    2.6    Apache License, Version 2.0
    commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
    commons-validator    commons-validator    1.6    Apache License, Version 2.0
        commons-beanutils    commons-beanutils    1.9.2    The Apache Software License, Version 2.0
            commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
            commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
        commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
        commons-digester    commons-digester    2.1    The Apache Software License, Version 2.0
            commons-beanutils    commons-beanutils    1.9.2    The Apache Software License, Version 2.0
                commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
                commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
            commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
        commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
    io.github.classgraph    classgraph    4.0.6    The MIT License (MIT)
    it.unimi.dsi    fastutil    8.2.1    Apache License, Version 2.0
    javax.activation    activation    1.1.1    COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0
    javax.resource    javax.resource-api    1.7    CDDL + GPLv2 with classpath exception
        javax.transaction    javax.transaction-api    1.2    CDDL + GPLv2 with classpath exception
    javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
    net.java.dev.jna    jna    4.1.0    LGPL, version 2.1
    net.sf.jopt-simple    jopt-simple    5.0.4    The MIT License
    org.apache.geode    geode-common    1.8.0    The Apache Software License, Version 2.0
    org.apache.geode    geode-json    1.8.0    The Apache Software License, Version 2.0
    org.apache.logging.log4j    log4j-api    2.11.0
    org.apache.logging.log4j    log4j-core    2.11.0
    org.apache.shiro    shiro-core    1.4.0    Apache License, Version 2.0
        org.apache.shiro    shiro-cache    1.4.0    Apache License, Version 2.0
            org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                org.slf4j    slf4j-api    1.7.25    MIT License
        org.apache.shiro    shiro-config-core    1.4.0    Apache License, Version 2.0
            org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                org.slf4j    slf4j-api    1.7.25    MIT License
        org.apache.shiro    shiro-crypto-cipher    1.4.0    Apache License, Version 2.0
            org.apache.shiro    shiro-crypto-core    1.4.0    Apache License, Version 2.0
                org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                org.slf4j    slf4j-api    1.7.25    MIT License
        org.apache.shiro    shiro-crypto-hash    1.4.0    Apache License, Version 2.0
            org.apache.shiro    shiro-crypto-core    1.4.0    Apache License, Version 2.0
                org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                org.slf4j    slf4j-api    1.7.25    MIT License
        org.apache.shiro    shiro-event    1.4.0    Apache License, Version 2.0
            org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                org.slf4j    slf4j-api    1.7.25    MIT License
        org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
            org.slf4j    slf4j-api    1.7.25    MIT License
    org.jgroups    jgroups    3.6.14.Final    Apache License 2.0
    org.slf4j    slf4j-api    1.7.25    MIT License
org.apache.geode    geode-cq    1.8.0    The Apache Software License, Version 2.0
    org.apache.geode    geode-core    1.8.0    The Apache Software License, Version 2.0
        antlr    antlr    2.7.7    BSD License
        com.fasterxml.jackson.core    jackson-annotations    2.9.6    The Apache Software License, Version 2.0
        com.fasterxml.jackson.core    jackson-databind    2.9.6    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-annotations    2.9.6    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-core    2.9.6    The Apache Software License, Version 2.0
        com.github.stephenc.findbugs    findbugs-annotations    1.3.9-1    Apache License, Version 2.0
        com.healthmarketscience.rmiio    rmiio    2.1.2    Apache License, Version 2.0
            commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
        com.sun.xml.bind    jaxb-core    2.2.11    CDDL+GPL License
        com.sun.xml.bind    jaxb-impl    2.2.11    CDDL+GPL License
        commons-digester    commons-digester    2.1    The Apache Software License, Version 2.0
            commons-beanutils    commons-beanutils    1.9.2    The Apache Software License, Version 2.0
                commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
                commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
            commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
        commons-io    commons-io    2.6    Apache License, Version 2.0
        commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
        commons-validator    commons-validator    1.6    Apache License, Version 2.0
            commons-beanutils    commons-beanutils    1.9.2    The Apache Software License, Version 2.0
                commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
                commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
            commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
            commons-digester    commons-digester    2.1    The Apache Software License, Version 2.0
                commons-beanutils    commons-beanutils    1.9.2    The Apache Software License, Version 2.0
                    commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
                    commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
                commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
            commons-logging    commons-logging    1.2    The Apache Software License, Version 2.0
        io.github.classgraph    classgraph    4.0.6    The MIT License (MIT)
        it.unimi.dsi    fastutil    8.2.1    Apache License, Version 2.0
        javax.activation    activation    1.1.1    COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0
        javax.resource    javax.resource-api    1.7    CDDL + GPLv2 with classpath exception
            javax.transaction    javax.transaction-api    1.2    CDDL + GPLv2 with classpath exception
        javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
        net.java.dev.jna    jna    4.1.0    LGPL, version 2.1
        net.sf.jopt-simple    jopt-simple    5.0.4    The MIT License
        org.apache.geode    geode-common    1.8.0    The Apache Software License, Version 2.0
        org.apache.geode    geode-json    1.8.0    The Apache Software License, Version 2.0
        org.apache.logging.log4j    log4j-api    2.11.0
        org.apache.logging.log4j    log4j-core    2.11.0
        org.apache.shiro    shiro-core    1.4.0    Apache License, Version 2.0
            org.apache.shiro    shiro-cache    1.4.0    Apache License, Version 2.0
                org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.shiro    shiro-config-core    1.4.0    Apache License, Version 2.0
                org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.shiro    shiro-crypto-cipher    1.4.0    Apache License, Version 2.0
                org.apache.shiro    shiro-crypto-core    1.4.0    Apache License, Version 2.0
                    org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                        org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.shiro    shiro-crypto-hash    1.4.0    Apache License, Version 2.0
                org.apache.shiro    shiro-crypto-core    1.4.0    Apache License, Version 2.0
                    org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                        org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.shiro    shiro-event    1.4.0    Apache License, Version 2.0
                org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.shiro    shiro-lang    1.4.0    Apache License, Version 2.0
                org.slf4j    slf4j-api    1.7.25    MIT License
        org.jgroups    jgroups    3.6.14.Final    Apache License 2.0
        org.slf4j    slf4j-api    1.7.25    MIT License
    org.apache.logging.log4j    log4j-api    2.11.0
org.scala-lang    scala-library    2.12.7    BSD 3-Clause

Connection

First of all you need to connect to the geode cache. In a client application, connection is handle by a ClientCache. A single ClientCache per application is enough. ClientCache also holds a single PDXSerializer.

scala
val reactiveGeode = new ReactiveGeode(geodeSettings)
java
GeodeSettings settings =
    GeodeSettings.create(geodeDockerHostname, 10334)
        .withConfiguration(c -> c.setPoolIdleTimeout(10));
return new ReactiveGeode(settings);

Apache Geode supports continuous queries. Continuous query relies on server event, thus reactive geode needs to listen to those event. This behaviour, as it consumes more resources is isolated in a scala trait and/or an specialized java class.

scala
val reactiveGeode = new ReactiveGeode(geodeSettings) with PoolSubscription
java
return new ReactiveGeodeWithPoolSubscription(settings);

Region

Define a region setting to describe how to access region and the key extraction function.

scala
val personsRegionSettings: RegionSettings[Int, Person] = RegionSettings("persons", (p: Person) => p.id)
val animalsRegionSettings: RegionSettings[Int, Animal] = RegionSettings("animals", (a: Animal) => a.id)
val complexesRegionSettings: RegionSettings[UUID, Complex] = RegionSettings("complexes", (a: Complex) => a.id)
java
protected RegionSettings<Integer, Person> personRegionSettings =
    RegionSettings.create("persons", Person::getId);
protected RegionSettings<Integer, Animal> animalRegionSettings =
    RegionSettings.create("animals", Animal::getId);

Serialization

Object must be serialized to flow in a geode region.

  • opaque format (eq json/xml)
  • java serialisation
  • pdx geode format

PDX format is the only one supported.

PDXEncoder support many options, see gemfire_pdx_serialization.html

PdxSerializer must be provided to geode when reading or writing to a region.

scala
object PersonPdxSerializer extends AkkaPdxSerializer[Person] {
  override def clazz: Class[Person] = classOf[Person]

  override def toData(o: scala.Any, out: PdxWriter): Boolean =
    if (o.isInstanceOf[Person]) {
      val p = o.asInstanceOf[Person]
      out.writeInt("id", p.id)
      out.writeString("name", p.name)
      out.writeDate("birthDate", p.birthDate)
      true
    } else
      false

  override def fromData(clazz: Class[_], in: PdxReader): AnyRef = {
    val id: Int = in.readInt("id")
    val name: String = in.readString("name")
    val birthDate: Date = in.readDate("birthDate")
    Person(id, name, birthDate)
  }
}
java
public class PersonPdxSerializer implements AkkaPdxSerializer<Person> {

  @Override
  public Class<Person> clazz() {
    return Person.class;
  }

  @Override
  public boolean toData(Object o, PdxWriter out) {
    if (o instanceof Person) {
      Person p = (Person) o;
      out.writeInt("id", p.getId());
      out.writeString("name", p.getName());
      out.writeDate("birthDate", p.getBirthDate());
      return true;
    }
    return false;
  }

  @Override
  public Object fromData(Class<?> clazz, PdxReader in) {
    int id = in.readInt("id");
    String name = in.readString("name");
    Date birthDate = in.readDate("birthDate");
    return new Person(id, name, birthDate);
  }
}

This project provides a generic solution for scala user based on shapeless, then case classe serializer if not provided will be generated compile time. Java user will need to write by hand their custom serializer.

Runtime reflection is also an option see auto_serialization.html.

Flow usage

This sample stores (case) classes in Geode.

scala
val flow: Flow[Person, Person, NotUsed] = reactiveGeode.flow(personsRegionSettings)

val fut = source.via(flow).runWith(Sink.ignore)
java
Flow<Person, Person, NotUsed> flow =
    reactiveGeode.flow(personRegionSettings, new PersonPdxSerializer());

CompletionStage<List<Person>> run =
    source.via(flow).toMat(Sink.seq(), Keep.right()).run(materializer);

Sink usage

scala
val sink = reactiveGeode.sink(animalsRegionSettings)

val fut = source.runWith(sink)
val sink = reactiveGeode.sink(complexesRegionSettings)

val fut = source.runWith(sink)
java
Sink<Animal, CompletionStage<Done>> sink =
    reactiveGeode.sink(animalRegionSettings, new AnimalPdxSerializer());

RunnableGraph<CompletionStage<Done>> runnableGraph = source.toMat(sink, Keep.right());

ource usage

Simple query

Apache Geode support simple queries.

scala
val source =
  reactiveGeode
    .query[Person](s"select * from /persons order by id")
    .runWith(Sink.foreach(e => log.debug(s"$e")))
java
CompletionStage<Done> personsDone =
    reactiveGeode
        .query("select * from /persons", new PersonPdxSerializer())
        .runForeach(
            p -> {
              LOGGER.debug(p.toString());
            },
            materializer);

Continuous query

scala
val source =
  reactiveGeode
    .continuousQuery[Person]('test, s"select * from /persons")
    .runWith(Sink.fold(0) { (c, p) =>
      log.debug(s"$p $c")
      if (c == 19) {
        reactiveGeode.closeContinuousQuery('test).foreach { _ =>
          log.debug("test cQuery is closed")
        }

      }
      c + 1
    })
java
CompletionStage<Done> fut =
    reactiveGeode
        .continuousQuery("test", "select * from /persons", new PersonPdxSerializer())
        .runForeach(
            p -> {
              LOGGER.debug(p.toString());
              if (p.getId() == 120) {
                reactiveGeode.closeContinuousQuery("test");
              }
            },
            materializer);

Geode basic commands

Assuming Apache geode is installed:

gfsh

From the geode shell:

start locator --name=locator
configure pdx --read-serialized=true
start server --name=server

create region --name=animals --type=PARTITION_REDUNDANT --redundant-copies=2
create region --name=persons --type=PARTITION_REDUNDANT --redundant-copies=2

Run the example code

Integration test are run against localhost geode, but IT_GEODE_HOSTNAME environment variable can change this:

Test code requires Geode running in the background. You can start it quickly using docker:

docker-compose up geode

Scala
sbt
> geode/testOnly *Spec
Java
sbt
> geode/testOnly *Test
Found an error in this documentation? The source code for this page can be found here. Please feel free to edit and contribute a pull request.