Distributed Publish Subscribe in Cluster

Publish

This is the true pub/sub mode. A typical usage of this mode is a chat room in an instant messaging application.

Actors are registered to a named topic. This enables many subscribers on each node. The message will be delivered to all subscribers of the topic.

For efficiency the message is sent over the wire only once per node (that has a matching topic), and then delivered to all subscribers of the local topic representation.

You register actors to the local mediator with DistributedPubSubMediator.Subscribe. Successful Subscribe and Unsubscribe is acknowledged with DistributedPubSubMediator.SubscribeAck and DistributedPubSubMediator.UnsubscribeAck replies. The acknowledgment means that the subscription is registered, but it can still take some time until it is replicated to other nodes.

You publish messages by sending DistributedPubSubMediator.Publish message to the local mediator.

Actors are automatically removed from the registry when they are terminated, or you can explicitly remove entries with DistributedPubSubMediator.Unsubscribe.

An example of a subscriber actor:

public class Subscriber extends UntypedActor {
  LoggingAdapter log = Logging.getLogger(getContext().system(), this);

  public Subscriber() {
    ActorRef mediator =
      DistributedPubSub.get(getContext().system()).mediator();
    // subscribe to the topic named "content"
    mediator.tell(new DistributedPubSubMediator.Subscribe("content", getSelf()),
      getSelf());
  }

  public void onReceive(Object msg) {
    if (msg instanceof String)
      log.info("Got: {}", msg);
    else if (msg instanceof DistributedPubSubMediator.SubscribeAck)
      log.info("subscribing");
    else
      unhandled(msg);
  }
}

Subscriber actors can be started on several nodes in the cluster, and all will receive messages published to the "content" topic.

system.actorOf(Props.create(Subscriber.class), "subscriber1");
//another node
system.actorOf(Props.create(Subscriber.class), "subscriber2");
system.actorOf(Props.create(Subscriber.class), "subscriber3");

A simple actor that publishes to this "content" topic:

public class Publisher extends UntypedActor {

  // activate the extension
  ActorRef mediator =
    DistributedPubSub.get(getContext().system()).mediator();

  public void onReceive(Object msg) {
    if (msg instanceof String) {
      String in = (String) msg;
      String out = in.toUpperCase();
      mediator.tell(new DistributedPubSubMediator.Publish("content", out),
        getSelf());
    } else {
      unhandled(msg);
    }
  }
}

It can publish messages to the topic from anywhere in the cluster:

//somewhere else
ActorRef publisher = system.actorOf(Props.create(Publisher.class), "publisher");
// after a while the subscriptions are replicated
publisher.tell("hello", null);

Send

This is a point-to-point mode where each message is delivered to one destination, but you still does not have to know where the destination is located. A typical usage of this mode is private chat to one other user in an instant messaging application. It can also be used for distributing tasks to registered workers, like a cluster aware router where the routees dynamically can register themselves.

The message will be delivered to one recipient with a matching path, if any such exists in the registry. If several entries match the path because it has been registered on several nodes the message will be sent via the supplied RoutingLogic (default random) to one destination. The sender() of the message can specify that local affinity is preferred, i.e. the message is sent to an actor in the same local actor system as the used mediator actor, if any such exists, otherwise route to any other matching entry.

You register actors to the local mediator with DistributedPubSubMediator.Put. The ActorRef in Put must belong to the same local actor system as the mediator. The path without address information is the key to which you send messages. On each node there can only be one actor for a given path, since the path is unique within one local actor system.

You send messages by sending DistributedPubSubMediator.Send message to the local mediator with the path (without address information) of the destination actors.

Actors are automatically removed from the registry when they are terminated, or you can explicitly remove entries with DistributedPubSubMediator.Remove.

An example of a destination actor:

public class Destination extends UntypedActor {
  LoggingAdapter log = Logging.getLogger(getContext().system(), this);

  public Destination() {
    ActorRef mediator =
      DistributedPubSub.get(getContext().system()).mediator();
    // register to the path
    mediator.tell(new DistributedPubSubMediator.Put(getSelf()), getSelf());
  }

  public void onReceive(Object msg) {
    if (msg instanceof String)
      log.info("Got: {}", msg);
    else if (msg instanceof DistributedPubSubMediator.SubscribeAck)
      log.info("subscribing");
    else
      unhandled(msg);
  }
}

Subscriber actors can be started on several nodes in the cluster, and all will receive messages published to the "content" topic.

system.actorOf(Props.create(Destination.class), "destination");
//another node
system.actorOf(Props.create(Destination.class), "destination");

A simple actor that publishes to this "content" topic:

public class Sender extends UntypedActor {

  // activate the extension
  ActorRef mediator =
    DistributedPubSub.get(getContext().system()).mediator();

  public void onReceive(Object msg) {
    if (msg instanceof String) {
      String in = (String) msg;
      String out = in.toUpperCase();
      boolean localAffinity = true;
      mediator.tell(new DistributedPubSubMediator.Send("/user/destination", out,
          localAffinity), getSelf());
    } else {
      unhandled(msg);
    }
  }
}

It can publish messages to the topic from anywhere in the cluster:

//somewhere else
ActorRef sender = system.actorOf(Props.create(Publisher.class), "sender");
// after a while the destinations are replicated
sender.tell("hello", null);

It is also possible to broadcast messages to the actors that have been registered with Put. Send DistributedPubSubMediator.SendToAll message to the local mediator and the wrapped message will then be delivered to all recipients with a matching path. Actors with the same path, without address information, can be registered on different nodes. On each node there can only be one such actor, since the path is unique within one local actor system.

Typical usage of this mode is to broadcast messages to all replicas with the same path, e.g. 3 actors on different nodes that all perform the same actions, for redundancy. You can also optionally specify a property (allButSelf) deciding if the message should be sent to a matching path on the self node or not.

DistributedPubSub Extension

In the example above the mediator is started and accessed with the akka.cluster.pubsub.DistributedPubSub extension. That is convenient and perfectly fine in most cases, but it can be good to know that it is possible to start the mediator actor as an ordinary actor and you can have several different mediators at the same time to be able to divide a large number of actors/topics to different mediators. For example you might want to use different cluster roles for different mediators.

The DistributedPubSub extension can be configured with the following properties:

# Settings for the DistributedPubSub extension
akka.cluster.pub-sub {
  # Actor name of the mediator actor, /system/distributedPubSubMediator
  name = distributedPubSubMediator

  # Start the mediator on members tagged with this role.
  # All members are used if undefined or empty.
  role = ""

  # The routing logic to use for 'Send'
  # Possible values: random, round-robin, broadcast
  routing-logic = random

  # How often the DistributedPubSubMediator should send out gossip information
  gossip-interval = 1s

  # Removed entries are pruned after this duration
  removed-time-to-live = 120s

  # Maximum number of elements to transfer in one message when synchronizing the registries.
  # Next chunk will be transferred in next round of gossip.
  max-delta-elements = 3000
  
  # The id of the dispatcher to use for DistributedPubSubMediator actors. 
  # If not specified default dispatcher is used.
  # If specified you need to define the settings of the actual dispatcher.
  use-dispatcher = ""
}

It is recommended to load the extension when the actor system is started by defining it in akka.extensions configuration property. Otherwise it will be activated when first used and then it takes a while for it to be populated.

akka.extensions = ["akka.cluster.pubsub.DistributedPubSub"]

Delivery Guarantee

As in Message Delivery Reliability of Akka, message delivery guarantee in distributed pub sub modes is at-most-once delivery. In other words, messages can be lost over the wire.

If you are looking for at-least-once delivery guarantee, we recommend Kafka Akka Streams integration.

Dependencies

To use Distributed Publish Subscribe you must add the following dependency in your project.

sbt:

"com.typesafe.akka" %% "akka-cluster-tools" % "2.4.20"

maven:

<dependency>
  <groupId>com.typesafe.akka</groupId>
  <artifactId>akka-cluster-tools_2.11</artifactId>
  <version>2.4.20</version>
</dependency>