Distributed Publish Subscribe in Cluster

How do I send a message to an actor without knowing which node it is running on?

How do I send messages to all actors in the cluster that have registered interest in a named topic?

This pattern provides a mediator actor, akka.contrib.pattern.DistributedPubSubMediator, that manages a registry of actor references and replicates the entries to peer actors among all cluster nodes or a group of nodes tagged with a specific role.

The DistributedPubSubMediator is supposed to be started on all nodes, or all nodes with specified role, in the cluster. The mediator can be started with the DistributedPubSubExtension or as an ordinary actor.

Changes are only performed in the own part of the registry and those changes are versioned. Deltas are disseminated in a scalable way to other nodes with a gossip protocol. The registry is eventually consistent, i.e. changes are not immediately visible at other nodes, but typically they will be fully replicated to all other nodes after a few seconds.

You can send messages via the mediator on any node to registered actors on any other node. There is four modes of message delivery.

1. DistributedPubSubMediator.Send

The message will be delivered to one recipient with a matching path, if any such exists in the registry. If several entries match the path the message will be sent via the supplied RoutingLogic (default random) to one destination. The sender() of the message can specify that local affinity is preferred, i.e. the message is sent to an actor in the same local actor system as the used mediator actor, if any such exists, otherwise route to any other matching entry. A typical usage of this mode is private chat to one other user in an instant messaging application. It can also be used for distributing tasks to registered workers, like a cluster aware router where the routees dynamically can register themselves.

2. DistributedPubSubMediator.SendToAll

The message will be delivered to all recipients with a matching path. Actors with the same path, without address information, can be registered on different nodes. On each node there can only be one such actor, since the path is unique within one local actor system. Typical usage of this mode is to broadcast messages to all replicas with the same path, e.g. 3 actors on different nodes that all perform the same actions, for redundancy. You can also optionally specify a property (allButSelf) deciding if the message should be sent to a matching path on the self node or not.

3. DistributedPubSubMediator.Publish

Actors may be registered to a named topic instead of path. This enables many subscribers on each node. The message will be delivered to all subscribers of the topic. For efficiency the message is sent over the wire only once per node (that has a matching topic), and then delivered to all subscribers of the local topic representation. This is the true pub/sub mode. A typical usage of this mode is a chat room in an instant messaging application.

4. DistributedPubSubMediator.Publish with sendOneMessageToEachGroup

Actors may be subscribed to a named topic with an optional property (group). If subscribing with a group name, each message published to a topic with the (sendOneMessageToEachGroup) flag is delivered via the supplied RoutingLogic (default random) to one actor within each subscribing group. If all the subscribed actors have the same group name, then this works just like Send and all messages are delivered to one subscriber. If all the subscribed actors have different group names, then this works like normal Publish and all messages are broadcast to all subscribers.

You register actors to the local mediator with DistributedPubSubMediator.Put or DistributedPubSubMediator.Subscribe. Put is used together with Send and SendToAll message delivery modes. The ActorRef in Put must belong to the same local actor system as the mediator. Subscribe is used together with Publish. Actors are automatically removed from the registry when they are terminated, or you can explicitly remove entries with DistributedPubSubMediator.Remove or DistributedPubSubMediator.Unsubscribe.

Successful Subscribe and Unsubscribe is acknowledged with DistributedPubSubMediator.SubscribeAck and DistributedPubSubMediator.UnsubscribeAck replies.

public class Subscriber extends UntypedActor {
  LoggingAdapter log = Logging.getLogger(getContext().system(), this);

  public Subscriber() {
    ActorRef mediator = 
      DistributedPubSubExtension.get(getContext().system()).mediator();
    // subscribe to the topic named "content"
    mediator.tell(new DistributedPubSubMediator.Subscribe("content", getSelf()), 
      getSelf());
  }

  public void onReceive(Object msg) {
    if (msg instanceof String)
      log.info("Got: {}", msg);
    else if (msg instanceof DistributedPubSubMediator.SubscribeAck)
      log.info("subscribing");
    else
      unhandled(msg);
  }
}

system.actorOf(Props.create(Subscriber.class), "subscriber1");
//another node
system.actorOf(Props.create(Subscriber.class), "subscriber2");
system.actorOf(Props.create(Subscriber.class), "subscriber3");

public class Publisher extends UntypedActor {

  // activate the extension
  ActorRef mediator = 
    DistributedPubSubExtension.get(getContext().system()).mediator();

  public void onReceive(Object msg) {
    if (msg instanceof String) {
      String in = (String) msg;
      String out = in.toUpperCase();
      mediator.tell(new DistributedPubSubMediator.Publish("content", out), 
        getSelf());
    } else {
      unhandled(msg);
    }
  }
}

//somewhere else
ActorRef publisher = system.actorOf(Props.create(Publisher.class), "publisher");
// after a while the subscriptions are replicated
publisher.tell("hello", null);

class Subscriber extends Actor with ActorLogging {
  import DistributedPubSubMediator.{ Subscribe, SubscribeAck }
  val mediator = DistributedPubSubExtension(context.system).mediator
  // subscribe to the topic named "content"
  mediator ! Subscribe("content", self)

  def receive = {
    case SubscribeAck(Subscribe("content", None, `self`)) ⇒
      context become ready
  }

  def ready: Actor.Receive = {
    case s: String ⇒
      log.info("Got {}", s)
  }
}

runOn(first) {
  system.actorOf(Props[Subscriber], "subscriber1")
}
runOn(second) {
  system.actorOf(Props[Subscriber], "subscriber2")
  system.actorOf(Props[Subscriber], "subscriber3")
}

class Publisher extends Actor {
  import DistributedPubSubMediator.Publish
  // activate the extension
  val mediator = DistributedPubSubExtension(context.system).mediator

  def receive = {
    case in: String ⇒
      val out = in.toUpperCase
      mediator ! Publish("content", out)
  }
}

runOn(third) {
  val publisher = system.actorOf(Props[Publisher], "publisher")
  later()
  // after a while the subscriptions are replicated
  publisher ! "hello"
}

# Settings for the DistributedPubSubExtension
akka.contrib.cluster.pub-sub {
  # Actor name of the mediator actor, /user/distributedPubSubMediator
  name = distributedPubSubMediator

  # Start the mediator on members tagged with this role.
  # All members are used if undefined or empty.
  role = ""

  # The routing logic to use for 'Send'
  # Possible values: random, round-robin, broadcast
  routing-logic = random

  # How often the DistributedPubSubMediator should send out gossip information
  gossip-interval = 1s

  # Removed entries are pruned after this duration
  removed-time-to-live = 120s

  # Maximum number of elements to transfer in one message when synchronizing the registries.
  # Next chunk will be transferred in next round of gossip.
  max-delta-elements = 3000

}

akka.extensions = ["akka.contrib.pattern.DistributedPubSubExtension"]