Multi Node Testing

Multi Node Testing Concepts

When we talk about multi node testing in Akka we mean the process of running coordinated tests on multiple actor systems in different JVMs. The multi node testing kit consist of three main parts.

• The Test Conductor. that coordinates and controls the nodes under test.
• The Multi Node Spec. that is a convenience wrapper for starting the TestConductor and letting all nodes connect to it.
• The SbtMultiJvm Plugin. that starts tests in multiple JVMs possibly on multiple machines.

The Test Conductor

The basis for the multi node testing is the TestConductor. It is an Akka Extension that plugs in to the network stack and it is used to coordinate the nodes participating in the test and provides several features including:

• Node Address Lookup: Finding out the full path to another test node (No need to share configuration between test nodes)
• Node Barrier Coordination: Waiting for other nodes at named barriers.
• Network Failure Injection: Throttling traffic, dropping packets, unplugging and plugging nodes back in.

This is a schematic overview of the test conductor.

The test conductor server is responsible for coordinating barriers and sending commands to the test conductor clients that act upon them, e.g. throttling network traffic to/from another client. More information on the possible operations is availible in the akka.remote.testconductor.Conductor API documentation.

The Multi Node Spec

The Multi Node Spec consists of two parts. The MultiNodeConfig that is responsible for common configuration and enumerating and naming the nodes under test. The MultiNodeSpec that contains a number of convenience functions for making the test nodes interact with each other. More information on the possible operations is available in the akka.remote.testkit.MultiNodeSpec API documentation.

The setup of the MultiNodeSpec is configured through java system properties that you set on all JVMs that's going to run a node under test. These can easily be set on the JVM command line with -Dproperty=value.

These are the available properties:

• multinode.max-nodes

  The maximum number of nodes that a test can have.

• multinode.host

  The host name or IP for this node. Must be resolvable using InetAddress.getByName.

• multinode.port

  The port number for this node. Defaults to 0 which will use a random port.

• multinode.server-host

  The host name or IP for the server node. Must be resolvable using InetAddress.getByName.

• multinode.server-port

  The port number for the server node. Defaults to 4711.

• multinode.index

  The index of this node in the sequence of roles defined for the test. The index 0 is special and that machine will be the server. All failure injection and throttling must be done from this node.

The SbtMultiJvm Plugin

The SbtMultiJvm Plugin has been updated to be able to run multi node tests, by automatically generating the relevant multinode.* properties. This means that you can easily run multi node tests on a single machine without any special configuration by just running them as normal multi-jvm tests. These tests can then be run distributed over multiple machines without any changes simply by using the multi-node additions to the plugin.

multi-node-test

multi-jvm:test

multi-node-test-only your.MultiNodeTest

multi-jvm:test-only your.MultiNodeTest

Preparing Your Project for Multi Node Testing

The multi node testing kit is a separate jar file. Make sure that you have the following dependency in your project:

"com.typesafe.akka" %% "akka-multi-node-testkit" % "2.3.16"

If you are using the latest nightly build you should pick a timestamped Akka version from https://repo.akka.io/snapshots/com/typesafe/akka/akka-multi-node-testkit_2.12/. We recommend against using SNAPSHOT in order to obtain stable builds.

A Multi Node Testing Example

First we need some scaffolding to hook up the MultiNodeSpec with your favorite test framework. Lets define a trait STMultiNodeSpec that uses ScalaTest to start and stop MultiNodeSpec.

package sample.multinode

import org.scalatest.{ BeforeAndAfterAll, WordSpecLike }
import org.scalatest.Matchers
import akka.remote.testkit.MultiNodeSpecCallbacks

/**
 * Hooks up MultiNodeSpec with ScalaTest
 */
trait STMultiNodeSpec extends MultiNodeSpecCallbacks
  with WordSpecLike with Matchers with BeforeAndAfterAll {

  override def beforeAll() = multiNodeSpecBeforeAll()

  override def afterAll() = multiNodeSpecAfterAll()
}

Then we need to define a configuration. Lets use two nodes "node1 and "node2" and call it MultiNodeSampleConfig.

package sample.multinode
import akka.remote.testkit.MultiNodeConfig

object MultiNodeSampleConfig extends MultiNodeConfig {
  val node1 = role("node1")
  val node2 = role("node2")
}

And then finally to the node test code. That starts the two nodes, and demonstrates a barrier, and a remote actor message send/receive.

package sample.multinode
import akka.remote.testkit.MultiNodeSpec
import akka.testkit.ImplicitSender
import akka.actor.{ Props, Actor }

class MultiNodeSampleSpecMultiJvmNode1 extends MultiNodeSample
class MultiNodeSampleSpecMultiJvmNode2 extends MultiNodeSample

object MultiNodeSample {
  class Ponger extends Actor {
    def receive = {
      case "ping" => sender() ! "pong"
    }
  }
}

class MultiNodeSample extends MultiNodeSpec(MultiNodeSampleConfig)
  with STMultiNodeSpec with ImplicitSender {

  import MultiNodeSampleConfig._
  import MultiNodeSample._

  def initialParticipants = roles.size

  "A MultiNodeSample" must {

    "wait for all nodes to enter a barrier" in {
      enterBarrier("startup")
    }

    "send to and receive from a remote node" in {
      runOn(node1) {
        enterBarrier("deployed")
        val ponger = system.actorSelection(node(node2) / "user" / "ponger")
        ponger ! "ping"
        expectMsg("pong")
      }

      runOn(node2) {
        system.actorOf(Props[Ponger], "ponger")
        enterBarrier("deployed")
      }

      enterBarrier("finished")
    }
  }
}

The easiest way to run this example yourself is to download Typesafe Activator and open the tutorial named Akka Multi-Node Testing Sample with Scala.

Things to Keep in Mind

There are a couple of things to keep in mind when writing multi node tests or else your tests might behave in surprising ways.

• Don't issue a shutdown of the first node. The first node is the controller and if it shuts down your test will break.
• To be able to use blackhole, passThrough, and throttle you must activate the failure injector and throttler transport adapters by specifying testTransport(on = true) in your MultiNodeConfig.
• Throttling, shutdown and other failure injections can only be done from the first node, which again is the controller.
• Don't ask for the address of a node using node(address) after the node has been shut down. Grab the address before shutting down the node.
• Don't use MultiNodeSpec methods like address lookup, barrier entry et.c. from other threads than the main test thread. This also means that you shouldn't use them from inside an actor, a future, or a scheduled task.

Configuration

There are several configuration properties for the Multi-Node Testing module, please refer to the reference configuration.