Actor configuration

By default, actors are not instrumented, this is to avoid potential impacts on application performance. You must explicitly specify which actors will be selected for instrumentation, via the cinnamon.akka.actors block in your application.conf file.

Actor selection

It is possible to select actors for instrumentation based on their class (and enclosing package), path or tags. Class based selection supports wildcards, and path based selection supports wildcards and templating. The following sections will provide examples:

Class and package selection

The following are examples of targeting a specific class (A), and a specific inner class (A$B):

"com.example.a.A"     // specific class
"com.example.a.A$B"   // specific inner class

The following examples demonstrate the use of wildcards to select all classes within a package,

"com.example.a.*"     // all classes within the package
"com.example.a.A$*"   // all inner classes within A

Path selection

The following is a valid actor selection using the full actor path:

"/user/a/b"           // full actor path

Actors under /user and /system can be targeted this way,

The following are examples use wildcards to select all actors under a given path:

"/user/a/*"           // all actors under a specific path

Some actors are created as system actors under the /system guardian actor, rather than the /user guardian actor. For example, sharded actors are created under /system/sharding by default. You can select system actors by class or by package, or by using a path selection such as /system/sharding/*.

Tag Selection

The following example will select actors that are tagged with specific entries:

"tag:my-tag"          // any actor whose props contain the tag "my-tag"

You can assign tags to your actors by creating an ActorTags instance, and passing to an actor’s props:

copyval includedTagProps = ActorTags("tag1", "tag2") // actors under /user/t/* with these tags would be included
val excludedTagProps = ActorTags("tag1", "exclude-me") // actors under /user/t/* with these tags would be excluded

There is no wildcard support for tag selectors.

Actor settings

Settings can be applied to each configured actor selection. The main setting is report-by, for enabling instrumentation and deciding how the selected actors will report metrics.

Report-by settings

Instrumentation is enabled for an actor selection using the report-by setting. This allows you to have metrics reported by:

• instance where all metrics will be broken down by individual actor path
• class where metrics for all actors of the same class will be aggregated under the class name (including package)
• tag where metrics are aggregated under a tag that is shared by one or more actors
• group where metrics are aggregated under an arbitrary group name if instance, class or tag are not appropriate

You can find more about group reporting, and dynamically generating groups via templating in the actor groups and template groups sections.

Here’s an example configuration that enables instrumentation for several actors:

copycinnamon.akka {
  actors {
    "com.example.a.A" {
      report-by = class
    }
    "com.example.a.b.*" {
      report-by = instance
    }
    "/user/x/y" {
      report-by = instance
    }
    "/user/x/y/*" {
      report-by = class
    }
    "tag:heavyweight" {
      report-by = tag
    }
  }
}

Actor excludes

When selecting multiple actors (e.g., selecting by class, tag or with wildcards), you are able to also specify exclusions, these are configured by the excludes directive in configuration blocks:

copycinnamon.akka {
  actors {
    "com.example.a.*" {                                     // include everything in package
      report-by = class
      excludes = ["com.example.a.B", "com.example.a.b.*"]   // exclude specific class and specific sub-package
    }
    "/user/x/*" {                                           // include all paths under /user/x
      report-by = instance
      excludes = "/user/x/y/z"                              // exclude specific sub-path
    }
    "/user/t/*" {                                           // include all paths under /user/t
      report-by = group
      excludes = "tag:heavyweight"                          // exclude any actors tagged with "heavyweight"
    }
  }
}

The above examples show that different kinds of actor selectors can be mixed for selecting and excluding actors, the examples here demonstrate:

• including by package, excluding by class and sub-package
• include by wildcard path, exclude by specific sub-path
• include by wildcard path, exclude by actor tag

Routers

Metrics for routers are created specifically by default. This comes with an impact on the performance and should you want to minimize this impact it is possible to turn off the router metrics for a selection with the setting routers = off:

copycinnamon.akka {
  actors {
    "com.example.a.b.*" {
      report-by = instance
      routers = off
    }
  }
}

Router configuration settings

Routers are unique in that they are created automatically by Akka. Consider the following example code:

// inside "myActor"
val router1: ActorRef = context.actorOf(RoundRobinPool(10).props(Props[SomeRoutee]), "r1")
//...
val router2: ActorRef = context.actorOf(RoundRobinPool(5).props(Props[AnotherRoutee]), "r2")

When setting up the configuration for the code above, it is important to understand the underlying mechanism in Akka. The router actor class used in Akka is akka.routing.RouterActor and this means that if you decide to use report-by=class all metrics for all routers in that actor system will be grouped together into this particular class. In other words, let’s say your actor system contains two routers, router1 and router2, and that you have report-by=class configured, this means that the metrics gathered will be merged into one class, i.e. akka.routing.RouterActor.

If you instead want to see metrics for each router then configure report-by=instance for each of them. In the case of instance, the metrics will be reported for each router instance, and each router actor path will be used as the identifier. In the example above this means that metrics will be generated for both /user/myActor/r1 and /user/myActor/r2.

Actor groups

Metrics can be grouped by actor class, as described above. Other groups can be formed using the group setting for report-by.

For example, the following configuration selects all actors in a package and then reports metrics for all these actors as a group, named by the package selection com.example.a.b.*:

copycinnamon.akka {
  actors {
    "com.example.a.b.*" {
      report-by = group
    }
  }
}

The following configuration selects all actors in a subtree and reports metrics for these actors as a group, named by the subtree selection /user/x/y/*:

copycinnamon.akka {
  actors {
    "/user/x/y/*" {
      report-by = group
    }
  }
}

Groups can have any name assigned. By default, the name and the actor selection are the same, but the name can also be defined separately by putting actor selections under includes.

For example, the following configuration selects all actors in either the com.example.a.b.* package, under the /user/x/y/* subtree, or tagged with heavyweight and then reports metrics for these actors as a group named actor-group:

copycinnamon.akka {
  actors {
    "actor-group" {
      report-by = group
      includes = ["com.example.a.b.*", "/user/x/y/*", "tag:heavyweight" ]
    }
  }
}

Template groups

It is also possible to define groups to be created dynamically using actor path patterns containing a ?. These definitions must have the report-by set to group and no includes setting since that is automatically created.

The following configuration will create groups for every actor below the /user/e/ path and have all children of that actor report to the corresponding group:

copycinnamon.akka {
  actors {
    "/user/e/?/*" {
      report-by = group
    }
  }
}

These groups can of course have the normal thresholds and excludes settings as well as special group relative excludes settings.

The following configuration will create groups for every actor below the /user/e/ path but exclude the path named /user/e/r2/*:

copycinnamon.akka {
  actors {
    "/user/e/?/*" {
      report-by = group
      excludes = "./r2/*"
    }
  }
}

Multiple report-by settings

Actors may match multiple actor selections and can report metrics under both class and instance. It’s also possible to specify multiple report-by settings for a single actor selection. For example, given the following configuration:

copycinnamon.akka {
  actors {
    "com.example.a.*" {
      report-by = class
    }
    "com.example.a.b.*" {
      report-by = instance
    }
    "/user/x/y/*" {
      report-by = [class, instance, tag]
    }
  }
}

An actor with the class com.example.a.b.C or the path /user/x/y/z would report metrics under both actor class and actor path and any actor under /user/x/y/* would also report by tag.

Metric sampling periods

The mailbox size and stash size metrics are sampled, at most once every 1 second by default, to avoid skew in the recorded distributions. The sampling period for these size metrics can be configured (or disabled by setting the sampling period to 0). Here’s an example of changing the sampling periods for a group of actors:

copycinnamon.akka {
  actors {
    "com.example.a.b.*" {
      report-by = class
      metrics {
        mailbox-size {
          sampling-period = 100ms
        }
        stash-size {
          sampling-period = 100ms
        }
      }
    }
  }
}

The mailbox and stash are only sampled when they change. To further correct skew in the recorded distributions, if the sampling period is smaller than the interval for changes, then missed samples will be automatically backfilled. So given a configured sampling period of 100ms, if the mailbox is sampled at 10 queued messages and 500ms later is decremented to 9 queued messages, then the mailbox size is recorded at 10 four times to backfill missed samples and then at 9 once. The default limit for backfilling is 4 samples. The sampling backfill limit can be configured (or disabled by setting the sampling backfill to 0). Here’s an example of changing the sampling periods and backfill limits for a group of actors:

copycinnamon.akka {
  actors {
    "com.example.a.b.*" {
      report-by = class
      metrics {
        mailbox-size {
          sampling-period = 100ms
          sampling-backfill = 9
        }
        stash-size {
          sampling-period = 100ms
          sampling-backfill = 9
        }
      }
    }
  }
}

Metric thresholds

Some metrics support configured thresholds. If the threshold is exceeded then an event is recorded. Threshold values are set under thresholds in the actor settings.

The following threshold settings are supported:

Setting	Type	Description
mailbox-size	Integer	Mailbox queue grows too large
stash-size	Integer	Stash queue grows too large
mailbox-time	Duration	Message has been in the mailbox for too long
processing-time	Duration	Message processing takes too long
remote-message-size	Bytes	Remote message is too big

Thresholds are only checked when an actor selection has a threshold enabled via configuration. Here is an example of adding thresholds to select actors:

copycinnamon.akka {
  actors {
    "com.example.a.A" {
      report-by = class
      thresholds {
        mailbox-size = 1000
        stash-size = 50
        mailbox-time = 3s
        processing-time = 500ms
      }
    }
    "com.example.a.b.*" {
      report-by = class
      thresholds {
        mailbox-size = 100
        remote-message-size = 512K
      }
    }
    "/user/x/y" {
      report-by = instance
      thresholds {
        mailbox-time = 200ms
      }
    }
    "/user/x/y/*" {
      report-by = class
      thresholds {
        stash-size = 100
        processing-time = 1s
        remote-message-size = 1M
      }
    }
  }
}

Setting fallbacks

An actor class may match multiple actor selections in the configuration. Reporting settings will be merged together but other settings, like thresholds, will ‘fallback’ to higher matches. Specific actor settings will inherit the higher settings.

For example:

copycinnamon.akka {
  actors {
    "com.example.a.*" {
      report-by = class
      thresholds {
        mailbox-size = 1000
        mailbox-time = 3s
        processing-time = 500ms
      }
    }
    "com.example.a.b.*" {
      report-by = class
      thresholds {
        mailbox-size = 100
        stash-size = 125
      }
    }
    "com.example.a.b.C" {
      report-by = class
      thresholds {
        mailbox-time = 1s
      }
    }
  }
}

Given the above configuration, the actor class com.example.a.b.C will have the following threshold settings:

mailbox-time = 1s        # provided explicitly
mailbox-size = 100       # inherited from "com.example.a.b.*"
stash-size = 125         # inherited from "com.example.a.b.*"
processing-time = 500ms  # inherited from "com.example.a.*"

Dimensions

When metrics and events are reported for actors, the identity of the actor will contain two dimensions: actor system and dispatcher. These may be represented in the metric key or in metric tags, depending on the backend used. Whether or not these dimensions are enabled can be configured using the dimensions setting. For example, here is the configuration to disable the grouping of actors by dispatcher and only use the actor system dimension:

copycinnamon.akka {
  dimensions = [actor-system]
}

There is also an address dimension that is disabled by default for backwards compatibility reasons. This dimension represents the actor system address in the form host:port, with local being used if the actor system doesn’t have remoting enabled. To enable the address dimension simply add it in configuration like this:

copycinnamon.akka {
  dimensions += address
}

Message type dimensions

Opt-in message type dimensions are available that enable collecting some metrics by message type. By default, these dimensions are off but can be enabled for each selection separately:

copycinnamon.akka {
  actors {
    "com.example.a.b.C" {
      report-by = class
      message-type = on
    }
  }
}

The message-type dimension provides better granularity collecting some metrics separately per message-type instead of all messages per actor group.