Akka Diagnostics Recorder

The Akka Diagnostics Recorder writes configuration and system information to a file that can be attached to your Lightbend support cases. The information will help us at Lightbend to give you the best possible support.

It will also register a MBean in the “akka” name space, which can be accessed from a JMX console such as JConsole. From JMX you can trigger thread dumps that will also be appended to the file.

Using the Diagnostics Recorder

To use the Diagnostics Recorder feature a dependency on the akka-diagnostics artifact must be added.

sbt

// Add Lightbend Platform to your build as documented at https://developer.lightbend.com/docs/lightbend-platform/introduction/getting-started/subscription-and-credentials.html
"com.lightbend.akka" %% "akka-diagnostics" % "1.1.16"

Gradle

// Add Lightbend Platform to your build as documented at https://developer.lightbend.com/docs/lightbend-platform/introduction/getting-started/subscription-and-credentials.html
dependencies {
  compile group: 'com.lightbend.akka', name: 'akka-diagnostics_2.11', version: '1.1.16'
}

Maven

<!-- Add Lightbend Platform to your build as documented at https://developer.lightbend.com/docs/lightbend-platform/introduction/getting-started/subscription-and-credentials.html -->
<dependency>
  <groupId>com.lightbend.akka</groupId>
  <artifactId>akka-diagnostics_2.11</artifactId>
  <version>1.1.16</version>
</dependency>

Before you can access this library, you’ll need to configure the Lightbend repository and credentials in your build.

When this dependency is included the Diagnostics Recorder is automatically run when the ActorSystem is started.

File Location

The diagnostics report file is by default created in directory named akka-diagnostics in the current working directory. This can be changed with configuration:

akka.diagnostics.recorder.dir = "target/diagnostics/"

The report file name contains the name of the actor system name and remote address if any.

The file is overwritten if the actor system is restarted with same name and remote address. If the actor system is configured with port=0 a new file will be created for each used port, which will lead to possibly quite a lot of files over time. It might make sense to clear the directory from such files before starting the actor system.

Configuration and system information

When the actor system is started the Diagnostics Recorder collects configuration and system information and writes to the diagnostics report file. This information includes:

• Start time in UTC
• Akka version
• full classpath
• configuration warnings reported by the Config Checker
• application configuration and JVM system properties
• JVM and system metrics such as
  • heap and memory pool settings
  • available processors

Sensitive configuration

The diagnostics report file is in human readable JSON format so you can inspect it before sending it to Lightbend.

Some configuration properties might not be appropriate to share with others and such properties can be excluded from the diagnostics report by adding the paths of the properties to the akka.diagnostics.recorder.sensitive-config-paths configuration string list. By default the following properties/sections are excluded:

sensitive-config-paths = [
  "user.home", "user.name", "user.dir",
  "socksNonProxyHosts", "http.nonProxyHosts", "ftp.nonProxyHosts",
  "akka.remote.secure-cookie",
  "akka.remote.netty.ssl.security",
  "akka.persistence.gdpr.jca-provider.keystore-password",
  "akka.persistence.gdpr.jca-provider.keystore-location",
  "cassandra-snapshot-store.authentication",
  "cassandra-journal.authentication",
  "lagom.persistence.read-side.cassandra.authentication",
  "couchbase-journal.connection.username",
  "couchbase-journal.connection.password",
  "akka-persistence-jdbc.shared-databases.slick.db.user",
  "akka-persistence-jdbc.shared-databases.slick.db.password",
  "jdbc-journal.slick.db.user",
  "jdbc-journal.slick.db.password",
  "jdbc-snapshot-store.slick.db.user",
  "jdbc-snapshot-store.slick.db.password",
  "jdbc-read-journal.slick.db.user",
  "jdbc-read-journal.slick.db.password"
]

Thread Dumps

The Diagnostics Recorder registers a JMX MBean in the “akka” name space, which can be accessed from a JMX console such as JConsole. From JMX you can trigger thread dumps that will also be appended to the diagnostics report file.

When you click the collectThreadDumps button a number of thread dumps are collected. The number of thread dumps can be configured with:

akka.diagnostics.recorder.collect-thread-dumps-count = 5

There is also a collectThreadDumps operation with a count parameter in case you want to collect more or less than the configured number of thread dumps.

There is a delay between each thread dump, which can be configured with:

akka.diagnostics.recorder.collect-thread-dumps-interval = 1s

In addition to the thread dump some basic metrics are also captured and written to the file, such as:

• system load average
• heap memory and non-heap usage
• GC collection count and time

To be able to access JMX on remote machines (e.g. cluster nodes) you must enable remote monitoring and management when starting the JVMs, as described in Monitoring and Management Using JMX Technology

Akka Management Integration

When using Akka Management the config checker can be accessed through the Akka Management HTTP endpoints
akka.diagnostics.management.enabled is set to true (default is false).

The following paths are available, all returning JSON:

• GET /diagnostics/startup-info - the same system and config information as dumped to a file on startup
• GET /diagnostics/thread-dump - a thread dump is taken triggered immediately by the call
• GET /diagnostics/config-check - the configuration of the app is checked with the config checker for problems and a report of that is returned.

Disable

It is possible to disable the recorder completely with:

akka.diagnostics.recorder.enabled = off

It is possible to disable the JMX feature but still enable other parts of the recorder:

akka.diagnostics.recorder.jmx-enabled = off

Configuration

Below is the configuration of the Diagnostics Recorder, which you may amend to adjust its behavior.

akka.diagnostics.recorder {

  # Set this to "off" to disable the Diagnostics Recorder completely
  enabled = on

  # Directory where the diagnostics report file is stored. The file name will
  # include the name of the actor system.
  dir = "akka-diagnostics/"

  # Some configuration properties might not be appropriate to share with others
  # and such properties can be excluded from the diagnostics report by adding
  # the paths of the properties to this list. All properties starting with
  # the paths defined here are excluded, i.e. you can add the path of a whole
  # section here to skip everything inside that section.
  sensitive-config-paths = [
    "user.home", "user.name", "user.dir",
    "socksNonProxyHosts", "http.nonProxyHosts", "ftp.nonProxyHosts",
    "akka.remote.secure-cookie",
    "akka.remote.netty.ssl.security",
    "akka.persistence.gdpr.jca-provider.keystore-password",
    "akka.persistence.gdpr.jca-provider.keystore-location",
    "cassandra-snapshot-store.authentication",
    "cassandra-journal.authentication",
    "lagom.persistence.read-side.cassandra.authentication",
    "couchbase-journal.connection.username",
    "couchbase-journal.connection.password",
    "akka-persistence-jdbc.shared-databases.slick.db.user",
    "akka-persistence-jdbc.shared-databases.slick.db.password",
    "jdbc-journal.slick.db.user",
    "jdbc-journal.slick.db.password",
    "jdbc-snapshot-store.slick.db.user",
    "jdbc-snapshot-store.slick.db.password",
    "jdbc-read-journal.slick.db.user",
    "jdbc-read-journal.slick.db.password"
  ]

  # The initial report information, such as configuration information, is
  # created in a background thread after this delay when the actor system has
  # been started.
  startup-report-after = 2s

  # Set this to "off" to disable the JMX interface
  jmx-enabled = on

  # This number of thread dumps are created when triggered from the `threadDump`
  # operation in the JMX interface. The thread dumps are collected with a delay
  # between each, which is configured by `collect-thread-dumps-interval`.
  collect-thread-dumps-count = 5

  # The delay between each thread dump. The number of thread dumps is
  # configured by collect-thread-dumps-count or by the JMX count
  # parameter to the `threadDump` operation.
  collect-thread-dumps-interval = 1s

  # All tasks of the Diagnostics Recorder are executed by this dispatcher.
  # Since some tasks involve blocking file operations it is important to
  # run them on a separate dispatcher.
  dispatcher = {
    executor = thread-pool-executor
    thread-pool-executor.core-pool-size-min = 1
    thread-pool-executor.core-pool-size-max = 1
  }
}