New Relic

Cinnamon provides a plugin that integrates with New Relic Insights using the New Relic Agent. This makes all Cinnamon metrics available as structured New Relic Insights events with rich metadata tags.

New Relic Java API dependency

Add the New Relic Java API dependency to your build:

sbt

copylibraryDependencies += "com.newrelic.agent.java" % "newrelic-api" % "5.1.0"

Maven

copy<dependency>
  <groupId>com.newrelic.agent.java</groupId>
  <artifactId>newrelic-api</artifactId>
  <version>5.1.0</version>
</dependency>

Gradle

copydependencies {
  implementation group: 'com.newrelic.agent.java', name: 'newrelic-api', version: '5.1.0'
}

Cinnamon dependency

We provide an easy-to-get-started plugin that contains all settings required for pushing Lightbend related metrics to New Relic.

First make sure that your build is configured to use the Cinnamon Agent.

Add the core Coda Hale Metrics dependency and the Cinnamon New Relic dependency to your build:

sbt

copylibraryDependencies += Cinnamon.library.cinnamonCHMetrics
libraryDependencies += Cinnamon.library.cinnamonNewRelic

Maven

copy<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-chmetrics</artifactId>
  <version>2.20.3</version>
</dependency>
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-newrelic</artifactId>
  <version>2.20.3</version>
</dependency>

Gradle

copydependencies {
  implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-chmetrics', version: '2.20.3'
}
dependencies {
  implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-newrelic', version: '2.20.3'
}

The Cinnamon New Relic reporter provides a preconfigured reporter for our Coda Hale Metrics integration. Here is how you enable the reporter, and an example of how to override some default values if you want to change the default behavior.

Required

copycinnamon.chmetrics {
  reporters += newrelic-reporter
}

Example

copycinnamon.chmetrics {
  reporters += newrelic-reporter

  newrelic-reporter {
    frequency = 60s

    report {
      histogram = ["min", "max", "p98", "p99", "p999"]
    }
  }
}

Reference

copycinnamon.chmetrics {
  newrelic-reporter {

    # Frequency of metric reporting to NewRelic Agent.
    frequency = 10s

    # Prefix for all metric keys.
    prefix = "lightbend"

    # Suffix for all metric keys.
    suffix = ""

    # Tags for all metrics.
    # Uses a "key-value" approach to generate the tags.
    # E.g. the following configuration:
    # tags {
    #   country = "UK"
    # }
    # will generate tags = "country" -> "UK"
    tags {}

    # Handle events.
    # Control if the reporter should send events to new relic as well.
    handle-events = false

    report {
      # Histogram fields to report for metrics.
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics.
      meter = ["samples", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]

      # Tags to include for metrics. Does not affect histogram or meter fields.
      include-tags = ["*"]

      # Tags to exclude for metrics. Does not affect histogram or meter fields.
      exclude-tags = []

      # Converts time from nano into specified unit. Use "ns" for no conversion.
      # Supported units: "ns" (default), "us", "ms", and "s".
      conversion-time-unit = "ns"

      # Group by category instead of name.
      # If enabled the New Relic event names will only include the metric category and not the name,
      # and a new tag named "metric" containing the metric name will be added.
      # E.g. the metric "lightbend:actors:mailbox_size" will be reported as "lightbend:actors" with
      # the tag "metric" -> "mailbox_size"
      group-by-category = false
    }
  }
}

Event reporting

By default all Cinnamon metrics are reported to New Relic, but it is also possible to report Cinnamon events to New Relic by changing the configuration like this:

copycinnamon.chmetrics.newrelic-reporter.handle-events = true

To manage the amount of data being reported to New Relic, you can disable or rate limit events via configuration.

Format

The information published to New Relic as Custom Events have their names formatted in this fashion:

[<prefix>:]<category>:<name>[:<suffix>]

As an example, the name of the akka actor mailbox size metric with the default configuration would be:

lighbend:actors:mailbox_size

All metrics and events have a field named type that is set to metric or event respectively.

Grouping by category

There is also the option to have the metrics and events, that are being sent to New Relic, grouped together at the category level, to make the number of names showing up in the New Relic UI smaller. To group the names together you need to change the configuration like this.

copycinnamon.chmetrics.newrelic-reporter.report.group-by-category = true

All metrics and events will now have a field named name, that contains the metric or event name, and the name of Custom Event reported to New Relic will be.

[<prefix>:]<category>[:<suffix>]

The same example for akka actor mailbox size will now have a Custom Event with the name lighbend:actors, as well as a field name containing mailbox_size.

New Relic time units

New Relic has support for time data formatting of Insight Events tags, but only for milliseconds and seconds. By default Cinnamon will report all time related metrics to New Relic in nanoseconds. See settings in the reference.conf for more information of how to configure the behavior.

New Relic queries

To plot the data that is reported into New Relic Insights, you need to create your own graphs with some queries.

To help you understand the metrics you are plotting, a good place to get more insights into the metrics is the Data Explorer UI. To use it, go to the Data Explorer page from New Relic Insights navigation panel. Under the Events tab, select the metric you are trying to plot. Then click on See All Samples button to see sample data with all the fields. By default, the graph plotted on the page shows how many events are received. To plot more interesting graphs, change the query at the top of the page, then click on Run.

This is a sample query for plotting the akka actor mailbox size metric per actor:

SELECT average(mean) FROM `lightbend:actors:mailbox_size` FACET application,actor,host WHERE type = 'metric' SINCE 30 MINUTES AGO TIMESERIES

And the same query when metrics and events are grouped by category:

SELECT average(mean) FROM `lightbend:actors` FACET application,actor,host WHERE type = 'metric' AND name = 'mailbox_size' SINCE 30 MINUTES AGO TIMESERIES

For graphs with time, make sure you use the most appropriate time unit instead of the default nano seconds.

SELECT average(mean)/1000000 FROM `lightbend:dispatchers:queue_time` FACET application,dispatcher,host WHERE type = 'metric' SINCE 30 MINUTES AGO TIMESERIES

Depending on the type of the graph, the field that you select to plot your graphs changes too. For example, you should plot the graph for dispatchers:processed_tasks using the value field

SELECT average(value) FROM `lightbend:dispatchers:processed_tasks` FACET application,dispatcher,host WHERE type = 'metric' SINCE 30 MINUTES AGO TIMESERIES

New Relic dashboards

Creating dashboards for metrics provides an easy way for fast metric look-ups. You can create dashboards from the New Relic Insights navigation panel. Then add metrics to these dashboards from the Data Explorer screen.

Each dashboard should be at most one category of metrics for better filtering capabilities. For example, group all dispatchers metrics in the dispatcher dashboard. Because all the dispatcher metrics have the same facets, you can enable filters on this dashboard and select available facets to plot your metrics per application, dispatcher or host.

New Relic dependency

For information about how to get started with New Relic we refer to their getting started instructions. Go to their website, create an account and follow the instructions for your particular environment. Please note that the Cinnamon New Relic integration uses the New Relic Agent to send information to New Relic Insights, so that agent needs to be enabled and configured.

Instructions on how to disable specific async frameworks and modules can be found on the on the New Relic site.