New Relic
Cinnamon provides a plugin that integrates with New Relic Insights using the New Relic Agent. This makes all Cinnamon metrics available as structured New Relic Insights events with rich metadata tags.
New Relic Java API dependency
Add the New Relic Java API dependency to your build:
- sbt
-
libraryDependencies += "com.newrelic.agent.java" % "newrelic-api" % "5.1.0"
- Maven
-
<dependency> <groupId>com.newrelic.agent.java</groupId> <artifactId>newrelic-api</artifactId> <version>5.1.0</version> </dependency>
- Gradle
-
dependencies { implementation group: 'com.newrelic.agent.java', name: 'newrelic-api', version: '5.1.0' }
Cinnamon dependency
We provide an easy-to-get-started plugin that contains all settings required for pushing Lightbend related metrics to New Relic.
First make sure that your build is configured to use the Cinnamon Agent.
Add the core Coda Hale Metrics dependency and the Cinnamon New Relic dependency to your build:
- sbt
-
libraryDependencies += Cinnamon.library.cinnamonCHMetrics libraryDependencies += Cinnamon.library.cinnamonNewRelic
- Maven
-
<dependency> <groupId>com.lightbend.cinnamon</groupId> <artifactId>cinnamon-chmetrics</artifactId> <version>2.21.2</version> </dependency> <dependency> <groupId>com.lightbend.cinnamon</groupId> <artifactId>cinnamon-newrelic</artifactId> <version>2.21.2</version> </dependency>
- Gradle
-
dependencies { implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-chmetrics', version: '2.21.2' } dependencies { implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-newrelic', version: '2.21.2' }
The default Coda Hale Metrics module uses Metrics 4 but Coda Hale Metrics 3 can also be used.
The Cinnamon New Relic reporter provides a preconfigured reporter for our Coda Hale Metrics integration. Here is how you enable the reporter, and an example of how to override some default values if you want to change the default behavior.
- Required
-
cinnamon.chmetrics { reporters += newrelic-reporter }
- Example
-
cinnamon.chmetrics { reporters += newrelic-reporter newrelic-reporter { frequency = 60s report { histogram = ["min", "max", "p98", "p99", "p999"] } } }
- Reference
-
cinnamon.chmetrics { newrelic-reporter { # Frequency of metric reporting to NewRelic Agent. frequency = 10s # Prefix for all metric keys. prefix = "lightbend" # Suffix for all metric keys. suffix = "" # Tags for all metrics. # Uses a "key-value" approach to generate the tags. # E.g. the following configuration: # tags { # country = "UK" # } # will generate tags = "country" -> "UK" tags {} # Handle events. # Control if the reporter should send events to new relic as well. handle-events = false report { # Histogram fields to report for metrics. histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"] # Meter fields to report for metrics. meter = ["samples", "min1_rate", "min5_rate", "min15_rate", "mean_rate"] # Tags to include for metrics. Does not affect histogram or meter fields. include-tags = ["*"] # Tags to exclude for metrics. Does not affect histogram or meter fields. exclude-tags = [] # Converts time from nano into specified unit. Use "ns" for no conversion. # Supported units: "ns" (default), "us", "ms", and "s". conversion-time-unit = "ns" # Group by category instead of name. # If enabled the New Relic event names will only include the metric category and not the name, # and a new tag named "metric" containing the metric name will be added. # E.g. the metric "lightbend:actors:mailbox_size" will be reported as "lightbend:actors" with # the tag "metric" -> "mailbox_size" group-by-category = false } } }
These settings are defined in the reference.conf
. You only need to specify any of these settings when you want to override the defaults.
Event reporting
By default all Cinnamon metrics are reported to New Relic, but it is also possible to report Cinnamon events to New Relic by changing the configuration like this:
cinnamon.chmetrics.newrelic-reporter.handle-events = true
To manage the amount of data being reported to New Relic, you can disable or rate limit events via configuration.
The total number of times that an event has fired is automatically reported as a metric, so there will be one instance with type = metric
and one instance with type = event
with the same name.
Format
The information published to New Relic as Custom Events have their names formatted in this fashion:
[<prefix>:]<category>:<name>[:<suffix>]
As an example, the name of the akka actor mailbox size metric with the default configuration would be:
lighbend:actors:mailbox_size
All metrics and events have a field named type
that is set to metric
or event
respectively.
Grouping by category
There is also the option to have the metrics and events, that are being sent to New Relic, grouped together at the category level, to make the number of names showing up in the New Relic UI smaller. To group the names together you need to change the configuration like this.
cinnamon.chmetrics.newrelic-reporter.report.group-by-category = true
All metrics and events will now have a field named name
, that contains the metric or event name, and the name of Custom Event reported to New Relic will be.
[<prefix>:]<category>[:<suffix>]
The same example for akka actor mailbox size will now have a Custom Event with the name lighbend:actors
, as well as a field name
containing mailbox_size
.
New Relic time units
New Relic has support for time data formatting of Insight Events tags, but only for milliseconds and seconds. By default Cinnamon will report all time related metrics to New Relic in nanoseconds. See settings in the reference.conf
for more information of how to configure the behavior.
New Relic queries
To plot the data that is reported into New Relic Insights
, you need to create your own graphs with some queries.
To help you understand the metrics you are plotting, a good place to get more insights into the metrics is the Data Explorer
UI. To use it, go to the Data Explorer
page from New Relic Insights
navigation panel. Under the Events
tab, select the metric you are trying to plot. Then click on See All Samples
button to see sample data with all the fields. By default, the graph plotted on the page shows how many events are received. To plot more interesting graphs, change the query at the top of the page, then click on Run
.
This is a sample query for plotting the akka actor mailbox size metric per actor:
SELECT average(mean) FROM `lightbend:actors:mailbox_size` FACET application,actor,host WHERE type = 'metric' SINCE 30 MINUTES AGO TIMESERIES
And the same query when metrics and events are grouped by category:
SELECT average(mean) FROM `lightbend:actors` FACET application,actor,host WHERE type = 'metric' AND name = 'mailbox_size' SINCE 30 MINUTES AGO TIMESERIES
For graphs with time, make sure you use the most appropriate time unit instead of the default nano seconds.
SELECT average(mean)/1000000 FROM `lightbend:dispatchers:queue_time` FACET application,dispatcher,host WHERE type = 'metric' SINCE 30 MINUTES AGO TIMESERIES
Depending on the type of the graph, the field that you select to plot your graphs changes too. For example, you should plot the graph for dispatchers:processed_tasks
using the value
field
SELECT average(value) FROM `lightbend:dispatchers:processed_tasks` FACET application,dispatcher,host WHERE type = 'metric' SINCE 30 MINUTES AGO TIMESERIES
New Relic dashboards
Creating dashboards for metrics provides an easy way for fast metric look-ups. You can create dashboards from the New Relic Insights
navigation panel. Then add metrics to these dashboards from the Data Explorer
screen.
Each dashboard should be at most one category of metrics for better filtering capabilities. For example, group all dispatchers
metrics in the dispatcher dashboard. Because all the dispatcher metrics have the same facets, you can enable filters on this dashboard and select available facets to plot your metrics per application
, dispatcher
or host
.
New Relic dependency
For information about how to get started with New Relic we refer to their getting started instructions. Go to their website, create an account and follow the instructions for your particular environment. Please note that the Cinnamon New Relic integration uses the New Relic Agent to send information to New Relic Insights, so that agent needs to be enabled and configured.
We recommend using newer versions of the New Relic Agent starting from version 5.1.0 to the latest. Older versions of the New Relic Agent are known to interfere with some of the instrumentations that the Cinnamon Agent does, to Akka and Scala, So if you are using version 5.0.0 and below then these must be disabled in the New Relic Agent.
Instructions on how to disable specific async frameworks and modules can be found on the on the New Relic site.