OpenTelemetry - Metrics
You can export Cinnamon metrics to OpenTelemetry.
OpenTelemetry dependency
First make sure that your build is configured to use the Cinnamon Agent and has instrumentations enabled, such as Akka instrumentation or Akka HTTP instrumentation.
- sbt
-
libraryDependencies += Cinnamon.library.cinnamonOpenTelemetry
- Maven
-
<dependency> <groupId>com.lightbend.cinnamon</groupId> <artifactId>cinnamon-opentelemetry</artifactId> <version>2.20.3</version> </dependency>
- Gradle
-
dependencies { implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-opentelemetry', version: '2.20.3' }
Metric exporters
There are two options for pushing metrics to an OpenTelemetry collector via the OpenTelemetry wire protocol (OTLP) using either gRPC or HTTP.
Furthermore, a logging exporter can be configured to print metrics to the console.
gRPC metric exporter
To use the gRPC exporter you need to provide the following configuration:
- Required
-
cinnamon.opentelemetry.metrics { exporters += grpc-exporter }
- Example
-
cinnamon.opentelemetry.metrics { exporters += grpc-exporter grpc-exporter { interval = 15 seconds } }
- Reference
-
cinnamon.opentelemetry.metrics { grpc-exporter { # Define the interval of reads. interval = 1 minute # Sets the OTLP endpoint to connect to. The endpoint must start with either http:// or https://. endpoint = "http://localhost:4317" # The maximum time to wait for the collector to process an exported batch of metrics. timeout = 10 seconds # The maximum time to wait for new connections to be established. connect-timeout = 10 seconds # The method used to compress payloads. If "none", compression is disabled. Currently supported compression methods include "gzip" and "none". compression = "none" # Add header to request. Optional. headers { # some-key = "some value" } # Set the retry policy. Retry is disabled by default. retry-policy = "none" # Enabled when `retry-policy = "default-retry-policy"`. Configured to use the defaults. default-retry-policy { # Set the maximum number of attempts, including the original request. Must be greater than 1 and less than 6. max-attempts = 5 # Set the initial backoff. Must be greater than 0. initial-backoff = 1 second # Set the maximum backoff. Must be greater than 0. max-backoff = 5 seconds # Set the backoff multiplier. Must be greater than 0.0. backoff-multiplier = 1.5 } # Set the aggregation temporality selector. # Pre-defined selectors are "always-cumulative" or "delta-preferred". # A custom selector can be defined with config in the following format: # # aggregation-temporality-selector = some-custom-selector # # # a custom selector that is equivalent to `delta-preferred` # # custom selector section can be named appropriately # some-custom-selector { # # specify the default, or otherwise defaults to cumulative # default = delta # # # select aggregation temporality for particular instrument types # select { # up-down-counter = cumulative # observable-up-down-counter = cumulative # } # } # aggregation-temporality-selector = always-cumulative } }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.
HTTP metric exporter
To use the HTTP exporter you need to provide the following configuration:
- Required
-
cinnamon.opentelemetry.metrics { exporters += http-exporter }
- Example
-
cinnamon.opentelemetry.metrics { exporters += http-exporter http-exporter { interval = 15 seconds } }
- Reference
-
cinnamon.opentelemetry.metrics { http-exporter { # Define the interval of reads interval = 1 minute # Sets the OTLP endpoint to connect to. The endpoint must start with either http:// or https://, and include the full HTTP path. endpoint = "http://localhost:4318/v1/metrics" # Sets the maximum time to wait for the collector to process an exported batch of metrics. timeout = 10 seconds # The maximum time to wait for new connections to be established. connect-timeout = 10 seconds # Sets the method used to compress payloads. If "none", compression is disabled. Currently supported compression methods include "gzip" and "none". compression = "none" # Add header to request. Optional. headers { # some-key = "some value" } # Set the retry policy. Retry is disabled by default. retry-policy = "none" # Enabled when `retry-policy = "default-retry-policy"`. Configured to use the defaults. default-retry-policy { # Set the maximum number of attempts, including the original request. Must be greater than 1 and less than 6. max-attempts = 5 # Set the initial backoff. Must be greater than 0. initial-backoff = 1 second # Set the maximum backoff. Must be greater than 0. max-backoff = 5 seconds # Set the backoff multiplier. Must be greater than 0.0. backoff-multiplier = 1.5 } # Set the aggregation temporality selector. # Pre-defined selectors are "always-cumulative" or "delta-preferred". # A custom selector can be defined with config in the following format: # # aggregation-temporality-selector = some-custom-selector # # # a custom selector that is equivalent to `delta-preferred` # # custom selector section can be named appropriately # some-custom-selector { # # specify the default, or otherwise defaults to cumulative # default = delta # # # select aggregation temporality for particular instrument types # select { # up-down-counter = cumulative # observable-up-down-counter = cumulative # } # } # aggregation-temporality-selector = always-cumulative } }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.
Logging metric exporter
To use the logging exporter you need to provide the following configuration:
- Required
-
cinnamon.opentelemetry.metrics { exporters += logging-exporter }
- Example
-
cinnamon.opentelemetry.metrics { exporters += logging-exporter logging-exporter { interval = 15 seconds } }
- Reference
-
cinnamon.opentelemetry.metrics { logging-exporter { # Define the interval of reads. interval = 1 minute # Set the aggregation temporality: `cumulative` or `delta` aggregation-temporality = cumulative } }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.
Custom metrics
Custom metrics are mapped to the following OpenTelemetry metrics:
Cinnamon type | OpenTelemetry type |
---|---|
Counter | LongUpDownCounter |
GaugeDouble | ObservableDoubleGauge |
GaugeLong | ObservableLongGauge |
ProvidingGaugeDouble | ObservableDoubleGauge |
ProvidingGaugeLong | ObservableLongGauge |
Rate | LongCounter |
Recorder | LongHistogram |
Resource Attributes
To attach a custom resource attribute to every metric, you can use the following config:
- Example
-
cinnamon.opentelemetry { # Based on https://opentelemetry.io/docs/specs/semconv/resource/ resource-attributes { cloud.provider = "aws" cloud.region = "us-east-1" cloud.region = ${?AWS_REGION} } }
Metric hints
Metrics—either created by Cinnamon instrumentation or custom metrics—can be given hints that will passed to the metric backend. These hints allow particular metrics to be configured differently from the default configuration.
For example, using the custom metrics API you can pass in the hint when creating a custom recorder:
- Scala
-
val recorder = CinnamonMetrics(context).createRecorder("recorder", hints = Set("custom-recorder"))
- Java
-
Recorder recorder = CinnamonMetrics.get(context).createRecorder("recorder", ImmutableSet.of("custom-recorder"));
Metric advice
OpenTelemetry metrics can be configured by providing “advice” to the OpenTelemetry SDK. Advice can be applied to all metrics under configuration for cinnamon.opentelemetry.metrics.defaults
, or applied to particular metrics that have metric hints.
Metric attributes
Providing metric advice for attributes acts as an attribute filter. Given the following custom metric:
- Scala
-
val counter = CinnamonMetrics(context).createCounter( "some-counter", tags = Map("some-attribute" -> "some-value", "other-attribute" -> "other-value"), hints = Set("filter-attributes") )
- Java
-
ImmutableMap<String, String> tags = ImmutableMap.of( "some-attribute", "some-value", "other-attribute", "other-value"); ImmutableSet<String> hints = ImmutableSet.of("filter-attributes"); Counter counter = CinnamonMetrics.get(context).createCounter("some-counter", tags, hints);
An attribute filter can be configured for this metric hint using the attributes
advice setting:
- Example
-
cinnamon.opentelemetry { metrics.hints { filter-attributes { advice { attributes = ["some-attribute"] } } } }
In this example, only the some-attribute
attribute will be exported for metrics with the configured metric hint. Any other attributes, including other-attribute
, will not be exported.
Histogram buckets
A Recorder
in Cinnamon is backed by an OpenTelemetry LongHistogram
. The buckets for a histogram can be configured for a hinted metric as follows:
- Example
-
cinnamon.opentelemetry { metrics.hints { recorded-buckets { advice { histogram.buckets = [1, 2, 4, 8, 16, 32, 64, 128] } } } }
- Defaults
-
# Advice for instruments, using instrument extensions. advice { # Recommended set of attribute keys to use for this instrument. attributes = null histogram { # Default bucket boundaries for OpenTelemetry Histograms (Cinnamon Recorders). buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000] # Default settings for OpenTelemetry Histograms (Cinnamon Recorders) of durations. # Configures Cinnamon-recorded timing metrics, custom timers, and custom metrics with time-based units. durations { # Default bucket boundaries for timing metrics, as durations. buckets = [10us, 50us, 100us, 500us, 1ms, 5ms, 10ms, 25ms, 50ms, 75ms, 100ms, 250ms, 500ms, 750ms, 1s, 2.5s, 5s, 7.5s, 10s] # Convert duration-based metrics to this time unit. # Cinnamon timing metrics are recorded in nanoseconds. # Convert to seconds by default (the OpenTelemetry convention). conversion-unit = seconds } } }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.
Duration histogram buckets
If a Histogram backs a Recorder that records time durations, then the buckets can be specified using durations and automatically converted to a different base time unit (note that Cinnamon provided timers are recorded in nanoseconds and then converted to seconds by default):
- Example
-
cinnamon.opentelemetry { metrics.hints { millisecond-recorder { advice { histogram.durations { buckets = [1ms, 5ms, 10ms, 50ms, 100ms, 500ms, 1s, 5s, 10s] conversion-unit = milliseconds } } } } }
- Defaults
-
# Advice for instruments, using instrument extensions. advice { # Recommended set of attribute keys to use for this instrument. attributes = null histogram { # Default bucket boundaries for OpenTelemetry Histograms (Cinnamon Recorders). buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000] # Default settings for OpenTelemetry Histograms (Cinnamon Recorders) of durations. # Configures Cinnamon-recorded timing metrics, custom timers, and custom metrics with time-based units. durations { # Default bucket boundaries for timing metrics, as durations. buckets = [10us, 50us, 100us, 500us, 1ms, 5ms, 10ms, 25ms, 50ms, 75ms, 100ms, 250ms, 500ms, 750ms, 1s, 2.5s, 5s, 7.5s, 10s] # Convert duration-based metrics to this time unit. # Cinnamon timing metrics are recorded in nanoseconds. # Convert to seconds by default (the OpenTelemetry convention). conversion-unit = seconds } } }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.
If the config above is used with a hinted recorder, the recorded duration values will be converted to milliseconds and the histogram will use the configured buckets.
Metric views
OpenTelemetry metric Views can be configured as part of a metric hint or as defaults for all metrics.
The following example shows how to configure the “drop” aggregation, which means the metric will not be reported to your OpenTelemetry collector:
- Example
-
cinnamon.opentelemetry { metrics.hints { drop-metric { views += { aggregation = drop } } } }
- Defaults
-
# Default settings for OpenTelemetry views. view-defaults { selector { # Select instruments for this view with the given type. # Can be one of: `counter`, `up-down-counter`, `histogram`, `gauge`. # If not set (null) then instruments with any type are selected. type = null # Select instruments for this view with the given name. # May contain the wildcard characters `*` and `?` where: # - `*` matches 0 or more instances of any character # - `?` matches exactly one instance of any character # If not set (null) then instruments with any name are selected. name = null # Select instruments for this view with the given unit. # If not set (null) then instruments with any unit are selected. unit = null } # Set the name of the resulting metric. # Or `null` if the matched instrument name should be used. name = null # Set the description of the resulting metric. # Or `null` if the matched instrument description should be used. description = null # Specify the view aggregation, how instrument measurements are combined into metrics. # Can be one of: # - `drop`: drop all measurements and don't export any metrics. # - `sum`: aggregate measurements into a sum. # - `last-value`: record the last seen measurement as a gauge. # - `histogram`: aggregate measurements into an explicit bucket histogram. # - `exponential-histogram`: aggregate measurements into a base-2 exponential histogram. # - `summary`: aggregate measurements in dynamically calculated quantiles using sliding time windows. # - `default`: use the default aggregation for the instrument type. aggregation = default histogram { # Default bucket boundaries for a histogram view aggregation. buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000] } exponential-histogram { # Default max buckets for an exponential-histogram view aggregation. # Sets the max number of positive buckets and negative buckets (max total buckets is 2 * max-buckets + 1 zero bucket). max-buckets = 160 # Default max (and initial) scale for an exponential-histogram view aggregation. max-scale = 20 } summary { # Quantiles reported for a summary view aggregation. quantiles = [0.5, 0.95, 0.99] # Duration of the sliding time window (how long observations are kept) for a summary view aggregation. max-age = 5 minutes # Number of buckets in the sliding time window for a summary view aggregation. number-of-buckets = 5 # Specify the precision to use for a summary view aggregation. # This is the number of significant decimal digits for the underlying histogram. # Must be a non-negative integer between 0 and 5. precision = 2 } # Set a filter where only the listed attribute keys are retained. # Or `null` to not apply an attribute filter. attribute-filter = null # Set the cardinality limit, the maximum number of series for a metric. cardinality-limit = 2000 }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.
Exponential histograms
OpenTelemetry Histograms will be aggregated into explicit buckets by default. Exponential Histograms are an alternative representation which calculate bucket boundaries with an exponential growth function and automatically adjust for the scale and range of the recorded data.
Exponential Histograms can be configured individually for hinted metrics, or all histogram-based metrics can be configured to aggregate to exponential histograms with a default view:
- Example
-
cinnamon.opentelemetry { metrics.defaults { views += { selector.type = histogram aggregation = exponential-histogram } } }
- Defaults
-
# Default settings for OpenTelemetry views. view-defaults { selector { # Select instruments for this view with the given type. # Can be one of: `counter`, `up-down-counter`, `histogram`, `gauge`. # If not set (null) then instruments with any type are selected. type = null # Select instruments for this view with the given name. # May contain the wildcard characters `*` and `?` where: # - `*` matches 0 or more instances of any character # - `?` matches exactly one instance of any character # If not set (null) then instruments with any name are selected. name = null # Select instruments for this view with the given unit. # If not set (null) then instruments with any unit are selected. unit = null } # Set the name of the resulting metric. # Or `null` if the matched instrument name should be used. name = null # Set the description of the resulting metric. # Or `null` if the matched instrument description should be used. description = null # Specify the view aggregation, how instrument measurements are combined into metrics. # Can be one of: # - `drop`: drop all measurements and don't export any metrics. # - `sum`: aggregate measurements into a sum. # - `last-value`: record the last seen measurement as a gauge. # - `histogram`: aggregate measurements into an explicit bucket histogram. # - `exponential-histogram`: aggregate measurements into a base-2 exponential histogram. # - `summary`: aggregate measurements in dynamically calculated quantiles using sliding time windows. # - `default`: use the default aggregation for the instrument type. aggregation = default histogram { # Default bucket boundaries for a histogram view aggregation. buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000] } exponential-histogram { # Default max buckets for an exponential-histogram view aggregation. # Sets the max number of positive buckets and negative buckets (max total buckets is 2 * max-buckets + 1 zero bucket). max-buckets = 160 # Default max (and initial) scale for an exponential-histogram view aggregation. max-scale = 20 } summary { # Quantiles reported for a summary view aggregation. quantiles = [0.5, 0.95, 0.99] # Duration of the sliding time window (how long observations are kept) for a summary view aggregation. max-age = 5 minutes # Number of buckets in the sliding time window for a summary view aggregation. number-of-buckets = 5 # Specify the precision to use for a summary view aggregation. # This is the number of significant decimal digits for the underlying histogram. # Must be a non-negative integer between 0 and 5. precision = 2 } # Set a filter where only the listed attribute keys are retained. # Or `null` to not apply an attribute filter. attribute-filter = null # Set the cardinality limit, the maximum number of series for a metric. cardinality-limit = 2000 }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.
Before configuring exponential histograms, make sure that these are supported by the telemetry backend being used.
For Prometheus, exponential histograms (native histograms) need to be enabled with a feature flag.
The OpenTelemetry developer sandbox is already configured to support exponential histograms.
Summary aggregation
OpenTelemetry Histograms will be aggregated as a bucketed histogram by default. Cinnamon supports a Summary aggregation to instead directly report quantiles for the recorded distribution, similar to the Prometheus Summary metric type.
The following example configures a summary aggregation in place of the default histogram aggregation and specifies which quantiles should be reported:
- Example
-
cinnamon.opentelemetry { metrics.hints { summary-metric { views += { aggregation = summary summary.quantiles = [0.5, 0.75, 0.90, 0.95, 0.99] } } } }
- Defaults
-
# Default settings for OpenTelemetry views. view-defaults { selector { # Select instruments for this view with the given type. # Can be one of: `counter`, `up-down-counter`, `histogram`, `gauge`. # If not set (null) then instruments with any type are selected. type = null # Select instruments for this view with the given name. # May contain the wildcard characters `*` and `?` where: # - `*` matches 0 or more instances of any character # - `?` matches exactly one instance of any character # If not set (null) then instruments with any name are selected. name = null # Select instruments for this view with the given unit. # If not set (null) then instruments with any unit are selected. unit = null } # Set the name of the resulting metric. # Or `null` if the matched instrument name should be used. name = null # Set the description of the resulting metric. # Or `null` if the matched instrument description should be used. description = null # Specify the view aggregation, how instrument measurements are combined into metrics. # Can be one of: # - `drop`: drop all measurements and don't export any metrics. # - `sum`: aggregate measurements into a sum. # - `last-value`: record the last seen measurement as a gauge. # - `histogram`: aggregate measurements into an explicit bucket histogram. # - `exponential-histogram`: aggregate measurements into a base-2 exponential histogram. # - `summary`: aggregate measurements in dynamically calculated quantiles using sliding time windows. # - `default`: use the default aggregation for the instrument type. aggregation = default histogram { # Default bucket boundaries for a histogram view aggregation. buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000] } exponential-histogram { # Default max buckets for an exponential-histogram view aggregation. # Sets the max number of positive buckets and negative buckets (max total buckets is 2 * max-buckets + 1 zero bucket). max-buckets = 160 # Default max (and initial) scale for an exponential-histogram view aggregation. max-scale = 20 } summary { # Quantiles reported for a summary view aggregation. quantiles = [0.5, 0.95, 0.99] # Duration of the sliding time window (how long observations are kept) for a summary view aggregation. max-age = 5 minutes # Number of buckets in the sliding time window for a summary view aggregation. number-of-buckets = 5 # Specify the precision to use for a summary view aggregation. # This is the number of significant decimal digits for the underlying histogram. # Must be a non-negative integer between 0 and 5. precision = 2 } # Set a filter where only the listed attribute keys are retained. # Or `null` to not apply an attribute filter. attribute-filter = null # Set the cardinality limit, the maximum number of series for a metric. cardinality-limit = 2000 }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.
View selector
As seen in the reference configuration for a view, view selectors can be used to select particular metrics based on type
, name
or unit
.
This example configures a view for a particular recorder based on its type (histogram) and a wildcard match on its name:
- Example
-
cinnamon.opentelemetry { metrics.hints { recorder-sum { views += { # select metric by type and last part of name selector { type = histogram name = "*.some-recorder" } # rename metric name = "some-recorder-sum" # add a description description = "histogram aggregated to sum" # use sum aggregation (rather than default histogram aggregation) aggregation = sum } } } }
- Defaults
-
# Default settings for OpenTelemetry views. view-defaults { selector { # Select instruments for this view with the given type. # Can be one of: `counter`, `up-down-counter`, `histogram`, `gauge`. # If not set (null) then instruments with any type are selected. type = null # Select instruments for this view with the given name. # May contain the wildcard characters `*` and `?` where: # - `*` matches 0 or more instances of any character # - `?` matches exactly one instance of any character # If not set (null) then instruments with any name are selected. name = null # Select instruments for this view with the given unit. # If not set (null) then instruments with any unit are selected. unit = null } # Set the name of the resulting metric. # Or `null` if the matched instrument name should be used. name = null # Set the description of the resulting metric. # Or `null` if the matched instrument description should be used. description = null # Specify the view aggregation, how instrument measurements are combined into metrics. # Can be one of: # - `drop`: drop all measurements and don't export any metrics. # - `sum`: aggregate measurements into a sum. # - `last-value`: record the last seen measurement as a gauge. # - `histogram`: aggregate measurements into an explicit bucket histogram. # - `exponential-histogram`: aggregate measurements into a base-2 exponential histogram. # - `summary`: aggregate measurements in dynamically calculated quantiles using sliding time windows. # - `default`: use the default aggregation for the instrument type. aggregation = default histogram { # Default bucket boundaries for a histogram view aggregation. buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000] } exponential-histogram { # Default max buckets for an exponential-histogram view aggregation. # Sets the max number of positive buckets and negative buckets (max total buckets is 2 * max-buckets + 1 zero bucket). max-buckets = 160 # Default max (and initial) scale for an exponential-histogram view aggregation. max-scale = 20 } summary { # Quantiles reported for a summary view aggregation. quantiles = [0.5, 0.95, 0.99] # Duration of the sliding time window (how long observations are kept) for a summary view aggregation. max-age = 5 minutes # Number of buckets in the sliding time window for a summary view aggregation. number-of-buckets = 5 # Specify the precision to use for a summary view aggregation. # This is the number of significant decimal digits for the underlying histogram. # Must be a non-negative integer between 0 and 5. precision = 2 } # Set a filter where only the listed attribute keys are retained. # Or `null` to not apply an attribute filter. attribute-filter = null # Set the cardinality limit, the maximum number of series for a metric. cardinality-limit = 2000 }
NoteThese settings are defined in the
reference.conf
. You only need to specify any of these settings when you want to override the defaults.