OpenTelemetry - Metrics

You can export Cinnamon metrics to OpenTelemetry.

OpenTelemetry dependency

First make sure that your build is configured to use the Cinnamon Agent and has instrumentations enabled, such as Akka instrumentation or Akka HTTP instrumentation.

sbt

copylibraryDependencies += Cinnamon.library.cinnamonOpenTelemetry

Maven

copy<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-opentelemetry</artifactId>
    <version>2.20.4</version>
</dependency>

Gradle

copydependencies {
    implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-opentelemetry', version: '2.20.4'
}

Metric exporters

There are two options for pushing metrics to an OpenTelemetry collector via the OpenTelemetry wire protocol (OTLP) using either gRPC or HTTP.

Furthermore, a logging exporter can be configured to print metrics to the console.

gRPC metric exporter

To use the gRPC exporter you need to provide the following configuration:

Required

copycinnamon.opentelemetry.metrics {
  exporters += grpc-exporter
}

Example

copycinnamon.opentelemetry.metrics {
  exporters += grpc-exporter
  grpc-exporter {
    interval = 15 seconds
  }
}

Reference

copycinnamon.opentelemetry.metrics {
  grpc-exporter {

    # Define the interval of reads.
    interval = 1 minute

    # Sets the OTLP endpoint to connect to. The endpoint must start with either http:// or https://.
    endpoint = "http://localhost:4317"

    # The maximum time to wait for the collector to process an exported batch of metrics.
    timeout = 10 seconds

    # The maximum time to wait for new connections to be established.
    connect-timeout = 10 seconds

    # The method used to compress payloads. If "none", compression is disabled. Currently supported compression methods include "gzip" and "none".
    compression = "none"

    # Add header to request. Optional.
    headers {
      # some-key = "some value"
    }

    # Set the retry policy. Retry is disabled by default.
    retry-policy = "none"

    # Enabled when `retry-policy = "default-retry-policy"`. Configured to use the defaults.
    default-retry-policy {
      # Set the maximum number of attempts, including the original request. Must be greater than 1 and less than 6.
      max-attempts = 5

      # Set the initial backoff. Must be greater than 0.
      initial-backoff = 1 second

      # Set the maximum backoff. Must be greater than 0.
      max-backoff = 5 seconds

      # Set the backoff multiplier. Must be greater than 0.0.
      backoff-multiplier = 1.5
    }

    # Set the aggregation temporality selector.
    # Pre-defined selectors are "always-cumulative" or "delta-preferred".
    # A custom selector can be defined with config in the following format:
    #
    #     aggregation-temporality-selector = some-custom-selector
    #
    #     # a custom selector that is equivalent to `delta-preferred`
    #     # custom selector section can be named appropriately
    #     some-custom-selector {
    #       # specify the default, or otherwise defaults to cumulative
    #       default = delta
    #
    #       # select aggregation temporality for particular instrument types
    #       select {
    #         up-down-counter = cumulative
    #         observable-up-down-counter = cumulative
    #       }
    #     }
    #
    aggregation-temporality-selector = always-cumulative
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

HTTP metric exporter

To use the HTTP exporter you need to provide the following configuration:

Required

copycinnamon.opentelemetry.metrics {
  exporters += http-exporter
}

Example

copycinnamon.opentelemetry.metrics {
  exporters += http-exporter
  http-exporter {
    interval = 15 seconds
  }
}

Reference

copycinnamon.opentelemetry.metrics {
  http-exporter {

    # Define the interval of reads
    interval = 1 minute

    # Sets the OTLP endpoint to connect to. The endpoint must start with either http:// or https://, and include the full HTTP path.
    endpoint = "http://localhost:4318/v1/metrics"

    # Sets the maximum time to wait for the collector to process an exported batch of metrics.
    timeout = 10 seconds

    # The maximum time to wait for new connections to be established.
    connect-timeout = 10 seconds

    # Sets the method used to compress payloads. If "none", compression is disabled. Currently supported compression methods include "gzip" and "none".
    compression = "none"

    # Add header to request. Optional.
    headers {
      # some-key = "some value"
    }

    # Set the retry policy. Retry is disabled by default.
    retry-policy = "none"

    # Enabled when `retry-policy = "default-retry-policy"`. Configured to use the defaults.
    default-retry-policy {
      # Set the maximum number of attempts, including the original request. Must be greater than 1 and less than 6.
      max-attempts = 5

      # Set the initial backoff. Must be greater than 0.
      initial-backoff = 1 second

      # Set the maximum backoff. Must be greater than 0.
      max-backoff = 5 seconds

      # Set the backoff multiplier. Must be greater than 0.0.
      backoff-multiplier = 1.5
    }

    # Set the aggregation temporality selector.
    # Pre-defined selectors are "always-cumulative" or "delta-preferred".
    # A custom selector can be defined with config in the following format:
    #
    #     aggregation-temporality-selector = some-custom-selector
    #
    #     # a custom selector that is equivalent to `delta-preferred`
    #     # custom selector section can be named appropriately
    #     some-custom-selector {
    #       # specify the default, or otherwise defaults to cumulative
    #       default = delta
    #
    #       # select aggregation temporality for particular instrument types
    #       select {
    #         up-down-counter = cumulative
    #         observable-up-down-counter = cumulative
    #       }
    #     }
    #
    aggregation-temporality-selector = always-cumulative
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Logging metric exporter

To use the logging exporter you need to provide the following configuration:

Required

copycinnamon.opentelemetry.metrics {
  exporters += logging-exporter
}

Example

copycinnamon.opentelemetry.metrics {
  exporters += logging-exporter
  logging-exporter {
    interval = 15 seconds
  }
}

Reference

copycinnamon.opentelemetry.metrics {
  logging-exporter {

    # Define the interval of reads.
    interval = 1 minute

    # Set the aggregation temporality: `cumulative` or `delta`
    aggregation-temporality = cumulative
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Custom metrics

Custom metrics are mapped to the following OpenTelemetry metrics:

Resource Attributes

To attach a custom resource attribute to every metric, you can use the following config:

Example

copycinnamon.opentelemetry {
  # Based on https://opentelemetry.io/docs/specs/semconv/resource/
  resource-attributes {
    cloud.provider = "aws"
    cloud.region = "us-east-1"
    cloud.region = ${?AWS_REGION}
  }
}

Metric hints

Metrics—either created by Cinnamon instrumentation or custom metrics—can be given hints that will passed to the metric backend. These hints allow particular metrics to be configured differently from the default configuration.

For example, using the custom metrics API you can pass in the hint when creating a custom recorder:

Scala

copyval recorder = CinnamonMetrics(context).createRecorder("recorder", hints = Set("custom-recorder"))

Java

copyRecorder recorder =
    CinnamonMetrics.get(context).createRecorder("recorder", ImmutableSet.of("custom-recorder"));

Metric advice

OpenTelemetry metrics can be configured by providing “advice” to the OpenTelemetry SDK. Advice can be applied to all metrics under configuration for cinnamon.opentelemetry.metrics.defaults, or applied to particular metrics that have metric hints.

Metric attributes

Providing metric advice for attributes acts as an attribute filter. Given the following custom metric:

Scala

copyval counter = CinnamonMetrics(context).createCounter(
  "some-counter",
  tags = Map("some-attribute" -> "some-value", "other-attribute" -> "other-value"),
  hints = Set("filter-attributes")
)

Java

copyImmutableMap<String, String> tags =
    ImmutableMap.of(
        "some-attribute", "some-value",
        "other-attribute", "other-value");
ImmutableSet<String> hints = ImmutableSet.of("filter-attributes");
Counter counter = CinnamonMetrics.get(context).createCounter("some-counter", tags, hints);

An attribute filter can be configured for this metric hint using the attributes advice setting:

Example

copycinnamon.opentelemetry {
  metrics.hints {
    filter-attributes {
      advice {
        attributes = ["some-attribute"]
      }
    }
  }
}

In this example, only the some-attribute attribute will be exported for metrics with the configured metric hint. Any other attributes, including other-attribute, will not be exported.

Histogram buckets

A Recorder in Cinnamon is backed by an OpenTelemetry LongHistogram. The buckets for a histogram can be configured for a hinted metric as follows:

Example

copycinnamon.opentelemetry {
  metrics.hints {
    recorded-buckets {
      advice {
        histogram.buckets = [1, 2, 4, 8, 16, 32, 64, 128]
      }
    }
  }
}

Defaults

copy# Advice for instruments, using instrument extensions.
advice {
  # Recommended set of attribute keys to use for this instrument.
  attributes = null

  histogram {
    # Default bucket boundaries for OpenTelemetry Histograms (Cinnamon Recorders).
    buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]

    # Default settings for OpenTelemetry Histograms (Cinnamon Recorders) of durations.
    # Configures Cinnamon-recorded timing metrics, custom timers, and custom metrics with time-based units.
    durations {
      # Default bucket boundaries for timing metrics, as durations.
      buckets = [10us, 50us, 100us, 500us, 1ms, 5ms, 10ms, 25ms, 50ms, 75ms, 100ms, 250ms, 500ms, 750ms, 1s, 2.5s, 5s, 7.5s, 10s]

      # Convert duration-based metrics to this time unit.
      # Cinnamon timing metrics are recorded in nanoseconds.
      # Convert to seconds by default (the OpenTelemetry convention).
      conversion-unit = seconds
    }
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Duration histogram buckets

If a Histogram backs a Recorder that records time durations, then the buckets can be specified using durations and automatically converted to a different base time unit (note that Cinnamon provided timers are recorded in nanoseconds and then converted to seconds by default):

Example

copycinnamon.opentelemetry {
  metrics.hints {
    millisecond-recorder {
      advice {
        histogram.durations {
          buckets = [1ms, 5ms, 10ms, 50ms, 100ms, 500ms, 1s, 5s, 10s]
          conversion-unit = milliseconds
        }
      }
    }
  }
}

Defaults

copy# Advice for instruments, using instrument extensions.
advice {
  # Recommended set of attribute keys to use for this instrument.
  attributes = null

  histogram {
    # Default bucket boundaries for OpenTelemetry Histograms (Cinnamon Recorders).
    buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]

    # Default settings for OpenTelemetry Histograms (Cinnamon Recorders) of durations.
    # Configures Cinnamon-recorded timing metrics, custom timers, and custom metrics with time-based units.
    durations {
      # Default bucket boundaries for timing metrics, as durations.
      buckets = [10us, 50us, 100us, 500us, 1ms, 5ms, 10ms, 25ms, 50ms, 75ms, 100ms, 250ms, 500ms, 750ms, 1s, 2.5s, 5s, 7.5s, 10s]

      # Convert duration-based metrics to this time unit.
      # Cinnamon timing metrics are recorded in nanoseconds.
      # Convert to seconds by default (the OpenTelemetry convention).
      conversion-unit = seconds
    }
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

If the config above is used with a hinted recorder, the recorded duration values will be converted to milliseconds and the histogram will use the configured buckets.

Metric views

OpenTelemetry metric Views can be configured as part of a metric hint or as defaults for all metrics.

The following example shows how to configure the “drop” aggregation, which means the metric will not be reported to your OpenTelemetry collector:

Example

copycinnamon.opentelemetry {
  metrics.hints {
    drop-metric {
      views += {
        aggregation = drop
      }
    }
  }
}

Defaults

copy# Default settings for OpenTelemetry views.
view-defaults {
  selector {
    # Select instruments for this view with the given type.
    # Can be one of: `counter`, `up-down-counter`, `histogram`, `gauge`.
    # If not set (null) then instruments with any type are selected.
    type = null

    # Select instruments for this view with the given name.
    # May contain the wildcard characters `*` and `?` where:
    #   - `*` matches 0 or more instances of any character
    #   - `?` matches exactly one instance of any character
    # If not set (null) then instruments with any name are selected.
    name = null

    # Select instruments for this view with the given unit.
    # If not set (null) then instruments with any unit are selected.
    unit = null
  }

  # Set the name of the resulting metric.
  # Or `null` if the matched instrument name should be used.
  name = null

  # Set the description of the resulting metric.
  # Or `null` if the matched instrument description should be used.
  description = null

  # Specify the view aggregation, how instrument measurements are combined into metrics.
  # Can be one of:
  #   - `drop`: drop all measurements and don't export any metrics.
  #   - `sum`: aggregate measurements into a sum.
  #   - `last-value`: record the last seen measurement as a gauge.
  #   - `histogram`: aggregate measurements into an explicit bucket histogram.
  #   - `exponential-histogram`: aggregate measurements into a base-2 exponential histogram.
  #   - `summary`: aggregate measurements in dynamically calculated quantiles using sliding time windows.
  #   - `default`: use the default aggregation for the instrument type.
  aggregation = default

  histogram {
    # Default bucket boundaries for a histogram view aggregation.
    buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]
  }

  exponential-histogram {
    # Default max buckets for an exponential-histogram view aggregation.
    # Sets the max number of positive buckets and negative buckets (max total buckets is 2 * max-buckets + 1 zero bucket).
    max-buckets = 160

    # Default max (and initial) scale for an exponential-histogram view aggregation.
    max-scale = 20
  }

  summary {
    # Quantiles reported for a summary view aggregation.
    quantiles = [0.5, 0.95, 0.99]

    # Duration of the sliding time window (how long observations are kept) for a summary view aggregation.
    max-age = 5 minutes

    # Number of buckets in the sliding time window for a summary view aggregation.
    number-of-buckets = 5

    # Specify the precision to use for a summary view aggregation.
    # This is the number of significant decimal digits for the underlying histogram.
    # Must be a non-negative integer between 0 and 5.
    precision = 2
  }

  # Set a filter where only the listed attribute keys are retained.
  # Or `null` to not apply an attribute filter.
  attribute-filter = null

  # Set the cardinality limit, the maximum number of series for a metric.
  cardinality-limit = 2000
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Exponential histograms

OpenTelemetry Histograms will be aggregated into explicit buckets by default. Exponential Histograms are an alternative representation which calculate bucket boundaries with an exponential growth function and automatically adjust for the scale and range of the recorded data.

Exponential Histograms can be configured individually for hinted metrics, or all histogram-based metrics can be configured to aggregate to exponential histograms with a default view:

Example

copycinnamon.opentelemetry {
  metrics.defaults {
    views += {
      selector.type = histogram
      aggregation = exponential-histogram
    }
  }
}

Defaults

copy# Default settings for OpenTelemetry views.
view-defaults {
  selector {
    # Select instruments for this view with the given type.
    # Can be one of: `counter`, `up-down-counter`, `histogram`, `gauge`.
    # If not set (null) then instruments with any type are selected.
    type = null

    # Select instruments for this view with the given name.
    # May contain the wildcard characters `*` and `?` where:
    #   - `*` matches 0 or more instances of any character
    #   - `?` matches exactly one instance of any character
    # If not set (null) then instruments with any name are selected.
    name = null

    # Select instruments for this view with the given unit.
    # If not set (null) then instruments with any unit are selected.
    unit = null
  }

  # Set the name of the resulting metric.
  # Or `null` if the matched instrument name should be used.
  name = null

  # Set the description of the resulting metric.
  # Or `null` if the matched instrument description should be used.
  description = null

  # Specify the view aggregation, how instrument measurements are combined into metrics.
  # Can be one of:
  #   - `drop`: drop all measurements and don't export any metrics.
  #   - `sum`: aggregate measurements into a sum.
  #   - `last-value`: record the last seen measurement as a gauge.
  #   - `histogram`: aggregate measurements into an explicit bucket histogram.
  #   - `exponential-histogram`: aggregate measurements into a base-2 exponential histogram.
  #   - `summary`: aggregate measurements in dynamically calculated quantiles using sliding time windows.
  #   - `default`: use the default aggregation for the instrument type.
  aggregation = default

  histogram {
    # Default bucket boundaries for a histogram view aggregation.
    buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]
  }

  exponential-histogram {
    # Default max buckets for an exponential-histogram view aggregation.
    # Sets the max number of positive buckets and negative buckets (max total buckets is 2 * max-buckets + 1 zero bucket).
    max-buckets = 160

    # Default max (and initial) scale for an exponential-histogram view aggregation.
    max-scale = 20
  }

  summary {
    # Quantiles reported for a summary view aggregation.
    quantiles = [0.5, 0.95, 0.99]

    # Duration of the sliding time window (how long observations are kept) for a summary view aggregation.
    max-age = 5 minutes

    # Number of buckets in the sliding time window for a summary view aggregation.
    number-of-buckets = 5

    # Specify the precision to use for a summary view aggregation.
    # This is the number of significant decimal digits for the underlying histogram.
    # Must be a non-negative integer between 0 and 5.
    precision = 2
  }

  # Set a filter where only the listed attribute keys are retained.
  # Or `null` to not apply an attribute filter.
  attribute-filter = null

  # Set the cardinality limit, the maximum number of series for a metric.
  cardinality-limit = 2000
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Summary aggregation

OpenTelemetry Histograms will be aggregated as a bucketed histogram by default. Cinnamon supports a Summary aggregation to instead directly report quantiles for the recorded distribution, similar to the Prometheus Summary metric type.

The following example configures a summary aggregation in place of the default histogram aggregation and specifies which quantiles should be reported:

Example

copycinnamon.opentelemetry {
  metrics.hints {
    summary-metric {
      views += {
        aggregation = summary
        summary.quantiles = [0.5, 0.75, 0.90, 0.95, 0.99]
      }
    }
  }
}

Defaults

copy# Default settings for OpenTelemetry views.
view-defaults {
  selector {
    # Select instruments for this view with the given type.
    # Can be one of: `counter`, `up-down-counter`, `histogram`, `gauge`.
    # If not set (null) then instruments with any type are selected.
    type = null

    # Select instruments for this view with the given name.
    # May contain the wildcard characters `*` and `?` where:
    #   - `*` matches 0 or more instances of any character
    #   - `?` matches exactly one instance of any character
    # If not set (null) then instruments with any name are selected.
    name = null

    # Select instruments for this view with the given unit.
    # If not set (null) then instruments with any unit are selected.
    unit = null
  }

  # Set the name of the resulting metric.
  # Or `null` if the matched instrument name should be used.
  name = null

  # Set the description of the resulting metric.
  # Or `null` if the matched instrument description should be used.
  description = null

  # Specify the view aggregation, how instrument measurements are combined into metrics.
  # Can be one of:
  #   - `drop`: drop all measurements and don't export any metrics.
  #   - `sum`: aggregate measurements into a sum.
  #   - `last-value`: record the last seen measurement as a gauge.
  #   - `histogram`: aggregate measurements into an explicit bucket histogram.
  #   - `exponential-histogram`: aggregate measurements into a base-2 exponential histogram.
  #   - `summary`: aggregate measurements in dynamically calculated quantiles using sliding time windows.
  #   - `default`: use the default aggregation for the instrument type.
  aggregation = default

  histogram {
    # Default bucket boundaries for a histogram view aggregation.
    buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]
  }

  exponential-histogram {
    # Default max buckets for an exponential-histogram view aggregation.
    # Sets the max number of positive buckets and negative buckets (max total buckets is 2 * max-buckets + 1 zero bucket).
    max-buckets = 160

    # Default max (and initial) scale for an exponential-histogram view aggregation.
    max-scale = 20
  }

  summary {
    # Quantiles reported for a summary view aggregation.
    quantiles = [0.5, 0.95, 0.99]

    # Duration of the sliding time window (how long observations are kept) for a summary view aggregation.
    max-age = 5 minutes

    # Number of buckets in the sliding time window for a summary view aggregation.
    number-of-buckets = 5

    # Specify the precision to use for a summary view aggregation.
    # This is the number of significant decimal digits for the underlying histogram.
    # Must be a non-negative integer between 0 and 5.
    precision = 2
  }

  # Set a filter where only the listed attribute keys are retained.
  # Or `null` to not apply an attribute filter.
  attribute-filter = null

  # Set the cardinality limit, the maximum number of series for a metric.
  cardinality-limit = 2000
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

View selector

As seen in the reference configuration for a view, view selectors can be used to select particular metrics based on type, name or unit.

This example configures a view for a particular recorder based on its type (histogram) and a wildcard match on its name:

Example

copycinnamon.opentelemetry {
  metrics.hints {
    recorder-sum {
      views += {
        # select metric by type and last part of name
        selector {
          type = histogram
          name = "*.some-recorder"
        }
        # rename metric
        name = "some-recorder-sum"
        # add a description
        description = "histogram aggregated to sum"
        # use sum aggregation (rather than default histogram aggregation)
        aggregation = sum
      }
    }
  }
}

Defaults

copy# Default settings for OpenTelemetry views.
view-defaults {
  selector {
    # Select instruments for this view with the given type.
    # Can be one of: `counter`, `up-down-counter`, `histogram`, `gauge`.
    # If not set (null) then instruments with any type are selected.
    type = null

    # Select instruments for this view with the given name.
    # May contain the wildcard characters `*` and `?` where:
    #   - `*` matches 0 or more instances of any character
    #   - `?` matches exactly one instance of any character
    # If not set (null) then instruments with any name are selected.
    name = null

    # Select instruments for this view with the given unit.
    # If not set (null) then instruments with any unit are selected.
    unit = null
  }

  # Set the name of the resulting metric.
  # Or `null` if the matched instrument name should be used.
  name = null

  # Set the description of the resulting metric.
  # Or `null` if the matched instrument description should be used.
  description = null

  # Specify the view aggregation, how instrument measurements are combined into metrics.
  # Can be one of:
  #   - `drop`: drop all measurements and don't export any metrics.
  #   - `sum`: aggregate measurements into a sum.
  #   - `last-value`: record the last seen measurement as a gauge.
  #   - `histogram`: aggregate measurements into an explicit bucket histogram.
  #   - `exponential-histogram`: aggregate measurements into a base-2 exponential histogram.
  #   - `summary`: aggregate measurements in dynamically calculated quantiles using sliding time windows.
  #   - `default`: use the default aggregation for the instrument type.
  aggregation = default

  histogram {
    # Default bucket boundaries for a histogram view aggregation.
    buckets = [5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]
  }

  exponential-histogram {
    # Default max buckets for an exponential-histogram view aggregation.
    # Sets the max number of positive buckets and negative buckets (max total buckets is 2 * max-buckets + 1 zero bucket).
    max-buckets = 160

    # Default max (and initial) scale for an exponential-histogram view aggregation.
    max-scale = 20
  }

  summary {
    # Quantiles reported for a summary view aggregation.
    quantiles = [0.5, 0.95, 0.99]

    # Duration of the sliding time window (how long observations are kept) for a summary view aggregation.
    max-age = 5 minutes

    # Number of buckets in the sliding time window for a summary view aggregation.
    number-of-buckets = 5

    # Specify the precision to use for a summary view aggregation.
    # This is the number of significant decimal digits for the underlying histogram.
    # Must be a non-negative integer between 0 and 5.
    precision = 2
  }

  # Set a filter where only the listed attribute keys are retained.
  # Or `null` to not apply an attribute filter.
  attribute-filter = null

  # Set the cardinality limit, the maximum number of series for a metric.
  cardinality-limit = 2000
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.