OpenTelemetry - Logs

You can export application logs to OpenTelemetry as OpenTelemetry log records by configuring a log record exporter and a log appender.

OpenTelemetry dependency

First make sure that your build is configured to use the Cinnamon Agent and has instrumentations enabled, such as Akka instrumentation or Akka HTTP instrumentation.

sbt

copylibraryDependencies += Cinnamon.library.cinnamonOpenTelemetry

Maven

copy<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-opentelemetry</artifactId>
    <version>2.21.2</version>
</dependency>

Gradle

copydependencies {
    implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-opentelemetry', version: '2.21.2'
}

Log Record Exporter

There are two log record exporters available to export application logs to an OpenTelemetry collector via the OpenTelemetry wire protocol (OTLP) using either gRPC or HTTP.

For debugging, a system out log record exporter can be configured to print log records to standard out.

gRPC Log Record Exporter

To use the gRPC exporter to export logs to OpenTelemetry you need to provide the following configuration:

Required

copycinnamon.opentelemetry.logs {
  exporters += grpc-exporter
}

Example

copycinnamon.opentelemetry.logs {
  exporters += grpc-exporter
  grpc-exporter {
    max-queue-size = 4096
    max-export-batch-size = 1024
  }
}

Reference

copycinnamon.opentelemetry.logs {
  grpc-exporter {

    # The delay interval between two consecutive exports.
    schedule-delay = 1 second

    # The maximum time an export will be allowed to run before being cancelled.
    exporter-timeout = 30 seconds

    # The maximum number of logs that are kept in the queue before starting to drop.
    max-queue-size = 2048

    # The maximum batch size for every export. This must be smaller or equal to `max-queue-size`.
    max-export-batch-size = 512

    # Sets the OTLP endpoint to connect to. The endpoint must start with either http:// or https://.
    endpoint = "http://localhost:4317"

    # The maximum time to wait for the collector to process an exported batch of logs.
    timeout = 10 seconds

    # The maximum time to wait for new connections to be established.
    connect-timeout = 10 seconds

    # The method used to compress payloads. If "none", compression is disabled. Currently supported compression methods include "gzip" and "none".
    compression = "none"

    # Add header to request. Optional.
    headers {
      # some-key = "some value"
    }

    # Set the retry policy. Retry is disabled by default.
    retry-policy = "none"

    # Enabled when `retry-policy = "default-retry-policy"`. Configured to use the defaults.
    default-retry-policy {
      # Set the maximum number of attempts, including the original request. Must be greater than 1 and less than 6.
      max-attempts = 5

      # Set the initial backoff. Must be greater than 0.
      initial-backoff = 1 second

      # Set the maximum backoff. Must be greater than 0.
      max-backoff = 5 seconds

      # Set the backoff multiplier. Must be greater than 0.0.
      backoff-multiplier = 1.5
    }
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

HTTP Log Record Exporter

To use the HTTP exporter to export logs to OpenTelemetry you need to provide the following configuration:

Required

copycinnamon.opentelemetry.logs {
  exporters += http-exporter
}

Example

copycinnamon.opentelemetry.logs {
  exporters += http-exporter
  http-exporter {
    max-queue-size = 4096
    max-export-batch-size = 1024
  }
}

Reference

copycinnamon.opentelemetry.logs {
  http-exporter {

    # The delay interval between two consecutive exports.
    schedule-delay = 1 second

    # The maximum time an export will be allowed to run before being cancelled.
    exporter-timeout = 30 seconds

    # The maximum number of logs that are kept in the queue before starting to drop.
    max-queue-size = 2048

    # The maximum batch size for every export. This must be smaller or equal to `max-queue-size`.
    max-export-batch-size = 512

    # Sets the OTLP endpoint to connect to. The endpoint must start with either http:// or https://, and include the full HTTP path.
    endpoint = "http://localhost:4318/v1/logs"

    # Sets the maximum time to wait for the collector to process an exported batch of logs.
    timeout = 10 seconds

    # The maximum time to wait for new connections to be established.
    connect-timeout = 10 seconds

    # Sets the method used to compress payloads. If "none", compression is disabled. Currently supported compression methods include "gzip" and "none".
    compression = "none"

    # Add header to request. Optional.
    headers {
      # some-key = "some value"
    }

    # Set the retry policy. Retry is disabled by default.
    retry-policy = "none"

    # Enabled when `retry-policy = "default-retry-policy"`. Configured to use the defaults.
    default-retry-policy {
      # Set the maximum number of attempts, including the original request. Must be greater than 1 and less than 6.
      max-attempts = 5

      # Set the initial backoff. Must be greater than 0.
      initial-backoff = 1 second

      # Set the maximum backoff. Must be greater than 0.
      max-backoff = 5 seconds

      # Set the backoff multiplier. Must be greater than 0.0.
      backoff-multiplier = 1.5
    }
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

System out log record exporter

To use the system out exporter you need to provide the following configuration:

Required

copycinnamon.opentelemetry.logs {
  exporters += system-out-exporter
}

Log Appenders

After configuring the log record exporter you need to configure your logging setup.

You need to provide an explicit dependency to your logging framework (logback or log4j).

The OpenTelemetry developer sandbox is already prepared to receive and visualize logs.

Logback Appender

Use the logback appender to forward Logback log records to OpenTelemetry.

Add the following module to your build:

sbt

copylibraryDependencies += Cinnamon.library.cinnamonOpenTelemetryLogbackAppender

Maven

copy<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-opentelemetry-logback-appender</artifactId>
    <version>2.21.2</version>
</dependency>

Gradle

copydependencies {
    implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-opentelemetry-logback-appender', version: '2.21.2'
}

Add a new <appender> to your logback.xml file. A basic example looks like this:

logback.xml

copy<configuration>
    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} ---- %msg%n</pattern>
        </encoder>
    </appender>
    <appender name="OpenTelemetry" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
    </appender>
    <root level="DEBUG">
        <appender-ref ref="console"/>
        <appender-ref ref="OpenTelemetry"/>
    </root>
</configuration>

Log4j Appender

Use the log4j appender to forward Log4j2 log records to OpenTelemetry.

Add the following module to your build:

sbt

copylibraryDependencies += Cinnamon.library.cinnamonOpenTelemetryLog4jAppender

Maven

copy<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-opentelemetry-log4j-appender</artifactId>
    <version>2.21.2</version>
</dependency>

Gradle

copydependencies {
    implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-opentelemetry-log4j-appender', version: '2.21.2'
}

Add a new <OpenTelemetry /> item to the <Appenders> section in your log4j2.xml file. A basic example looks like this:

log4j2.xml

copy<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.appender.v2_17">
    <Appenders>
        <Console name="ConsoleAppender" target="SYSTEM_OUT" follow="true">
            <PatternLayout pattern="log4j2: %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <OpenTelemetry name="OpenTelemetryAppender" captureMapMessageAttributes="true" captureExperimentalAttributes="true"/>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="OpenTelemetryAppender" />
            <AppenderRef ref="ConsoleAppender" />
        </Root>
    </Loggers>
</Configuration>