Predefined JMX importers

Lightbend Telemetry comes with some predefined importers to provide for easy import from JMX.

Apache Kafka importers

Lightbend Telemetry comes with two importers for Apache Kafka to generate metrics for Kafka consumers and producers.

Apache Kafka importer dependency

Add the desired Kafka JMX Importer dependencies to your build:

sbt

copy// Consumer metric generation
libraryDependencies += Cinnamon.library.cinnamonKafkaConsumerJmxImporter

// Producer metric generation
libraryDependencies += Cinnamon.library.cinnamonKafkaProducerJmxImporter

Maven

copy<!-- Generate consumer metrics -->
<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-jmx-importer-kafka-consumer</artifactId>
    <version>2.20.4</version>
</dependency>
<!-- Generate producer metrics -->
<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-jmx-importer-kafka-producer</artifactId>
    <version>2.20.4</version>
</dependency>

Gradle

copydependencies {
    // generate consumer metrics
    implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-jmx-importer-kafka-consumer', version: '2.20.4'

    // generate producer metrics
    implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-jmx-importer-kafka-producer', version: '2.20.4'
}

Apache Kafka importer configuration

Required

No additional configuration is required for the Kafka JMX importers to import the MBeans.

Reference (Consumer)

copy# Fetch Metrics
query = "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*"
attributes = [
  {
    attribute = records-lag-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = fetch-size-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = fetch-size-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = bytes-consumed-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = records-per-request-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = records-consumed-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = fetch-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = fetch-latency-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = fetch-latency-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = fetch-throttle-time-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = fetch-throttle-time-max
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  }
]
# Topic-level Fetch Metrics
query = "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*"
attributes = [
  {
    attribute = fetch-size-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = fetch-size-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = bytes-consumed-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = records-per-request-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = records-consumed-rate
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  },
  {
    category = "topics"
    key = "topic"
  }
]
# Consumer Group Metrics
query = "kafka.consumer:type=consumer-coordinator-metrics,client-id=*"
attributes = [
  {
    attribute = assigned-partitions
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = commit-latency-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = commit-latency-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = commit-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = join-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = join-time-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = join-time-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = sync-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = sync-time-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = sync-time-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = heartbeat-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = heartbeat-response-time-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = last-heartbeat-seconds-ago
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  }
]
# Global Request Metrics
query = "kafka.consumer:type=consumer-metrics,client-id=*"
attributes = [
  {
    attribute = request-latency-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = request-latency-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = request-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = response-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = incoming-byte-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = outgoing-byte-rate
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  }
]
# Global Connection Metrics
query = "kafka.consumer:type=consumer-metrics,client-id=*"
attributes = [
  {
    attribute = connection-count
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = connection-creation-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = connection-close-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = io-ratio
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = io-time-ns-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = io-wait-ratio
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = select-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = io-wait-time-ns-avg
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  }
]
# Per-Broker Metrics
query = "kafka.consumer:type=consumer-node-metrics,client-id=*,node-id=*"
attributes = [
  {
    attribute = request-size-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = request-size-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = request-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = response-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = incoming-byte-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = outgoing-byte-rate
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  },
  {
    category = "node-ids"
    key = "node-id"
  }
]

Reference (Producer)

copy# Global Request Metrics
query = "kafka.producer:type=producer-metrics,client-id=*"
attributes = [
  {
    attribute = request-latency-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = request-latency-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = request-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = response-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = incoming-byte-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = outgoing-byte-rate
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  }
]
# Global Connection Metrics
query = "kafka.producer:type=producer-metrics,client-id=*"
attributes = [
  {
    attribute = connection-count
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = connection-creation-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = connection-close-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = io-ratio
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = io-time-ns-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = io-wait-ratio
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = select-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = io-wait-time-ns-avg
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  }
]
# Per-Broker Metrics
query = "kafka.producer:type=producer-node-metrics,client-id=*,node-id=*"
attributes = [
  {
    attribute = request-size-max
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = request-size-avg
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = request-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = response-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = incoming-byte-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = outgoing-byte-rate
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  },
  {
    category = "node-ids"
    key = "node-id"
  }
]
# Per-Topic Metrics
query = "kafka.producer:type=producer-topic-metrics,client-id=*,topic=*"
attributes = [
  {
    attribute = byte-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = record-send-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = compression-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = record-retry-rate
    metric-type = "GAUGE_DOUBLE"
  },
  {
    attribute = record-error-rate
    metric-type = "GAUGE_DOUBLE"
  }
]
additional-identity-keys = [
  {
    category = "client-ids"
    key = "client-id"
  },
  {
    category = "topics"
    key = "topic"
  }
]

Extending the settings

It is possible to override and extend the above settings, as well as add more MBeans by extending the settings.