Akka Projection

Lightbend Telemetry is capable of capturing data for Akka Projection.

Cinnamon Akka Projection module dependency

After adding the Cinnamon Agent as described in the setup, make sure that you add the Cinnamon Akka Projection module dependency to your build file:

sbt

copylibraryDependencies += Cinnamon.library.cinnamonAkkaProjection

Maven

copy<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-akka-projection_2.13</artifactId>
  <version>2.20.4</version>
</dependency>

Gradle

copydependencies {
    implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-akka-projection_2.13', version: '2.20.4'
}

Projection metrics & events

The following metrics are recorded, type of metric in parenthesis.

• Active projections (gauge) — number of running projections. This is a counter of instances so if you run a sharded projection the counter will indicate the number of shards.
• Processed envelopes (gauge) — number of processed envelopes or number of times an envelope handler ran successfully.
• Offsets committed (gauge) — number of envelopes marked as processed or number of offset positions advanced.
• Wait time (recorder) — time (in milliseconds) an envelope spent on the queue/broker/journal. This is the time since the envelope was created until it was read, deserialized, and considered ready to be processed by the projection. This metric depends on the type of source since some sources don’t provide the creation timestamp. At the same time, some producers record the creation time in the producing process, while others record the timestamp on the database/broker when accepting the envelope. Note this metric is affected by existing time skew between nodes on your infrastructure.
• Service Time (recorder) — time (in nanoseconds) from the instant the envelope was considered ready until the projected data was available for reading. Note this time doesn’t include the period until data being readable and the offset being committed. As a consequence, when using at-least-once semantics, this metric will be reported for every envelope handler completing successfully (even if storing the offset failed and the envelope is retried).
• Residence Time (recorder) — time (in milliseconds) spent since the creation of the envelope until its offset was committed.
• Processing errors (rate) — error rate during the envelope processing. If you use a handler strategy including a retry operation this number will increase.
• Projection failures (rate) — failure rate for projections. Failures indicate the projection required restarting.

Depending on the offset strategy and the retry strategy of the projection the number of processed envelopes and offsets committed will differ. For example, in case you opt to retryAndSkip as a retry stratgegy, when the envelope is skipped, the number of processed envelopes will not be increased but its offset will be committed so the number of offsets committed will be increased. Similarly, when using, for example, at-least-once semantics, envelopes may be processed successfully more than once causing processed envelopes to exceed offsets committed. Read these metrics considering the details of your projection.

Clock Skew

An event processed by a projection may have been created on a different machine. Wait Time and Residence Time, for example, measure the time period between the moment the envelope was created, and the moment the projection completed successfully.

As a consequence, given the skew between the clocks of different machines, some of the metrics above will only be reported partially. Specifically, negative time periods will not be recorded. Non-negative values, though, will be recorded but may be impacted by clock skew so the data displayed only be as good as the underlying clock coherence.