rkallos

Peep - Efficient TelemetryMetrics reporter supporting Prometheus and StatsD

Peep is a new TelemetryMetrics reporter that supports both StatsD (and Dogstatsd) and Prometheus.

While load testing a new Websocket-based API gateway written in Elixir, I encountered performance issues with TelemetryMetricsPrometheus.Core and TelemetryMetricsStatsd. This prompted me to write Peep, which makes different choices about storing and sending TelemetryMetrics data.

Instead of sampling or on-demand aggregation, Peep uses histograms (backed by :ets.update_counter/*) to store distributions, copying the approach taken by DDSketch.
Instead of sending StatsD packets for each telemetry event, StatsD data is periodically sent in a small(er) number of large(r) packets.

This library is currently running in production, in a service handling >1 million requests per minute. With a moderate number of metrics defined, the service emits StatsD data at a rate of 4KiB/s, with no observed packet drops (we use Unix Domain Sockets to send Dogstatsd lines to Datadog agents, so it’s possible for :gen_udp to return :eagain when attempting to send packets).

Here’s an image showing a drop in CPU use after replacing TelemetryMetricsPrometheus.Core and TelemetryMetricsStatsd with Peep:

Here’s another dashboard for the same period of time, showing a slight (but not unwelcome!) drop in memory usage:

Feedback and contributions welcome!

14 comments

#library #prometheus #datadog #telemetry #statsd #telemetry-metrics

73 2007 14

2026-01-23 21:01:06 UTC

Most Liked

rkallos

Hello again, everyone!

Peep v4.3.0 was released today, and since it has been a while since I gave an update here, I thought I’d share what’s changed in Peep over the past few months.

v4.1.0

Thanks to @hauleth, Peep now emits a :telemetry event when reading storage size. This should remove some tedious boilerplate for those who wrote their own :telemetry_poller callbacks for reporting on Peep’s storage size.
@hauleth contributed another change that should improve performance slightly; using a tuple instead of a map to access an ETS table in Peep.Storage.Striped.
@hauleth also contributed some further performance improvements to Peep.Codegen, removing some runtime indirection overhead, and inlining a hot function by turning it into a macro.
@hauleth added some much needed flexibility to Peep.Buckets.Custom, making it possible to pass in variables and the results of function calls in the list of buckets passed when invoking use Peep.Buckets.Custom. He also fine-tuned the threshold length that Peep.Buckets.Custom uses to decide between linear search and binary search for finding which index to increment.
@hauleth also fixed some incorrect escaping, and made Peep’s Prometheus export text conform to OpenMetrics.
I added Dialyzer and Credo to the project, which are now run in CI.

v4.1.1

@hauleth added support for 2-ary callbacks for the :keep option in Telemetry.Metrics

v4.2.0

@pepicrft fixed an issue where Peep’s Prometheus export would crash while trying to convert terms to strings that don’t implement the String.Chars protocol.

v4.2.1

@v0idpwn fixed an incorrect type spec.

v4.3.0

@edgurgel made it possible for folks to provide their own implementations of Peep.Storage. He also made it possible to test these new implementations against the test suite Peep has been using internally. Maybe we’ll see some Peep.Storage modules implemented with NIFs someday!

Post #15

rkallos

I missed posting a few minor releases of Peep in the past few months, but today, Peep 4.0.0 was released.

Upgrading from 3.x should be straightforward, as the only backwards-incompatible change made is that you can no longer store non-integer values in last-value metrics. You may have somehow gotten away with it in earlier versions of Peep, but it won’t work anymore.

Here’s a changelog of releases from Peep 3.3.1 to 4.0.0:

v3.3.1

I added :on_unmatched_path to Peep.Plug, allowing for users to decide the behaviour when Plug.Peep is called with an unexpected path. In short, you probably want :continue if serving Peep metrics from the same HTTP server as your application (e.g. in a Phoenix-based service), and :halt when serving Peep metrics from a separate listener (e.g. when you want to serve metrics on a different port)
@yordisprieto contributed some documentation

v3.4.0

@josevalim implemented several optimizations the Prometheus export code, making it more efficient.
@yordisprieto added a compile-time :bucket_calculator option to Peep, making it possible to globally specify a Peep.Buckets implementation for all distribution metrics.
@hst337 implemented some optimizations, most impressively a change to Peep.Buckets.Custom that switches from linear-ish function head matching to logarithmic binary search. For small numbers of buckets, the performance is roughly equivalent. However, for large numbers of buckets, performance is much improved. This PR inspired me to contribute a similar optimization to a Cassandra client library that we use in production for a nice performance improvement!

v3.4.1

@scudelletti fixed an issue of requiring Plug at compile-time, making Plug truly an optional dependency for Peep.

v3.4.2

@hauleth fixed a bug around quote-escaping labels in Peep’s Prometheus export code.

v3.5.0

At @hauleth’s request, I implemented Peep.prune_tags/2, which deletes metrics based on matching tag values. This is useful for metrics with unavoidable high cardinality, but may not be all that useful for more typical users of Peep.

v4.0.0

Introduces Peep.Codegen, an internal module that compiles a module on Peep startup with a handle_event/4 function that replaces Peep.EventHandler.handle_event/4. This avoids the overhead of copying data out of :telemetry’s ETS table into a calling process’s heap, as that data could be stored in the compiled module’s literal pool instead. Further, Peep now gives integer ids to each metric, and uses those integer ids instead of Telemetry.Metrics structs when looking up data in ETS tables, further reducing unnecessary copying of terms.
Peep now drops non-number measurements for last-value metrics. Thanks to @akoutmos for originally pointing out this (mis)behaviour.
Peep now automatically adds global tags to metrics, rather than forcing users to repeatedly specify the global tag keys in :tags for every metric. Thanks to @yordisprieto for pointing this out, and pointing out the fix.

Peep v4.0.0 appears to be quite a bit faster than v3.x. Here’s a chart showing p50, p95, and p99 latency before and after deployment of Peep 4.0.0 in an application that uses Peep heavily:

Thanks to those who contributed to Peep these past few months. With your help, Peep is now better than ever!

Post #11

hauleth

Thank you for that project. It allowed me to give Supavisor ~30x boost in latency (measured by pgbench) over using telemetry_metrics_prometheus_core. I also have prepared PR for prom_ex to be able to use peep as a metrics store.

Post #3