What are the best practices for persisting telemetry data?

The whole idea of telemetry is that you can plug in any data sink. Have a look at something like telemetry_metrics_prometheus. Never used it, but seems you should be able to pick it up pretty fast.

I haven’t used any of this in action but it might be worth to look into the ecosystem around OpenTelemetry which includes collectors which can publish to a variety of backends, among them Prometheus and others.

I know that this isn’t exactly what you were looking for but I felt it could be helpful to point it out, if not to you then maybe to others.

I think PromEx is a good choice.

Thanks a lot @stefanchrobot, @wolf4earth and @tejpochiraju for your answers! They’ve all been useful to me

Apparently, there is no “Elixir replacement” for Prometheus, which is what I initially suspected. And it makes sense, no point in reeinventing the wheel.

telemetry_metrics_prmetheus looks like a super quick way to expose a /metrics endpoint for Prometheus to scrape, PromEx does pretty much the same thing AFAICS but it adds a custom Grafana dashboard that you can import. Sweet. Thanks for the pointers, now I know what I need to do

Another option, especially if you have many Phoenix apps or if you dynamically launch them, is to write to a central metrics repository such as statsd_exporter and then have prometheus scrape it. I remember using statix on Phoenix side to send metrics.

Basically you have this:

+-------------------+        +---------------------+                        +--------------+
|  Phoenix + statix |------->|   statsd_exporter   |<---(scrape /metrics)---|  Prometheus  |
+-------------------+        +---------------------+                        +--------------+

with as many Phoenix instances as you need.

What’s the difference between Telemetry and OpenTelemetry? Should the Erlang ecosystem adopt OpenTelemetry instead?

My understanding was the Erlang OpenTelemetry bindings can be used to export Telemetry events but maybe I’m wrong.

Integration and adoption is already happening:
https://opentelemetry.io/docs/erlang/

But also telemetry can be a low level detail to fetching the data needed for open telemetry as far as I understand it. They’re not competing.

Interesting. I believe this would be overkill for my current setup, but it’s something to keep in mind.

Note that OpenTelemetry Metrics is still very early, so if you are mainly looking at exporting metrics like with prometheus you will want to use something else unless you are ok with lots of breakage and having to dig in to the code

And yes, telemetry the library has a much more narrow purpose, it can be integrated with OpenTelemetry through handlers.