How are you using (open)telemetry in your Elixir applications

Excellent question. I think that a proper knowledge base on OTel in Elixir is long overdue for whoever might have the time to author it.

I’m going to self bump in the hope that someone has any insights. Given all the talks and recommendations I see here about telemetry it can’t be that no one has some good resources/recommendations here ( I hope)

First of all, Telemetry and OpenTelementry are 2 different things, 2 different packages that do not depends on each other.

Since you want to “adds some value but doesn’t involve too much work”, you can start with adding Telemetry to events of interest in your app, then you can observe time based chart that will be generated by Phoenix Dashboard and gain some insight of how your app is doing.It is very easy to setup.

Once you have done that, you may want to go further, to persist, export, and analysze the telemetry data that you are collecting. Then you might need OpenTelemetry.

Any recommendations here?

I just set up a telemetry stack for a Phoenix Application. Its on my homelab, so I’m not sure how it’ll work on fly.io, but hopefully it’ll give you a place to start. For traces you will need, at minimum, opentelemetry, opentelemetry_api, and opentelemetry_exporter. On top of that there are packages like opentelemetry_phoenix, which will trace important libraries for you out of the box. For metrics Elixir’s OTEL SDK isn’t ready yet, but you can use prom_ex, which is really good. And you can use an OTEL collector to handle both those metrics and those traces, and then forward them to whatever tool you are using. For logs you’ll just have to use whatever tool to collect them your stack offers (like promtail for grafana loki). Hopefully that’s enough to start, but if you have any questions just ask.

Just to make it easier to get into using any telemetry backend, one might take a look at telemetria library which is backend-agnostic and allows telemetry adoption via @telemetria module attributes, like

defmodule Forecast do
  use Telemetria

  @telemetria level: :info, group: :weather_reports, locals: [:farenheit]
  def weather(city) do
    fahrenheit = ExternalService.retrieve(city)
    Converter.fahrenheit_to_celcius(fahrenheit)
  end
end

Thank you all for the replies.

So if I get this correctly. If I just use telemetry, the only thing I currently get is a view in the phoenix dashboard of the current situation.

If I want to get more insights, I need to use OpenTelemetry and deploy or use a hosted backend for OpenTelementry?

Any recommendations for that? Preferably something that’s easy to self host?

Do you have anything specific in mind? Opentelemetry is just a backend for telemetry events. The amount of information will be determined by what metrics the libraries are sending by default and you from your application. Check the documentation for telemetry events that libraries are sending: Telemetry — Phoenix v1.8.8

Back in the day when telemetry was just released I used a self-hosted influxdb to store the events, nowadays there are a lot of options including opentelemetry which includes both a database for storage as well as a UI for building insights using those measurements.

• GitHub - openobserve/openobserve: Open source observability platform for logs, metrics, traces, frontend monitoring, pipelines and LLM observability. A sophisticated, simple and highly performant alternative to Datadog, Splunk, and Elasticsearch with 140x lower storage costs and single binary deployment. · GitHub
• GitHub - SigNoz/signoz: SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool · GitHub

I have used the first successfully, at least locally. Should be fine for production usage but do your own extra research on that.

Not open telemetry, but you could also have a look at this project: GitHub - lalabuy948/PhoenixAnalytics: 📊 Plug and play analytics for Phoenix applications. · GitHub