Excellent question. I think that a proper knowledge base on OTel in Elixir is long overdue for whoever might have the time to author it.

Just to make it easier to get into using any telemetry backend, one might take a look at telemetria library which is backend-agnostic and allows telemetry adoption via @telemetria module attributes, like

defmodule Forecast do
  use Telemetria

  @telemetria level: :info, group: :weather_reports, locals: [:farenheit]
  def weather(city) do
    fahrenheit = ExternalService.retrieve(city)
    Converter.fahrenheit_to_celcius(fahrenheit)
  end
end

I just set up a telemetry stack for a Phoenix Application. Its on my homelab, so I’m not sure how it’ll work on fly.io, but hopefully it’ll give you a place to start. For traces you will need, at minimum, opentelemetry, opentelemetry_api, and opentelemetry_exporter. On top of that there are packages like opentelemetry_phoenix, which will trace important libraries for you out of the box. For metrics Elixir’s OTEL SDK isn’t ready yet, but you can use prom_ex, which is really good. And you can use an OTEL collector to handle both those metrics and those traces, and then forward them to whatever tool you are using. For logs you’ll just have to use whatever tool to collect them your stack offers (like promtail for grafana loki). Hopefully that’s enough to start, but if you have any questions just ask.