But I don’t know where to start, so I wanted to ask you what your experiences are with telemetry and/or open telemetry. How you use it, how your setup looks like,…
I currently have some applications deployed on fly.io and I was wondering what a good way to start would be. A start could be something that adds some value but doesn’t involve too much work.
So I’d love to hear how you’re using it, what value does it bring for you,…
I’m going to self bump in the hope that someone has any insights. Given all the talks and recommendations I see here about telemetry it can’t be that no one has some good resources/recommendations here ( I hope)
First of all, Telemetry and OpenTelementry are 2 different things, 2 different packages that do not depends on each other.
Since you want to “adds some value but doesn’t involve too much work”, you can start with adding Telemetry to events of interest in your app, then you can observe time based chart that will be generated by Phoenix Dashboard and gain some insight of how your app is doing.It is very easy to setup.
Once you have done that, you may want to go further, to persist, export, and analysze the telemetry data that you are collecting. Then you might need OpenTelemetry.
I just set up a telemetry stack for a Phoenix Application. Its on my homelab, so I’m not sure how it’ll work on fly.io, but hopefully it’ll give you a place to start. For traces you will need, at minimum, opentelemetry, opentelemetry_api, and opentelemetry_exporter. On top of that there are packages like opentelemetry_phoenix, which will trace important libraries for you out of the box. For metrics Elixir’s OTEL SDK isn’t ready yet, but you can use prom_ex, which is really good. And you can use an OTEL collector to handle both those metrics and those traces, and then forward them to whatever tool you are using. For logs you’ll just have to use whatever tool to collect them your stack offers (like promtail for grafana loki). Hopefully that’s enough to start, but if you have any questions just ask.
Just to make it easier to get into using any telemetry backend, one might take a look at telemetria library which is backend-agnostic and allows telemetry adoption via @telemetria module attributes, like
defmodule Forecast do
use Telemetria
@telemetria level: :info, group: :weather_reports, locals: [:farenheit]
def weather(city) do
fahrenheit = ExternalService.retrieve(city)
Converter.fahrenheit_to_celcius(fahrenheit)
end
end
Do you have anything specific in mind? Opentelemetry is just a backend for telemetry events. The amount of information will be determined by what metrics the libraries are sending by default and you from your application. Check the documentation for telemetry events that libraries are sending: Telemetry — Phoenix v1.7.21
Back in the day when telemetry was just released I used a self-hosted influxdb to store the events, nowadays there are a lot of options including opentelemetry which includes both a database for storage as well as a UI for building insights using those measurements.
Im currently using a grafana stack (grafana, loki, tempo). All open source and they can store the data in the file system, which means I don’t have to deploy another db. The only problem is that tempo is a huge memory hog.
