Elixir Blog Post: Elixir Observability: OpenTelemetry, Lightstep, Honeycomb

Really good post, thanks a lot!

I’ve been wrestling with OpenTelemetry in Rust a few months ago and it took me a day or two to even find my way around all the terminology and then be able to send metrics to Jaeger and Zipkin. And Prometheus afterwards, with Grafana.

Your post added a few extra strokes to the picture that I was not clear on. Much appreciated!

Thanks for the kind words. If there are sub-topics that you’d like me to dive into, please let me know and I’ll consider them!

Yes, there’s one thing.

Most OpenTelemetry guides never talk about what piece fits where. A simple diagram showing how on one of the “floors” of this “building” you can have Jaeger, Zipkin, or Prometheus for example would be hugely helpful.

When I was learning OT this was very hard to grasp initially. Nobody seems to care to give the big picture – literally – with a diagram.

So something like: “To have OT work well you need the pieces X, Y and Z – for collecting/ingesting data, aggregating them, and visualizing them. For Y you can choose between Jager and Zipkin, for Z you can choose between this or that (e.g. Grafana and friends)”.

Strangely, such perspective is almost always missing.

This is a great post! Thanks

If I might also draw attention to the “Mobius” library for elixir. It doesn’t do anything like the above, however, it solves a similar problem when all you wanted was something simple, right now. So I came across it when I was doing a little embedded stuff and had no desire to setup a bunch of servers, timeseries databases, grafana, etc, etc.

Mobius is just a stupidly simple little mini timeseries database, complete with the ability to plot basic graphs from the command line… It in no way does anything as powerful as all the stuff you described… But you know what! For me at least it created the DEMAND to go and get something better! Before that I had no way to see how useful telemetry was. Mobius let me quickly answer simple questions, which in turn has led me to want more. So I went from having no telemetry to adding stuff to all my key functions. From here I can use opentelemetry/grafana/etc to get proper analysis. However, I don’t think I would have bothered without that leg up

Give it a whirl. I think it’s a great stepping stone to create demand for a proper solution

Thanks for the reminder, I’ve been meaning to try it for months now.

Very cool! 3 months ago i was implementing opentelemetry on elixir and it was a bumpy ride. It would be a smoother ride if i had read your blog post then.

Regarding subtopic, there were some additional problem that i was wrestling with besides basic otel:

1. How to implement opentelemetry instrumenter in elixir: if you have a custom library/plugin which are not yet included in standard opentelemetry instrumenter for elixir/erlang. how you should develop it or extend existing?
2. Performance tuning: I once has a performance problem with opentelemetry (memory usage and latency) for high request throughput server, mainly caused by number of span and trace stored by opentelemetry. i was able to fix it at the time with fiddling following variable:

OTEL_BSP_SCHEDULE_DELAY_MILLIS: "500"
OTEL_BSP_MAX_QUEUE_SIZE: "1024"
OTEL_SPAN_SWEEPER_INTERVAL: "120000"
OTEL_SPAN_SWEEPER_STRATEGY: failed_attribute_and_end_span
OTEL_SPAN_SWEEPER_SPAN_TTL: "120000"

also, for someone which new to opentelemetry integration on elixir, CNCF Slack for opentelemetry erlang working group is very welcoming and i was able to get a lot of help there GitHub - open-telemetry/opentelemetry-erlang: OpenTelemetry Erlang SDK · GitHub

Great post, I noticed that you focused more on high cardinality data and how that can be exported to Lightstep or Honeycomb, but I am wondering what would be if I would like to export metrics (low cardinality) as well and what would be the main differences.