Yes, there’s one thing.

Most OpenTelemetry guides never talk about what piece fits where. A simple diagram showing how on one of the “floors” of this “building” you can have Jaeger, Zipkin, or Prometheus for example would be hugely helpful.

When I was learning OT this was very hard to grasp initially. Nobody seems to care to give the big picture – literally – with a diagram.

So something like: “To have OT work well you need the pieces X, Y and Z – for collecting/ingesting data, aggregating them, and visualizing them. For Y you can choose between Jager and Zipkin, for Z you can choose between this or that (e.g. Grafana and friends)”.

Strangely, such perspective is almost always missing.

Really good post, thanks a lot!

I’ve been wrestling with OpenTelemetry in Rust a few months ago and it took me a day or two to even find my way around all the terminology and then be able to send metrics to Jaeger and Zipkin. And Prometheus afterwards, with Grafana.

Your post added a few extra strokes to the picture that I was not clear on. Much appreciated!

Very cool! 3 months ago i was implementing opentelemetry on elixir and it was a bumpy ride. It would be a smoother ride if i had read your blog post then.

Regarding subtopic, there were some additional problem that i was wrestling with besides basic otel:

1. How to implement opentelemetry instrumenter in elixir: if you have a custom library/plugin which are not yet included in standard opentelemetry instrumenter for elixir/erlang. how you should develop it or extend existing?
2. Performance tuning: I once has a performance problem with opentelemetry (memory usage and latency) for high request throughput server, mainly caused by number of span and trace stored by opentelemetry. i was able to fix it at the time with fiddling following variable:

OTEL_BSP_SCHEDULE_DELAY_MILLIS: "500"
OTEL_BSP_MAX_QUEUE_SIZE: "1024"
OTEL_SPAN_SWEEPER_INTERVAL: "120000"
OTEL_SPAN_SWEEPER_STRATEGY: failed_attribute_and_end_span
OTEL_SPAN_SWEEPER_SPAN_TTL: "120000"