Telemetry - Dynamic dispatching library for metrics and instrumentations (v0.3.0 released!)

Your Libraries & OS Mentoring Libraries
1

Hi all!

I’m happy to announce that Telemetry v0.3.0 is out! This release marks the conversion from Elixir to Erlang so that all the libraries and projects on the BEAM can expose useful instrumentation data via Telemetry events! :heart:

This version is not compatible with the previous versions, below you will find the description of how to move from 0.2.0 to 0.3.0.

For those who publish

The only change in how the events are emitted is the module name:

# 0.2.0
Telemetry.execute([:my, :event], 20, %{some: :metadata})

# 0.3.0
:telemetry.execute([:my, :event], 20, %{some: :metadata})

For those who subscribe

The attach and attach_many have been changed a bit - now they accept an anonymous function as the handler function instead of a module/function pair.

# 0.2.0
Telemetry.attach("my-handler", [:my, :event], EventHandler, :handle, %{some: :config})

# 0.3.0
:telemetry.attach("my-handler", [:my, :event], &EventHandler.handle/4, %{some: :config})

To see all the changes please check out the change log.

Credits

Many thanks to @tristan for all of his help on the rewrite :heart: We wouldn’t have made it in 2018 without him :smiley:

Also, a big thank you to all people supporting and using the project, asking questions and submitting feedback!

Feedback

Please let us know what you think about the release here or on the issue tracker.

Links

19 Likes
2

You should send this out to the Erlang mailing list as well :slight_smile:

5 Likes
3

What is necessary for this to become 1.0.0?

2 Likes
4

Polishing the edges, finding “the best API for the job” and overall live-code testing I believe.

5

To add to what @hauleth said, marking 1.0.0 means that we’re sure there will be no breaking changes for a while. To do that, we need to know if the API suits the use cases we had in mind and those we didn’t predict. For that, we need feedback - if you have any suggestions, or you have found any problems, please open up an issue on Github :slightly_smiling_face:

Unfortunately, I don’t think I can give you more specific answer at the moment.

6

Hi there! I’ve recently started to add monitoring support (with Prometheus and Grafana) to my Phoenix application. I stumbled accross your Telemetry project and also Prometheus Elixir by @deadtrickster. It is not entirely clear to me, however, how the two libraries complement one another, although I vaguely sense that they do :stuck_out_tongue:

Specifically, when I use Prometheus.ex, isn’t it already possible to add custom metrics? Which is kind of what Telemetry is for as well? Or is it for example possible to achieve better decoupling by using Telemetry “backed” by Prometheus.Ex, so that it is simpler to swap out Prometheus for another monitoring solution? I would appreciate it if you could help me understand the two libraries’ specific purpose a bit better :slight_smile:

7

@jsm great question! And you’re right, Prometheus.ex lets you define all the custom metrics you want. However, Telemetry tries to achieve another objective.

The goal of Telemetry project is to have a single interface for exposing instrumentation data across the whole ecosystem. Libraries can publish events with this data, and you can subscribe to them and build metrics using these values.

For example, Ecto already uses Telemetry to publish an event on each query. You could subscribe to this event and bump the Prometheus metric without having to instrument every database query yourself. Soon Telemetry events will be also published by Plug and Phoenix.

Today we’ve also released another project, Telemetry.Metrics, which provides a common interface for metric aggregations, like counters, sums, etc. If libraries like Prometheus.ex would integrate with it, setting up metrics which are based on Telemetry events will be even more simple.

Note that neither Telemetry nor Telemetry.Metrics is able to export metrics to external systems. You will always need a library like Prometheus.ex to translate the events to some useful representation :slightly_smiling_face:

3 Likes
8

As I understand it - I instrumented Ecto via Logger, I instrumented Phoenix via instrumenter. Now when they integrated with Telemetry, I work with the same interface and concepts for both Ecto and Phoenix. This is true for all other libs utilizing Telemetry. In a way, it’s a “hooks” library, just with more defined domain. But having “real” counters/exposition still required.

For your own projects, Telemetry abstraction will help when you decide to switch to other monitoring system.

@arkgil please confirm :slight_smile:

3 Likes
9

@deadtrickster yes, exactly! Thanks for this, you put it really well :slightly_smiling_face:

1 Like
10

Thanks to you both for those answers :slight_smile: That cleared some stuff up for me!

11

opencensus_telemetry integration have been released that supports Telemetry.Metrics as well.

1 Like
12

Hi folks! :wave:

We have just released Telemetry 0.4.0. With this version, Telemetry events now carry a map of measurements instead of a single measurement. This is useful in cases when there are multiple values describing what just happened, e.g. in case of HTTP request the event might include decompression time, payload size, and processing time:

:telemetry.execute(
  [:http, :request],
  %{payload_size: 12874, processing_time: 120, decompression_time: 5},
  %{status: 200, path: "/..."}
)

Previously, each measurement needed to be emitted in a dedicated event. Now the event tells that a thing happened, and measurements and metadata describe its properties.

This change is backwards-compatible for event publishers, i.e. you can still call :telemetry.execute/3 with a single value. However, event handlers need to handle a map of measurements now.

Last but not least, thanks for the awesome feedback from the community which lead to this release!

14 Likes
13

@arkgil it would be useful to see a sample config for Telemetry that adds OpenTracing instrumentation for all Phoenix controller actions and Ecto database queries.

14

Right now Telemetry do not implement tracing at all. What you can use though is OpenCensus with Plug integration.

15

@hauleth thanks for this! Is tracing on the roadmap for telemetry? Seems like that would be the canonical use case…

16

I would love to see a tracing library based on Telemetry. I don’t think we have enough resources now to pursue this, but contributions are always welcome and we’ll be happy to discuss things :slight_smile:

17

I think that tracing via Logger would be more feasible, like @ferd tracelog which uses Erlang’s logger metadata to create OpenCensus traces.

2 Likes
18

I’ve always found it confusing when people talk of telemetry having a tracing library.

Technically it can work. Either use the pdict or pass the current span context as part of the metadata to each execute and then use events start_span and end_span to create/finish children.

That is, I don’t think spans should be considered and treated like events.

And then Telemetry really starts to just be an alternative to OpenCensus, hehe, but with a different philosophy of being vendor neutral. Telemetry tracing would delegate to other libraries like telemetry stats do (so like ex_ray or New Relic) while OpenCenus works to create a common core.

Anyway, I see Telemetry events as perfect for adding attributes and time events (https://opencensus.io/tracing/span/time_events/annotation/) to existing spans and was planning to have it plug in to OpenCensus tracing that way.

2 Likes
19

Anyway, I see Telemetry events as perfect for adding attributes and time events (OpenCensus) to existing spans and was planning to have it plug in to OpenCensus tracing that way.

This is interesting!

I’ve always found it confusing when people talk of telemetry having a tracing library.

To me it’s all about reusing existing bits and pieces - if telemetry events are already in place, I can use them to generate spans, e.g. I could have a trace of Phoenix request + all DB queries made by Ecto. This is probably not as good as spans carefully placed by hand, but it is something for start :slight_smile:

20

True, and I do see the appeal of wanting to something for “free” if the information is already there in some form.

But keep in mind how much influence you’ll have with the Elixir community with this library and the documentation around it. And since it is mostly new, not something existing in people’s code they can simply tap into today but will instead be instrumenting in the future, it can start for the “right place”. Meaning instead of creating spans have a lib for annotating spans through different libs like opencensus/ex_ray/etc, and documentation for Telemetry explaining how its events can be used to annotate spans in a distributed trace.

Because of the influence on the community to come from a library with backing like telemetry has this will help guide people to checkout distributed tracing solutions.

6 Likes