netDalek

Telemetry influxdb reporter

I’m thinking about rewriting metrics subsystem in my project. I wish to do it in some unified manner so it can be easily copied to other projects. I’m looking at telemetry and inspecting its reporters. We use influxdb, so I have to write a reporter because there isn’t any.

The reporter’s idea is probably the most complex part of telemetry. I compare Statsd and Prometheus reporters. Prometheus one does intermediate aggregation by itself, whereas statsd reporter doesn’t. With influxdb, we even don’t have different data types as we have with statsd. All aggregation in done afterwards. Intermediate aggregation looks very efficient because it is done inside Erlang VM.

So I want to create a summing counter that flushes data to influxdb on a regular basis. What is the right way to do it with telemetry? Should I write a reporter with an aggregation feature or write some external summing counter and flush its values using Telemetry.Poller and then write a telemetry reporter that writes data to influx without aggregation? In the second case, the new reporter should treat all types of telemetry metrics mostly the same. In the first case I need to specify some time interval when starting the reporter.

18 comments

#metrics #telemetry #influxdb

0 4008 18

2020-10-13 13:49:19 UTC

First Post!

Phillipp

There is one, but I am not sure if it fulfills your needs.

https://github.com/ludwikbukowski/telemetry_influxdb

Post #1

Most Liked

rawkode

Hi,

I work at InfluxData, the company behind InfluxDB.

We’re actively looking at how we can contribute to the BEAM ecosystem, so it’s great to see some people working with this stack use InfuxDB.

I generally advise avoiding pre-aggregated metrics, but obviously there’s a trade off and sometimes it’s the best route forward.

You can do this currently with Telegraf, which can proxy InfuxDB; this would allow you to do aggregation without code.

I’ll put an example together if you think it would be useful

Post #2

tristan

Rebar3 Core Team

Is InfluxData involved at all with https://opentelemetry.io/? The first draft of the metrics specification is being finished up now I believe.

If you are involved in OpenTelemetry or plan to support it then I think the best way for InfluxData to contribute to the BEAM ecosystem is through the Erlang/Elixir libraries, GitHub - open-telemetry/opentelemetry-erlang: OpenTelemetry Erlang SDK · GitHub

We have a SIG as part of the OpenTelemetry project GitHub - open-telemetry/community: OpenTelemetry community content · GitHub

Post #7

LostKobrakai

I guess the solution for doing it within the telemetry ecosystem would be have the telemetry reporter do aggregation/sampling. Afaik it’s the place meant to handle conversion between reported metrics and what’s actually send out to elsewhere. This can be as easy as just forwarding data or be quite complex by doing pre-aggregation or sampling. In the end it depends on how many moving parts one likes to have inside and/or outside of telemetry.

Post #3

Last Post!

hauleth

There is EEF Observability WG that is working on monitoring facilities in Erlang (and naturally - Elixir). The current consensus is to use telemetry as event dispatcher that is backend agnostic. On top of that you can use any metrics gatherer you like, for example mentioned above telemetry_influxdb. If you want to have more “holistic” solution for monitoring you applications, then you can check out opentelemetry application that will provide you metrics and traces, in future maybe even logs, gathering together with tooling to dispatch that data to various storage and processing engines.

Post #18