So I have been using Telemetry for a while, but on hosts themselves been mostly doing simple metrics that are counters / last_value, and report to statsd via a simple reporter. Nothing fancy, and the actual aggregation mostly happens outside application.
I have another project, however, where we want to calculate some of these metrics on host itself. The reason being of havin millions events per minute that we do not really care about in any other way than let’s say “give me number of events per minute emitted”. So we don’t want to flood the external system with the irrelevant data and pre-calculate the values on Elixir application, and send aggregate to a reporting dashboard via API call.
I’m not sure if I am doing it right, however. Let’s simplify the use case a bit, let’s say I am interested in a metric of “throughput”, defined as “number of requests served in given minute”.
I am currently using a combination of two metrics to achieve that. First, I have “counter()” that simply increments number of requests served by the application.
Then, I have a custom poller, a GenServer, which wakes up every 60 seconds, stores the current counter value in it’s state, and subtract previous value from current value, and emits “last_value()” type of metric.
This last_value() is my “throughput” that I report to the dashboard using custom reporter.
Is this the right way to do it?