As tracing and metrics gathering seems to still be problem for some of the newcomers I have started few projects to improve that by utilising OpenCensus. During my last 2 days of work I have added few 3 new libraries to the OpenCensus BEAM organisation:
DataDog - support for DogStatsD metrics and for DataDog’s APM traces.
plug integration (not yet released on Hex) - tracing and measurements. For now there is support for pipeline as a whole, maybe in future I will add “per plug” support.
Phoenix integration - maybe, maybe not. I am not sure how much it will be needed when Phoenix integrate telemetry. Maybe tracing, but that would be covered by opencensus_plug library
Ecto integration - while measurements are covered by opencensus_telemetry, support for traces would still be desirable, however no idea how to support it right now
Wrapper library for erlang:trace/3 - this would be really interesting project, that would allow us to trace enormous amount of data. However there are few problems I see right now, and finding solution for them can be a little bit costly, so for now it is left deep in the backlog.
Let me know what other libraries and integrations you would like to see.
Using trace data is likely a bad idea. There can only be one tracer at any point in time (no concurrency), so if you integrate it to the metrics, nobody can use it to debug at the same time.
I’m happy to see that OpenCensus is gaining more traction in the BEAM ecosystem! I think it was about 6-9 months ago that I last checked and it didn’t seem like there was enough there yet to go all in, but now it seems that there are a number of people working on it.
I am maintaining the Spandex project along with @zachdaniel, with the goal of being an OpenTracing-compatible API that supports multiple vendor back-ends, very similar to the goals of OpenCensus. Currently, we only support DataDog APM as a back-end, but I think it should be relatively simple to support others if someone wanted to (and I’d love to know where they got stuck if it didn’t work out). We’re also looking forward to using Telemetry to collect span data in the future as more libraries support it. In the meantime, you may be interested to look at how Spandex integrates with Plug/Phoenix and Ecto. We have an example project that demonstrates how to integrate Spandex into an existing app, if you’re curious (see the links in the README for a code diff).
Also, as you probably know, several vendors have their own first-party support for tracing that is proprietary to only their back-ends, but mostly open-source (e.g. AppSignal, New Relic, Scout).
I’d be interested to look more into how the OpenCensus BEAM core compares to Spandex, and see whether there’s anything that can be shared somehow so everybody wins.
I agree :). I spent a really long time on Spandex, but I don’t want to be working against a more standardized implementation of tracing in Elixir. I’d hope that either spandex could be a standard interface for tracing an Elixir application, or the good parts of it could be salvaged/copied into a new standard. My point being that we should only hold tight into spandex if it’s going to be good for the community. If the community needs a more standard implementation, I’d be happy to support it.
I see no problems with making Spandex thin wrapper over OpenCensus library as API and concepts are quite similar. To be honest I even thought about creating OpenCensus wrapper for Elixir, that would end like something like Spandex, but in the end I didn’t see much value in that (at least yet, because I have proposed moving some utilities to macros). I also believe that we should reduce fragmentation in the matter of tracing and metrics gathering, but not only in Elixir, but in BEAM as a platform. It would be best if we would have single toolset to export metrics form both Erlang and Elixir.
Main reasons why I have joined OpenCensus project rather than Spandex are:
OpenCensus is Erlang project, so by adding new integrations there I can improve all BEAM languages at once. While we can still use Elixir from other languages in BEAM-family it isn’t that pretty, especially as a lot of Elixir code relies on Elixir’s macros. Porting Erlang macro to Elixir is much more straightforward than other way around.
In theory OpenCensus have bigger backing by being Google project that was open-sourced. It also has some big names in “Partners & Contributors” section.
OpenCensus also provide interface for metrics gathering, and I wanted something for my project Imager that would provide me single interface to export metrics and traces for different systems defined at runtime (PR with support for that is still WIP).
My only main problem with OpenCensus library right now is that it completely lacks documentation, which I will try to tackle soon. A lot of data is outdated, and it would be really wort to improve that, as it could help building community around such standard.
About integrating with Plug/Phoenix and Ecto I would need to create example projects in opencensus/plug and opencensus/telemetry. Only place where I still need to find how to hook into Ecto is tracing, but I have seen that you are using :trace level of Ecto logger for that, it can be worth checking this out that for adding new integration.
Another useful project would be some benchmarks to both check overhead and eventually compare different ways of setting traces and collecting metrics. For example, I have a work in progress for using a message to a pool of workers for collecting metrics instead of doing them inline. In many cases just doing them inline is likely preferred – when not updating lots of views per measure – but useful to have numbers.
I should also mention that there are other OpenTracing-related projects in the BEAM ecosystem that seem relatively robust as options: otter for Erlang project and ExRay, which is an Elixir wrapper around it.
I agree that the ideal situation would be a robust core that allows the difficult/complicated parts to be shared by all BEAM languages, and perhaps wrapper interfaces native to each language so that it feels more idiomatic (for example, with respect to the macros). I think Spandex could be adapted somewhat-easily to that because it doesn’t rely in any magic or macros at its core - only for convenience so that we can offer function decorators and automatically generate some code for you so that you don’t have to explicitly pass in all of the default options to each API call.
I was fully aware of other solutions available out there. The points that (at least for me) were the most important are:
OpenCensus Erlang library seems to be backed by Google, and even if it isn’t then it seems more “official” than community projects like otter and Spandex
It supports not only tracing but metrics as well. Having common API for both of these is quite handy.
Is supported by the author of the Prometheus implementation which I quite liked and used in some projects.
Adding new integrations was a breeze
Metrics supports tags (in contrast to exometer/elixometer or Folsom)
So everything I needed was in just one package, what I really appreciate. So, with all respect to you guys, I place my bet on OpenCensus.
Yea, I would love if we could all share an Erlang core. But now that there are so many implementations I doubt that is ever going to happen. And I’m partly to blame :). I started opencensus-erlang after trying otter when it was released.
I find opencensus to be important as a core because it defines so much (like propagation encoding and even a stacktrace proto format so services will actually be able to understand Erlang stacktraces! – small potatoes to Elixir world since services are starting to pick up on supporting Elixir).
It is also very open development of the specs and accepted Erlang as part of this process from early on.
tracelog by @ferd (known as author of Learn You Some Erlang, Erlang in Anger and * Property-Based Testing with Proper, Erlang, and Elixir) which allows you to trace your application by using new Erlang’s logger module and metadata assigned to log messages.
tracelog 0.1.0 have landed on Hex, we encourage you to test this out
opencensus_telemetry 0.1.0 have been released, it provides seamless integration between Telemetry 0.3.0 and additionally supports Telemetry.Metrics via oc_telemetry:track/1
oc_datadog 0.2.0 (name will probably change in future to opencensus_datadog instead) released with fix that now allows user to define views with exporter-independent naming and exporter will normalise it to the format used by the DogStatsD (non alphanumeric characters will be replaced with . (dot))
Future work:
I have started work on zPages for Erlang, which will be separate application that will provide development dashboard with look into current traces and metrics. The plan is to provide similar features to the (now abandoned) wobserver on top of OpenCenus library.
Can you guys that write the libraries and wrappers give us here a quick breakdown on which does what? I scanned this thread several times and I am still confused.
I’d like to experiment with metrics and tracing in a hobby project so I’m interested in:
How to use a BEAM library that can register events emitted from both Erlang and Elixir code under the umbrella of the same project.
How to install that in an Elixir Mix project, exactly.
How to emit metrics events.
How to emit tracing events.
How to store those events.
How to query those events, with nice graphs, drill-downs and all the statistics goodies (a la Prometheus or the ELK stack).
Bonus points: some explanations on which Hex dependencies are just interfaces, which are implementations, which comply to which standard etc.
Apologies if this is too noob a question but somewhere along the messages here I got lost trying to understand how to achieve the basics – coming from the standpoint of a guy who never used any metric/tracing events when working with Elixir.
Being ready by early September is likely. We’ve been in a bit of a holding pattern until the new spec and java implementation are near completion (July 6th).
