What are the best practices for persisting telemetry data?

I think PromEx is a good choice.

Hi @akoutmos! Here I am after having added PromEx on my application in a Kubernetes cluster, and I have some feedback. My application uses Ecto, Oban, PhoenixLiveView, and Broadway.

First of all: congrats for pulling off such an ambitious project! It really feels like a lot of work, especially writing the plugins for all those libraries. What I absolutely love about PromEx is that it can automatically upload the dashboards to Grafana, this is really a powerful feature. You deploy your application to a new environment and bam, the dashboards are there! Same if PromEx dashboards are updated. Great idea.

• Because the automatic provisioning of the dashboards is IMO such an awesome feature, I think it should be mentioned more explicitly in the docs. Initially I was thinking “how am I gonna get these amazing dashboards into Grafana?”. After looking at the config values and through other hints in the docs, I eventually realized that they can be automatically uploaded at application startup. In my opinion this should be mentioned explicitly here: README — PromEx v1.12.0
• I wrote a PromEx plugin to expose my application’s specific metrics, and then I wanted to create a dashboard for it and upload it automatically just like PromEx’s default dashboards. The Plugin guide doesn’t mention a way to do that, so I had to figure it out by myself. I ended up adding {:my_app, “path_to_my_dashboard”} to the PromEx.dashboard/0 callback. My opinion is that this callback should be part of the PromEx.Plugin behaviour, so that each plugin can expose its own dashboard(s) and tell PromEx where they are, without listing all of them in the main PromEx module.
• More importantly, I think it would be great to provide a workflow to easily write dashboards templates for a plugin. For my plugin, I put together the dashboard on Grafana, then I downloaded the JSON definition and I manually edited it to create a “json.eex” template. It would be great if we could find a way to automate this process. Something like:
  • Put together the dashboard on Grafana
  • Run a mix task that fetches the dashboard from Grafana via the API and turns it into a “PromEx” template
  • Being able to run the same task after I update a dashboard on Grafana
• It’s not very easy to get the Broadway plugin to work with some real-world producers. The documentation suggests to overwrite the message’s acknowledger through a transformation. While this allows PromEx to see the messages, it also removes the original message’s acknowledger, which is needed by some producers (for example AWS-SQS) to receive confirmation that the message has been handled. The issue is explained here, towards the end and the solution proposed by @dsschneidermann finds me in agreement. I’d love to hear your input and I’d be willing to help with a PR
• Final minor nitpick: the dashboards are using the old Graph panel instead of the new Time series panel. Maybe it’s worth to migrate them.

I hope you found at least some of this feedback helpful. Let me know what you think and how I can provide further help. Thanks again for creating PromEx!

Thanks! It was and remains a lot of work . Glad that it is making things easier for you from a metrics standpoint.

That is a good point. I can probably make this kind of stuff more apparent in the README. I need to work on my copywriting and marketing skills lol. I’ll try and update the docs this week. I will also include my Code BEAM talk in the doc: https://www.youtube.com/watch?v=0SkVsUdUutE

I have been meaning to write some more docs/blog posts around this, but like many things limited time is the killer. More on this at the end of the post.

100% agree with this one. I feel this pain every time I create 1st party dashboards for PromEx. Grafana has also released a lot of new tools and updated Grafana quite a bit since this project started. There is a lot that can be done here now that things have evolved.

Unfortunately I don’t have any production applications where I use Broadway at the moment (I used to), so it is tough for me to feel those production pains. Hopefully people that are using Broadway+PromEx in production can contribute back to the project and help iron out some of these pain points.

This is on the todo list especially now with Grafana 9 being released.

Thanks for the feedback @trisolaran. I genuinely appreciate all the feedback :). I will also add some additional commentary here to provide some context as to when some of these issues/features may be addressed.

Recently, a lot of my spare time outside of FT work, consulting and running a business has been spent on two projects. Those being Elixir Patterns and MjmlEEx. MjmlEEx came about because of some business needs that I had while bootstrapping my business https://eaglemms.com/ (funny enough, PromEx is also open sourced work from the same bootstrapped business haha). MjmlEEx is luckily at a point where it is stable, and has all the features that I wanted out of the library and so I don’t foresee a lot of active development there features wise. I do plan however on creating a Mjml EEx Pro offering that will provide some pre-created email templates as well as some other goodies .

So am I brining this up you may be wondering? To be completely transparent, my hope is that I can follow a similar path with PromEx and create a PromEx Pro offering so that I can buy some of my time back from other ventures and instead devote more time to open source work. There are plenty of stories all over the internet of open source maintainers burning out as a result of not finding a good balance between open source and paying the bills, and I really hope that I never have to write a blog post like that. Hopefully by following in the foot steps of people like Adam Wathan (Tailwind and TailwindUI), Caleb Porzio (great story by him on funding open source) and our very own Parker Selbert (creator of Oban and Oban Pro) I can find a way to devote more of my time to open source initiatives. This is not to say that PromEx is a dead project and that I won’t address the items that you listed (far from it). But rather to say that my development on the project varies depending on external circumstances and funding the project may help increase & normalize the rate at which I can deliver features in the PromEx project.

In addition to those open source balance kind of concerns, there are also some other factors at play that have contributed to me slowing down slightly on PromEx (but this is changing soon). Specifically, there has been a lot that has changed with Grafana as of late and in very good ways. Grafana has recently released a whole slew of open source tools that I would like to leverage more of in future versions of PromEx like their dashboard linter and Tempo. But I want to be a bit cautious here and not adopt these tools prior to them maturing a little. In addition, there has been a lot of developments on the Open Telemetry side of things and I want to make sure that I can properly incorporate that work into PromEx.

To perhaps give people a sneak peek as to what I have planned for PromEx in the near future. I am experimenting with getting PromEx to the point where it incorporates both metrics and traces into a single library so that with the same development experience that you have today with PromEx (i.e creating a single module and adding 1 thing to your supervision tree) you can have metrics and traces as well as exemplars so that you can correlate metrics back to traces (Grafana blog post of metrics/trace correlation).

All this to say that I really appreciate your feedback and will definitely open up some GitHub issues from what you shared to make sure that I don’t forget some of the points that you brought up and that I will hopefully be addressing many of these in the coming months!