PromEx - Prometheus metrics and Grafana dashboards for all of your favorite Elixir libraries

akoutmos · February 18, 2021, 10:40pm

Prometheus metrics and Grafana dashboards for all of your favorite Elixir libraries

I have been putting off creating an ElixirForum post about PromEx until it hit the coveted 1.0.0 mark. But that time has finally come! PromEx is a metrics framework that ties together all of the BEAM Telemetry libraries in a simple to use package. It comes with plugins for all of the popular Elixir ecosystem libraries including Phoenix, Ecto, LiveView, Oban, the BEAM itself, and may more coming soon. Each PromEx plugin also comes with an accompanying Grafana dashboard which PromEx will automatically upload for you on application start. PromEx will also annotate all of your PromEx Grafana dashboards so that you know when application instances come up and go down:

PromEx is extensible in that it also allows you to write you own plugins and dashboards to support gathering metrics specific to you application.

Check out the thorough Hex Docs to learn more about PromEx Contents — PromEx v1.0.0
And Check out the snapshots of the dashboards if you are curious what you get out of the box Dashboards Screenshots — PromEx v1.0.0

Exadra37 · February 19, 2021, 11:00am

Thanks for this awesome library

I gave a quick look into the docs and it looks like the endpoint to get the metrics is public, instead of private, aka is not protected with a token or behind authentication. Am I wrong in my assumption?

akoutmos · February 19, 2021, 1:30pm

Great question! The metrics plug that comes with PromEx has no authorization configuration available since I have another library that I maintain that deals with things like that. I will probably pull in authorization into the builtin plug since it is a common request in the near future. But for now I recommend people use Unplug which can unhook plugs from your plug pipeline.

PromEx is currently capturing metrics for The Changelog and you can look at how they secure the metrics endpoint using Unplug:

github.com

thechangelog/changelog.com/blob/master/lib/changelog_web/endpoint.ex#L53-L55


plug Unplug,
  if: ChangelogWeb.Plugs.MetricsPredicate,
  do: {PromEx.Plug, prom_ex_module: Changelog.PromEx}

github.com

thechangelog/changelog.com/blob/master/lib/changelog_web/plugs/metrics_predicate.ex

defmodule ChangelogWeb.Plugs.MetricsPredicate do
  @moduledoc """
  This Unplug predicate is used to authorize requests to the PromEx metrics
  """

  @behaviour Unplug.Predicate

  @impl true
  def call(conn, _) do
    conn
    |> Plug.Conn.get_req_header("authorization")
    |> case do
      ["Bearer " <> token] ->
        token == System.get_env("PROMETHEUS_AUTHORIZATION")

      _ ->
        false
    end
  end
end

Exadra37 · February 19, 2021, 2:04pm

First, I want you to know that I work as a Developer Advocate in API Security, just to put you in context why I am so picky with insecure software by default.

In my opinion developers of any of software should consider security as op-out not opt-in.

Users of any software, be them also developers or end users should not have to learn how to secure them, instead they should have to learn how to disable security if that is really what they want.

But at least I am glad that you are open to make it secure by default, aka security as opt-out.

The docs should make very clear, and in BOLD, that the endpoints are public and not protected. Remember that attackers leverage all they can to build attacks based on chains of everything that they can leverage from the target, be them metrics, logs, too much info in the request response, etc.

akoutmos · February 19, 2021, 3:43pm

Totally see where you are coming from on this and honestly I have very similar feelings.

Unfortunately, Prometheus has security as opt-in (i.e you can configure a scrape target without specifying any kind of auth configuration), and in most of the production deployments I have seen, prom, grafana and the scrapable apps are all behind a load balance in their own VPN and metrics endpoints are usually filtered out at the load balancer layer and wide open on the intranet.

I wanted to make sure that PromEx didn’t introduce a barrier to entry by forcing authorization, hence my decision not to have auth as a default (for now at least). I cover authorization in the PromEx.Plug documentation (PromEx.Plug — PromEx v1.0.0) but agree that I should probably have a section right in the readme to address security concerns just like i had a section to address performance overhead concerns.

Thanks for the feedback! Appreciate it

Exadra37 · February 19, 2021, 3:59pm

That['s exactly the mentality that needs to change in our industry and while it prevails we will never have a more secure internet.

You even have a search engine to find all software exposed to the internet:

https://shodan.io

Search for prometheus:

https://www.shodan.io/search?query=prometheus

Search for Grafana:

https://www.shodan.io/search?query=grafana

In the results just check the ones that return a 200 response and they are some that are just opne, and that happens because software by default is insecure, instead of being secure.

And why not making this plug required in the installation of PromEx and then have this section showing how to remove it for the ones that don’t want security by default?

akoutmos · February 19, 2021, 9:01pm

Not all applications that expose metrics leverage Phoenix (think like a Broadway+RabbitMQ job worker). As a result it wouldn’t be feasible to couple the metrics to the Plug.

Exadra37 · February 19, 2021, 9:25pm

I don’t know your library in detail, but if exposes an endpoint to the public then I am of the opinion that it should be secure.

Now, when consumed internally by another code, the obviously it doesn’t need to perform authentication.

akoutmos · March 5, 2021, 7:03pm

Version 1.0.1 of PromEx has been published to Hex

This release includes mostly documentation fixes and a minor bug fix to the Oban plugin. Give it a whirl!

arton · May 18, 2021, 6:23pm

@akoutmos Thanks for the awesome library! I’m replacing our previous prometheus setup with it, but I’m running into some odd failing tests in various areas (Ecto and Phoenix LiveView for example). Is it possible to disable PromEx during testing?

arton · May 18, 2021, 9:31pm

@akoutmos After further investigation, the test failures seem to be caused by some other dependency updates I applied when installing PromEx, so don’t seem to be related at all. Thanks for all your hard work on this excellent library!

akoutmos · June 18, 2021, 7:17pm

Version 1.2.1 of PromEx has been published to Hex

This release contains an important bug fix for Phoenix applications using the forward macro.

akoutmos · July 2, 2021, 5:06pm

Version 1.3.0 of PromEx has been published to Hex

This release contains:

The new Absinthe plugin and dashboard
BEAM plugin updates to capture metrics for persistent_term
A bug fix for later version of LiveView where the LiveView module could not be resolved (backwards compatible)

akoutmos · September 1, 2021, 4:43am

Version 1.4.0 of PromEx has been published to Hex.

Changelog for this version:

Changed

The Phoenix plugin now requires an :endpoint configuration option to be passed to it containing the module
for which metrics will be captured.

Added

Plug.Router plugin and dashboard.
PlugCowboy plugin and dashboard.
Phoenix plugin now supports multiple routers and multiple endpoints.
Phoenix plugin and dashboards now contains endpoint configuration data.
Phoenix plugin now captures socket metrics (dashboard not yet updated though).
Ecto plugin captures total_time metrics (dashboard not yet updated though).
Add an optional configuration to dashboard renderer and each plugin so that the metrics_prefix can be altered.

Fixed

Oban dashboard overview stat panels.

akoutmos · September 6, 2021, 9:22pm

Version 1.4.1 of PromEx has been published to Hex

Changelog for this version:

Added

Added a configuration to the dashboard assigns so that the default time interval can be specified
by the user as opposed to being hard coded to 30s.

Fixed

Fixed Plug.Router plugin to handle requests without conn.private.plug_route info