Instrumenting my business logic with Prometheus

Phoenix Forum Questions / Help
,
1

I have what I thinks is a standar layout of a Phoenix project, in my lib folder I have two projects app_core and app_interface. In app_interface I have the dependency to phoenix and following this post I also have the dependency to

{:prometheus_ex, "~> 1.0"},
{:prometheus_plugs, "~> 1.0"}

Given this setup my Phoenix project will expose an metrics endpoint.

Now the problem I have is that I want to instrument the app_core project and have those metrics made available on the metrics endpoint in the app_interface project.

Is this even possible or how are you guys using metrics with the business logic side of a Phoneix project???

2

Prometheus is an external tool, so you send events/metrics to prometheus. You can send metrics from different apps/projects, your metrics endpoint could retrieve data from prometheus to show it to a user? I think prometheus already has a UI/works with grafana so it might be better to just use those?

1 Like
3

I am using the prometheus package without a phoenix installation.

The package lets you define, set, update and delete metrics.

It’s useful to look at the docs for the specific metrics types to see how you can use them. E.g. https://github.com/deadtrickster/prometheus.ex/blob/master/lib/prometheus/metric/gauge.ex

4

Not sure we have the same view of Prometheus, you dont send data. Prometheus is polling the metrics endpoint wich contains data from instrumented parts of your app.

5

Maybe I don’t understand correctly how Prometheus works, but I got from this overview you can send your metrics to a push gateway from which prometheus will pull info? If so, that might make it easier to use? Otherwise you’d have to gather the metrics yourself, store/remember them before prometheus pull that info from your metrics endpoint?

1 Like
6

That is only meant for things like queue workers where you want to send metrics of individual jobs.

That is what the prometheus_ex lib does.

2 Likes
7

I came across a project that might function as an example: https://github.com/oestrich/ex_venture -> https://exventure.org/metrics/ . Hope that helps :slight_smile:

8

Ok interesting, my understanding was that prometheus.ex is only used for instrumenting your code, it does not provide an exporter, that is the metrics endpoint will not be provided by it. For that you have to use som other package that provides the endpoint?

Do you have any github projects where you have this setup?

9

The package provides everything you need.

So, given you use it in your code to declare and update metrics like this:

Prometheus.Metric.Boolean.declare([name: :my_metric, help: "My Metric", labels: [:label1, :label2], registry: :all])
Prometheus.Metric.Boolean.set([name: :my_metric, labels: ["foo", "bar"], registry: :all], true)

(I use registry: :all here because I don’t need the system metrics that are added by default by prometheus_ex)

You just need plug, or anything else that serves your metrics endpoint. The prometheus package gives you the metrics in the correct form. My code is simply this in a normal plug router.

  get "/metrics" do
    metrics = Prometheus.Format.Text.format(:all)
    send_resp(conn, 200, metrics)
  end

(Here I pass it the :all argument because I pushed my metrics to the :all registry)

3 Likes
10

So lets say in my lib folder I have 5 projects that all depend on each other, would I need to provide 5 /metrics endpoints on 5 web servers on 5 different ports or could I use one metrics endpoints for all 5 projects? This is sort of my original question.

11

Do you mean 5 apps in an umbrella setup?

If you don’t have an umbrella setup and just one elixir app, there is nothing special to do.

If you have an umbrella setup with different apps, I would suggest that you create another app just for the metrics. That metrics app has the dependency to prometheus_ex and exposes functions for your other apps to add and update their metrics. That way, your other apps can have the metrics app as dependency and push their metrics to it. That metrics app also exposes Prometheus.Format.Text.format which is important for your web facing app to get the metrics for the endpoint. The point is that you have a central place for the metrics so you don’t end up having one metrics list per app which is not very practical in most cases.

3 Likes
12

Just thinking, prometheus_ex uses ETS to store the metrics, so you might not even need an additional metrics app in an umbrella setup. I can’t say for sure but if all apps in the umbrella setup use the prometheus_ex package directly, they should all use the same ETS tables and therefore have a shared metrics list.

13

I use prometheus.ex in a very similar way but I use the Prometheus.Metric.Summary type of metric:

    Summary.declare(
      name: :vm_check_duration_seconds,
      help: "VM Check duration in seconds",
      labels: [:tenant, :hypervisor],
    )

When I later call Prometheus.Format.Text.format I’m getting the _sum and _count metrics tagged with the labels correctly.

However, I’m not getting the quantiles. Does anyone know how to enable quantiles?

14

quantiles just aren’t implemented (yet). I don’t use it because they do not aggregate. PR is welcome though.

15

Thanks for answering my question.
I took a quick look at the source code. Although I’d love to contribute and learn Elixir better, I see a lot of meta-programming in the code which I don’t understand yet so I’m not able to contribute at this time especially a large feature like quantiles.
I hope to keep learning Elixir so maybe one day I will know enough to contribute.