Instrumenting my business logic with Prometheus

Prometheus is an external tool, so you send events/metrics to prometheus. You can send metrics from different apps/projects, your metrics endpoint could retrieve data from prometheus to show it to a user? I think prometheus already has a UI/works with grafana so it might be better to just use those?

I am using the prometheus package without a phoenix installation.

The package lets you define, set, update and delete metrics.

It’s useful to look at the docs for the specific metrics types to see how you can use them. E.g. prometheus.ex/lib/prometheus/metric/gauge.ex at master · prometheus-erl/prometheus.ex · GitHub

Not sure we have the same view of Prometheus, you dont send data. Prometheus is polling the metrics endpoint wich contains data from instrumented parts of your app.

Maybe I don’t understand correctly how Prometheus works, but I got from this overview you can send your metrics to a push gateway from which prometheus will pull info? If so, that might make it easier to use? Otherwise you’d have to gather the metrics yourself, store/remember them before prometheus pull that info from your metrics endpoint?

That is only meant for things like queue workers where you want to send metrics of individual jobs.

That is what the prometheus_ex lib does.

I came across a project that might function as an example: GitHub - oestrich/ex_venture: Text based MMORPG engine written in Elixir · GitHub → Exventure - . Hope that helps

Ok interesting, my understanding was that prometheus.ex is only used for instrumenting your code, it does not provide an exporter, that is the metrics endpoint will not be provided by it. For that you have to use som other package that provides the endpoint?

Do you have any github projects where you have this setup?

The package provides everything you need.

So, given you use it in your code to declare and update metrics like this:

Prometheus.Metric.Boolean.declare([name: :my_metric, help: "My Metric", labels: [:label1, :label2], registry: :all])
Prometheus.Metric.Boolean.set([name: :my_metric, labels: ["foo", "bar"], registry: :all], true)

(I use registry: :all here because I don’t need the system metrics that are added by default by prometheus_ex)

You just need plug, or anything else that serves your metrics endpoint. The prometheus package gives you the metrics in the correct form. My code is simply this in a normal plug router.

  get "/metrics" do
    metrics = Prometheus.Format.Text.format(:all)
    send_resp(conn, 200, metrics)
  end

(Here I pass it the :all argument because I pushed my metrics to the :all registry)

So lets say in my lib folder I have 5 projects that all depend on each other, would I need to provide 5 /metrics endpoints on 5 web servers on 5 different ports or could I use one metrics endpoints for all 5 projects? This is sort of my original question.

Do you mean 5 apps in an umbrella setup?

If you don’t have an umbrella setup and just one elixir app, there is nothing special to do.

If you have an umbrella setup with different apps, I would suggest that you create another app just for the metrics. That metrics app has the dependency to prometheus_ex and exposes functions for your other apps to add and update their metrics. That way, your other apps can have the metrics app as dependency and push their metrics to it. That metrics app also exposes Prometheus.Format.Text.format which is important for your web facing app to get the metrics for the endpoint. The point is that you have a central place for the metrics so you don’t end up having one metrics list per app which is not very practical in most cases.