Connecting prom_ex to grafana.com

PJUllrich

Author of Building Table Views with Phoenix LiveView

I tried to set up a Grafana/Prometheus dashboard with prom_ex and a free account at grafana.com and let me tell you upfront that I’m a total idiot when it comes to these things. So, it might not surprise you that I cannot find out which values to put in the grafana-config of prom_ex even if my life depended on it. I put the following config:

# config.exs
config :my_app, MyApp.PromEx, disabled: true

# prod.exs
config :my_app, MyApp.PromEx,
  disabled: false,
  manual_metrics_start_delay: :no_delay,
  drop_metrics_groups: [],
  grafana: [
    host: System.get_env("GRAFANA_HOST"),
    username: System.get_env("GRAFANA_USER"),
    password: System.get_env("GRAFANA_PASSWORD")
  ],
  metrics_server: :disabled

Now, if I click through the myriad of config sites on https://some-name-some-numbers.grafana.net I find values like this:

URL: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom
User: 1234567 # Some number
Password: my-secret-password # I set a password here

I found these by clicking on the prometheus-SomeNameSomeNumbers-prom data source in the Settings-tab.

I put these env variables, but when I start my server, I get the following logs:

Recieved a 404 from Grafana because: %Finch.Response{body: "404 page not found\n", ...}

So clearly, one of the credentials above is wrong. Could somebody with more brain matter help me out please?

PS: Also, what is the datasource_id I have to set in PromEx? Must it be prometheus-SomeNameSomeNumbers-prom?

Ping @akoutmos

6 comments

#prometheus #grafana #prom_ex

1 2188 6

2025-01-03 05:36:43 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

akoutmos

Author of Build a Weather Station with Elixir and Nerves

Hey Peter!

These are all good questions, and to be honest, it is something that I would love to get some guides around…but alas, limited time .

Luckily I use Grafana Cloud, so I know roughly where you are going wrong here . The Prometheus credentials that you are referring to are used when you are using Prometheus’s remote_write functionality. By default Prometheus will scrape metrics from your /metrics endpoint, but with Grafana cloud, that is not an option. Instead they want you to go down the route of pushing metrics via Grafana Agent (I have an issue for this in PromEx to make it easier for people Add support to bundle GrafanaAgent to push metrics · Issue #44 · akoutmos/prom_ex · GitHub, but I haven’t had a chance to wrap it up). How are you running your app? Is it easy for you to run Grafana Agent along side it?

In addition, you should be making an access token in Grafana (go to your instance’s /org/apikeys route to do so) and using the :auth_token value in PromEx so that PromEx can publish dashboards for you. The :host value should just be the url of your Grafana Cloud instance.

Hopefully that help

Post #1

PJUllrich

Author of Building Table Views with Phoenix LiveView

Hey Alex!

Thank you very much for that elaborate description Now I understand prom_ex and Grafana much better

I run my app on Gigalixir and to be honest, I don’t feel like adding a Port or similar to it “only” to use Grafana Cloud I will have a look around for other providers (can you recommend one where the config is simpler?)

I wonder though: Why is pushing info to Grafana so different from e.g. pushing Logs to an external service? Can’t they “simply” offer an API-endpoint to which prom_ex pushes updates? Or is the amount of data prohibiting such an approach?

Also: Once I get prom_ex up and running with another service, I will be very happy to write a blog post about it

Post #2

PJUllrich

Author of Building Table Views with Phoenix LiveView

I attempted to use the AWS hosted Prometheus solution yesterday, but they also relied on remote_write, which to my understanding is not yet supported by prom_ex, correct?

I looked for other services that offer a hosted grafana, but all of them require a monthly fee without offering a free tier for experimenting. The two with a free tier (Grafana Cloud and AWS) rely on remote_write unfortunately. I’m afraid that I hit a bit of a blocker here

Post #3

akoutmos

Author of Build a Weather Station with Elixir and Nerves

Supporting remote_write directly within PromEx itself would be a huge undertaking and, to be completely honest, not something that I can do unless PromEx becomes an income source for me just due to time limitations.

That said, integrating GrafanaAgent into PromEx is something that is doable, and I have a working version already completed and running in production. I need to do some clean up around the implementation and get it to a point where it is seamlessly added to PromEx, but that will be a viable route and should be coming soon.

In addition, with GrafanaAgent running via a port under PromEx, you can also use GrafanaAgent to ship logs to loki and traces to tempo. All things that are on my PromEx todo list

Post #4

mukhtarkv

@akoutmos thank you for your hard work on PromEx library which I am currently using. I wonder is there a charge on Grafana Cloud Dashboard HTTP API calls? I tried to search on the pricing but it is quite vague. I intend to run Elixir application in production with more than 1000 container instances and I am concerned that all 1000 Elixir containers might be incurring dashboard upload api charge.

Post #5

akoutmos

Author of Build a Weather Station with Elixir and Nerves

I don’t think Grafana charges you for API calls…but don’t quote me on that lol. If you have that many instances running simultaneously you may want to consider only having a single instance deploying the dashboards. If you can, have your runtime.exs file set upload_dashboards_on_start: false on all instances except for one or something like that.

Post #6

— All posts loaded —