Connecting prom_ex to grafana.com

Questions / Help
, ,
1

I tried to set up a Grafana/Prometheus dashboard with prom_ex and a free account at grafana.com and let me tell you upfront that I’m a total idiot when it comes to these things. So, it might not surprise you that I cannot find out which values to put in the grafana-config of prom_ex even if my life depended on it. I put the following config:

# config.exs
config :my_app, MyApp.PromEx, disabled: true

# prod.exs
config :my_app, MyApp.PromEx,
  disabled: false,
  manual_metrics_start_delay: :no_delay,
  drop_metrics_groups: [],
  grafana: [
    host: System.get_env("GRAFANA_HOST"),
    username: System.get_env("GRAFANA_USER"),
    password: System.get_env("GRAFANA_PASSWORD")
  ],
  metrics_server: :disabled

Now, if I click through the myriad of config sites on https://some-name-some-numbers.grafana.net I find values like this:

URL: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom
User: 1234567 # Some number
Password: my-secret-password # I set a password here

I found these by clicking on the prometheus-SomeNameSomeNumbers-prom data source in the Settings-tab.

I put these env variables, but when I start my server, I get the following logs:

Recieved a 404 from Grafana because: %Finch.Response{body: "404 page not found\n", ...}

So clearly, one of the credentials above is wrong. Could somebody with more brain matter help me out please?

PS: Also, what is the datasource_id I have to set in PromEx? Must it be prometheus-SomeNameSomeNumbers-prom?

Ping @akoutmos

2

Hey Peter!

These are all good questions, and to be honest, it is something that I would love to get some guides around…but alas, limited time :frowning: .

Luckily I use Grafana Cloud, so I know roughly where you are going wrong here :slight_smile:. The Prometheus credentials that you are referring to are used when you are using Prometheus’s remote_write functionality. By default Prometheus will scrape metrics from your /metrics endpoint, but with Grafana cloud, that is not an option. Instead they want you to go down the route of pushing metrics via Grafana Agent (I have an issue for this in PromEx to make it easier for people Add support to bundle GrafanaAgent to push metrics · Issue #44 · akoutmos/prom_ex · GitHub, but I haven’t had a chance to wrap it up). How are you running your app? Is it easy for you to run Grafana Agent along side it?

In addition, you should be making an access token in Grafana (go to your instance’s /org/apikeys route to do so) and using the :auth_token value in PromEx so that PromEx can publish dashboards for you. The :host value should just be the url of your Grafana Cloud instance.

Hopefully that help :slight_smile:

1 Like
3

Hey Alex!

Thank you very much for that elaborate description :slight_smile: Now I understand prom_ex and Grafana much better :mechanical_arm:

I run my app on Gigalixir and to be honest, I don’t feel like adding a Port or similar to it “only” to use Grafana Cloud :confused: I will have a look around for other providers (can you recommend one where the config is simpler?)

I wonder though: Why is pushing info to Grafana so different from e.g. pushing Logs to an external service? Can’t they “simply” offer an API-endpoint to which prom_ex pushes updates? Or is the amount of data prohibiting such an approach? :thinking:

Also: Once I get prom_ex up and running with another service, I will be very happy to write a blog post about it :slight_smile:

4

I attempted to use the AWS hosted Prometheus solution yesterday, but they also relied on remote_write, which to my understanding is not yet supported by prom_ex, correct? :confused:

I looked for other services that offer a hosted grafana, but all of them require a monthly fee without offering a free tier for experimenting. The two with a free tier (Grafana Cloud and AWS) rely on remote_write unfortunately. I’m afraid that I hit a bit of a blocker here :frowning_face:

5

Supporting remote_write directly within PromEx itself would be a huge undertaking and, to be completely honest, not something that I can do unless PromEx becomes an income source for me just due to time limitations.

That said, integrating GrafanaAgent into PromEx is something that is doable, and I have a working version already completed and running in production. I need to do some clean up around the implementation and get it to a point where it is seamlessly added to PromEx, but that will be a viable route and should be coming soon.

In addition, with GrafanaAgent running via a port under PromEx, you can also use GrafanaAgent to ship logs to loki and traces to tempo. All things that are on my PromEx todo list :slight_smile: