Hello everyone,
I hope you are all doing well. I’m seeking some guidance regarding an API that I plan to integrate with PromEX. This API is deployed for various customers, so we have customer1.api.com, http://customer2.api.com, etc. While the code remains identical, it is deployed in different environments but using the same prom/grafana instance. I would like to add custom labels to the metrics to filter them by customer, such as customer=customer1/2/3. Is there any way to do that easily?
Thank you in advance!
Depending on how many customers you have, I would caution you against adding a label for the customer as each unique customer label value increases the number of time series that you will be storing in Prometheus. It sounds like you also want to add this additional label to all of the PromEx metrics and that for sure will generate a lot of additional time series (see the Prometheus docs on this exact topic). If you are in Grafana Cloud, Grafana charges you per time series so you can expect your bill to sky rocket as well.
I would instead lean on your other observability tools to get specifics like customer/user/org/etc and keep prometheus for the high level metrics. If your metrics can surface that a particular endpoint is slow, it’ll be relatively slow for everyone.
2 Likes
Thank you @akoutmos ! You are right, it’s not a good idea. Ill explore other ways of doing it.
