Well, it is important to remember that these are still logs not metrics. So while nice addition, in real systems I would beware of using such setup as it can easily fill whole IO with “metrics” alone leaving no bandwidth for the “useful work”. If you want something like that then you should use local aggregator to not overload network with sending all that events one by one.

It is like pyramid, you measure what you can (metrics), you message where you need some debugging (logs), and trace on the high level. So you end with a lot of metrics, some logs, and few traces. Often you even implement sampling on traces to reduce amount of them, because as you noticed, some traces can be huge.

Each of these is also useful in different situations:

• metrics are used for “early warning”, we want to know what is going on and be able to react before there is a problem
• logs are used to find out where is the problem, check out for bugs and sometimes for other finding malicious parties (fail2ban)
• traces are used for profiling applications, finding bottlenecks, and monitoring how services interact with each other

So you see why it is often important to differentiate between them.

In short:

• Metrics tell you when is something happening in your application
• Logs tell you what is happening in your application
• Traces tell you why and how is something happening in your application

Sometimes it help, sometimes it doesn’t.

I would say that it depends on the amount of data you want to gather. Often, with broad monitoring, it will come to you sooner than later, even before “reaching scale”.

Great.

I meant systems with many metrics and heavy traffic. If we are monitoring only for the HTTP requests then often it will be enough, however, as in article I linked, even if you batch them you are limited by size of each log. See that each one log entry will contain about 28 bytes of data that isn’t really needed there (timestamp) as we are more interested about rate of the events, not exact time when these events happen.

Yeah, for starting projects it may be useful and interesting solution, however if you grow at least a little then you may encounter some problems (AFAIK, please correct me if I am wrong)

• Logflare UI do not support comparing and looking for correlations in graphs of the metrics
• There is no way to do more complex analysis like counting derivations, computing trends, etc. in the Logflare queries
• There is no alerting mechanism built into Logflare, which is the reason for using metrics
• There is only one graph in the Logflare UI - bar graph, which shows only rate of the events which is useful, but sometimes you need other graphs (heat maps, gauges, etc.). I do not really see how you would check CPU or memory usage using such UI

So as I said, it is useful, but you will very quickly grow out of it and you will need a “real” metrics gathering setup.

Sure sure, yeah and there’s no reason why you can’t aggregate metrics on the client and then send them over when you need to.

We have solutions of your examples I think. There is definitely work to do here on the features themselves and surfacing the more advanced use cases.

With BigQuery and the way we implemented it, you get Google Data Studio for free. While not perfect it does let you build out any kind of dashboard you can think of. We will be adding some initial dashboarding features though too.

Mostly true but because we’ve setup BigQuery the way we have if you’re on a paid account you can query your data directly with SQL and include that in your Data Studio reports.

With LQL … you can do like m.inventory.product.color:~"blue|black" and it’ll give you a bar chart of events where the product is blue or black via a regex match. You can then do like m.inventory.product.color:~"blue|black" m.inventory.product.status:"sold" c:sum(m.inventory.product.qty) and the chart will be a timeseries of the sum blue or black sold products moving through your inventory.

You can also use the same LQL queries to route logs to another source and alert when that source gets events. Some examples would be:

m.vm.memory.last_values.total:>8000000000 and you’ll get alerts when you ram usage is over 8GB
m.phx.status:>499 to get alerted about all your 5xx responses
m.level:"error" on a LogflareLogger source and get alerted for all your error logs
m.phx.request_path:~"/signup" and regex match looking for paths with /signup in them for signup alerts
m.phx.endpoint.stop.every.duration:>5000000000 to get all requests over 5 seconds
m.repo_name.repo.query.every.total_time:>5000000000 all ecto queries over 5 seconds