Future of Logger in Elixir

Interested in what you put together if you can share? I’m currently working on Logflare and we actually need beta testers for our new logger backend.

I really like being able to keep the raw event data as you can drill into it vs the time series approach. As long as your db is fast enough to query, you’re able to insert in real-time and you’re happy with the visualization / BI tools available … I think the time series approach is less favorable.

With our Logger backend you can structure logs and add any data you want. Logflare uses BigQuery as the db and manages the BigQuery schema so you can easily query each field specifically. And BigQuery plus Data Studio is very nice although I wish it would give me minutely graphs out of the box.

I do not quite get what is the problem here. Erlang’s logger allows to use structured logging, so instead of (in Elixir syntax) doing Logger.info("Responded HTTP 200 in #{time}") you can do: Logger.info(%{http_response: 200, http_response_time: time}) and at the same time provide more information in metadata. This allows to eat cookie and have cookie because:

• you can programmatically get meaningful data without processing text
• you still can insert metadata (that aren’t part of the log message) into metadata

Grafana describes difference quite nicely, so you can see that in most (all?) cases metrics can be extracted from logs. Of course some of these logs will need to be discarded to not needlessly increase logs volume, but that can be done quite easily later.

Structured logging is great and I hope it sees wide adoption.

There are at least a couple situations that would motivate me to use something like telemetry as well.

1. High-level, aggregate metrics often help people intuit what’s going on in a way fine-grain, detailed log messages don’t.
2. Logging incurs a higher latency and storage penalty than metrics. Consider what happens when any part of the structured log is a large data structure that must be serialized and perhaps truncated.

Once I helped investigate why a production system was suffering occasional latency spikes. The problem was one log message sometimes emitted lines that were tens of megabytes long.

That’s a good point. For production services I can access I want the ability to aggregate and view real-time metrics without relying on logging overhead, and additionally logs for when I need to drill deeper. For services and other programs which I can’t access, but can perhaps acquire logs through support channels I want the ability to feed the log into the metrics pipeline and get fancy charts.

Yes I think we’re in agreement here

I’ve found Elastic based solutions to be prohibitively expensive at any sort of decent volume. With something like BigQuery as your db you can keep orders of magnitude more event data and still query quickly. Logflare and our logger backend lets you get that structured data into BigQuery easily.

I agree. I am part of the EEF Observability WG and my question isn’t whether we should rely only on the logger but whether we should include metrics in logs and then provide an logger handler that will extract these from logs and aggregate them. So everything would happen without sending high volume messages to external log handler.

So in the end there still would be some kind of aggregate that would reduce volume, so even when there happen error that is gigabyte long it shouldn’t be a problem, as most of the data would be discarded by handler anyway.

So my question is mostly about libraries like Ecto which use telemetry right now. Would it be better to keep telemetry or maybe use structured logs that could provide the same informations.

Congratulations! I’m happy about EEF’s formation and that there’s working groups.

For context I’ve only recently started using BEAM-based languages again, and still need to get up to speed with OTP’s logger, telemetry, and Ecto.

That sounds like a useful option, especially for situations when doing remote software support based on provided logs. Do you anticipate the use case requiring any changes to logger’s front-end API? Would there be some relationship between the logger backends you’re proposing and telemetry metrics?

My general stance is it’s nice to give developers options which cover their common use cases. Is there a sensible common interface people could opt-into, and then configure whether telemetry, logging, or both would aggregate metrics?

If they’re just too different for a common abstraction layer, I’m inclined–from my admittedly naive standpoint–to want common library projects such as Ecto using both telemetry and structured logging, while giving application developers the ability to configure each according to their needs.

Maybe others more experienced with these tools than me can offer more specific feedback?

No. I do not see any reason for such changes.

Current Telemetry.Metrics would need to be implemented by the backend that would use logs to aggregate metrics. So in theory we could have shared API for defining aggregations, but I do not think that it would be needed in that extent.

I am also working on OpenCensus (to be renamed to OpenTelemetry) integrations, so this is how I see the API to be done.

The API is almost identical as Telemetry has execute/2-3 function and in case of metrics it would be one of the macros (probably on notice level as info is filtered out by default) and if you look closely then you will see that the arguments (in case of structured logs) are exactly the same in both cases. The only difference is how you digest the data, but TBH this is matter that interest only integration authors, not users.

While I am not at all an expert in any of this, structured logging has been something I wanted ever since back in my Java days (2002 and on). I see no reason why we must always dump a ton of metadata in a hard-to-parse format like the text; formats change, new team leads or CTOs come, policies change and in the end I’ve had to maintain a system that parsed 11 different log formats for 4 years.

We can do better.

OpenCensus => OpenTelemetry sounds very good as well.

Sounds like you are a fan of systemd instead of sysinitv then? ^.^

When the crap-show there started (good old “us vs. them” style) I completely stopped paying attention. I have to say nowadays I remain blissfully ignorant on the topic.

I will be grateful to be enlightened though.