Structured logging is great and I hope it sees wide adoption.
There are at least a couple situations that would motivate me to use something like telemetry as well.
- High-level, aggregate metrics often help people intuit what’s going on in a way fine-grain, detailed log messages don’t.
- Logging incurs a higher latency and storage penalty than metrics. Consider what happens when any part of the structured log is a large data structure that must be serialized and perhaps truncated.
Once I helped investigate why a production system was suffering occasional latency spikes. The problem was one log message sometimes emitted lines that were tens of megabytes long.
That’s a good point. For production services I can access I want the ability to aggregate and view real-time metrics without relying on logging overhead, and additionally logs for when I need to drill deeper. For services and other programs which I can’t access, but can perhaps acquire logs through support channels I want the ability to feed the log into the metrics pipeline and get fancy charts.
