New to Elixir, trying to explore the language and idiomatic problem solving, mostly for fun and personal projects.
As a learning project, i’m trying to setup a dynamic HTTP server in order to play around with networking capabilities, concurrency and such.
I’ve got a basic HTTP server setup using Plug & Cowboy(directly), and got to have a simple streaming endpoint working.
Now, I’m wanting to expose internal log streams, and later on monitoring metrics, through HTTP endpoints.
That is, I want to hook into the logging subsystem of my application(or whatever process supervisor is shepherding everything), and get a hold of the live stream of log messages and expose them to the HTTP endpoint.
From my understanding, logs are managed through a process tree, consisting of one or many logging backends processing the log messages sent to that process group.
Am I right that in order to do what I want, I would have to
Implement and configure a Logging backend(or perhaps a handler?) that implements all the proper interfaces to receive log messages through the Logging subsystem and distribute them to other interested processes(through some sort of internal PubSub protocol)
Each time a connection is established to the endpoint, the request handler process subscribes to the custom logging backend and streams back logging messages to the http requester as they are received.
So this can be done in two parts, the logging backend, and the http endpoint.
Is there something that can already do this kind of internal log routing and minimize the work needed for the Logging backend part?
Are there simple examples of Logging backends and internal pub-sub/router processes that can guide me a bit?
Thank you for your response! Yes, telemetry is something that looks interesting for what I’m looking to achieve later on. I do want to capture metrics as well.
That telemetry library provides an API to synchronously execute handlers on each event generated.
I suppose I could provide a handler that sends the events directly to the HTTP worker serving the stream. Though I’m wondering about performance implications, say if I have a million HTTP workers serving clients listening to the /metrics endpoint. That means just as many handlers being executed on each event being generated, in the context where the event is generated, spreading the load all over the app. Though those handlers would not be blocking, only dispatching asynchronous messages to other processes, so perhaps that’s as quick as can be, but seems there’s still wasted effort there.
I’m curious as to where such events are normally dispatched to? Wouldn’t there be a telemetry process tree handling all those events generated throughout the application modules, which can scale as necessary? And then, why install handlers directly on the event producer? Why not install handlers in the context of that centralized telemetry process? Somewhat like I’m suggesting for logging(basically the same thing, telemetry events and logs).
This may not be what you’re looking for but I created my own Logging backend and Log Formatter, all of the logs in the Phoenix system go to the Logging Backend and then the Log Formatter decides what to do with that log depending on the metadata that I provide it, e.g.: write to file, send somewhere else etc etc…
Yes I would be interested in your logging backend as an example i could learn from!
Do you have it hosted on an open source repo? Do you have a link for it?
I believe that RingLogger is exactly what you’re looking for:
You can run RingLogger.next to get the recent logs and then RingLogger.attach to stream incoming logs to the current processes group leader. You may need to do something special to receive the logs as a data structure but I think this should help get you most of the way there.
