Systemd - Erlang library for communicating with (surprise) systemd

Version 0.3.0 released:

This update adds support for systemd’s journal, which makes library feature complete for now. Now it needs a little polishing and real-world usage before I mark it as a 1.0.0.

Again - feel free to test this out and to submit issues if any. If someone needs any additional feature, then also feel free to submit it, however for me it is (as said above) feature complete.

I have also prepared simple Elixir application that uses this library:

https://github.com/hauleth/plug_systemd_example

Unfortunately, as mentioned in original post, it cannot use socket activation (yet), but rest of features is implemented there.

Ooo, this looks useful!! Thanks for it!!

There are still some bugs there and here (mostly in biggest part of the repo which is systemd_journal_formatter) with fixes on master (I will publish 0.3.3 soon).

Additionally I have finally got socket activation working. Unfortunately due to ERL-1138 the passed FDs can be used only in UNIX sockets (as these do not bind(3) and this is the culprit there), so I just have done some “socket jumping” and translate TCP port to UNIX Stream socket and then to the Cowboy.

So in other words there are 5 systemd units:

• example-application.service - which is THE application service file, it will just simply run the application when started
• example-application.socket - UNIX Stream socket definition that will be handled by systemd, this is required due to the above bug; this will start example-application.service on any connection incoming
• example-activation.service - service that will take the systemd activated FD and forward all it all to another socket provided as an argument, think. of it as an alternative implementation of socat
• example-activation.socket - definition of TCP socket that systemd will listen on and will pass it down when there will be any request incoming
• example.target - simple target for ease of use

Within the application the magic happens there:

https://github.com/hauleth/plug_systemd_example/blob/1a03a50b461f3657870489523e9cd3302eec2a45/lib/plug_systemd_example/application.ex#L34

It creates new gen_tcp socket from the provided file descriptor and then pass it to the Ranch (via Cowboy). The only problem is that currently each first request (when service is down) will fail with no response sent, but all consecutive requests will work as expected.

Hi,

I use systemd and saw this but could you expand on the normal use case where you would use this and how? My interpretation is that instead of running the usual systemctl commands at the OS level you run them from inside the VM itself?

It depends, because it provides few features. I tried to describe them the best I could in the docs (if you find something unclear or missing, then open the issue, I will try my best to fix it).

The features are as follows:

Notify socket

This is used to inform systemd about state of your application. So if you have NotifyAccess=main or NotifyAccess=all then you can for example notify systemd when your application is truly ready (especially when you will also use Type=notify). Erlang applications rarely are ready to work just when VM start and there is need for some setup which happens in the supervisor (DB connection, HTTP server setup, etc.) and this can take some time as well. So with this library you can just call :systemd.notify(:ready) when everything is done and systemctl will block until it receives that message. So when systemctl start my_app.service ends you are sure that your app is actually ready.

Additionally to that also will receive information when the application will be shutting down, but not yet completely dead. So for example if you use Plug.Cowboy.Drainer in your application, which will hold the application from shutting down until all HTTP requests will not be served, then you will have information about application shutting down in systemctl status my_app.service, like this:

https://asciinema.org/a/jqTbUdgFkc7206vFK4AScqq5p

You can also send other notifications but I am not truly sure how these are handled.

In future, when ERL-1139 and ERL-1138 will land, it will also be possible to “save” sockets between restarts, so your application will be able to restart without closing existing connections.

Watchdog

Watchdog is heart-beat systemd subsystem that expects to receive special messages within specified timespan to make sure that application is alive. This is controlled by WatchdogSec= option in [Service] section of systemd unit. This option will be handled automatically for you (so the pinging process is automatically handled), but you still have an option to call :systemd.watchdog(:trigger) which will send special triggering message which will work like there would be no keep-alive message sent within requested timespan. This will trigger action that depends on Restart= option. More can be found in systemd.service(5) and sd_watchdog_enabled(3).

Listen FDs

This is just the function that will return all file descriptors passed from systemd to the service. This is useful for socket activation, and in future may be used for keeping sockets open between restarts. It is simply call :systemd.listen_fds(false) that will return list of descriptors in form of integer() | {integer(), charlist()} where 2nd entry in tuple is descriptor name.

Journal

This is set of 3 modules for Erlang’s logger utility (usable as well with Elixir 1.10+):

• systemd_stderr_formatter - this is formatter that will append special severity (log level) tag to messages passed to stdout/stderr which will allow journal to assign required severity level. Configuration is simple, it takes map with one additional field :parent that should contain the formatter module where all other options will be passed, defaults to logger_formatter.
• systemd_journal_h and systemd_journal_formatter - these two SHOULD always be used together unless you really know what you are doing, this will use systemd journal datagram socket to send log messages which will allow for:
  • structured logging with metadata extracted, so you will be able to filter messages in the journalctl command, by default these metadata fields are:
    • SYSLOG_PRIORITY - log level in syslog integer format
    • SYSLOG_TIMESTAMP - timestamp of log entry
    • SYSLOG_PID - OS PID of the VM
    • ERL_PID - PID of the Erlang process that sent message
    • CODE_LINE
    • CODE_FILE
    • CODE_MFA - module, function, and arity of the function in the Erlang format Module:Function/Arity

While controlling systemd overall would in theory possible via DBus, but IMHO this is out of scope for this library, and should be implemented as a separate one as this is not that needed in Erlang applications.

@hauleth thank you very much for such an extensive overview.

So this is to be used with a systemd service/task that is set to take advantage of the functionality you expose right? (like the NotifyAccess, Type, etc, settings)

Yes. This isn’t for interfacing with systemd, this could be written as a separate library that will implement DBus, this is for integrating of “operating” and “observability” features.

Version 0.4.0 released.

The minor version bump was due the fact that I have removed function systemd:listen_fds/1 which was taking boolean argument that defined whether remove environment variables after fetching them. Now this is replaced by calling systemd:unset_env(listen_fds). Rest is left as is, however now user can decide that library should not automatically clear environment variables on startup (however it is discouraged to do so).

Version 0.5.0 released

This introduced few breaking changes:

• Removal of systemd_journal_formatter - now everything happenes in the systemd_journal_h and do not require user to remember to use one particular formatter
• systemd_stderr_formatter was renamed to systemd_kmsg_formatter as it can be used with other output streams, not only standard_error, new name is derivation of format that the journal uses which is based on Linux kernel messages known as kmsg

Other notable changes:

• systemd:notify/2 was removed
• systemd:notify/1 can now take list of messages to be sent at once
• systemd_kmsg_formatter will be used automatically for all handlers that use logger_std_h with type standard_io or standard_error if these are attached to the journal stream (wich is checked via JOURNAL_STREAM env variable set automatically by systemd)
• systemd:ready/0 helper function that returns child spec for temporary child that will send ready notification, useful for adding it in the supervisor children list
• Watchdog can now be manually forced to run, even if the PID mismatch