Systemd - Erlang library for communicating with (surprise) systemd

Chat / Discussions
, ,
1

I have written (and published) small Erlang library that helps with integrating your release with systemd

Features

The idea of this library is to port some parts of the libsystemd into Erlang without need for NIFs. Currently implemented features:

  • Notify socket which allows for sending notifications to systemd. This for example allows blocking startup until our application is completely ready, so not only it started up, but is ready to do it’s work. Unfortunately the feature of saving FDs between runs (which would allow for restarting application without disconnecting users) is not currently possible, but will be probably added as soon as Erlang stabilise socket interface instead of inet.
  • Watchdog which allows systemd to watch for keep-alive messages via notify socket. This allows you to provide simple information to the systemd when it should restart your application before your failures hit the init process.
  • Listen FDs which allow for socket activation of your application (i.e. application is started only when there is incoming data in socket) and keeping sockets opened between restarts. However while it is implemented it is currently pointless feature as you cannot use these fds in any way due to fact that gen_tcp and gen_udp requires unbind(3)ed file descriptors. Maybe this will became more usable once socket interface land.
  • Check for systemd booted system - pretty pointless, as all functionalities above are safe which mean that you can use all of above even on non-systemd OS and it will not (or at least should not) cause any problems. This was implemented just for functionality parity.

The best thing? It is mostly automagical. Watchdog and notify socket will be established without any user interaction. Only thing needed is to tell systemd that your application is ready (not possible to be done automatically unfortunately).

TODO in future:

  • journald backend for logger (will be useful for Elixir 1.10+ as well, as this shares the logging implementation with Erlang)

Example

Simple Plug example

Just fetch repo on machine with systemd, and run make (requires sudo to setup systemd services). Then you can see that service is up and running:

hauleth@test-ubuntu:~/plug$ systemctl status example-watchdog.service
● example-watchdog.service - Example BEAM application that uses watchdog
   Loaded: loaded (/usr/local/lib/systemd/system/example-watchdog.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-16 14:46:02 UTC; 3s ago
 Main PID: 1662 (beam.smp)
    Tasks: 20 (limit: 1108)
   CGroup: /system.slice/example-watchdog.service
           ├─1662 /home/hauleth/plug/_build/prod/rel/plug_systemd_example/erts-10.6.2/bin/beam.smp -- -root /home/hauleth/plug/_build/prod/rel/plug_systemd_example -progname erl
           ├─1701 /home/hauleth/plug/_build/prod/rel/plug_systemd_example/erts-10.6.2/bin/epmd -daemon
           └─1704 erl_child_setup 1024

Jan 16 14:46:01 test-ubuntu systemd[1]: Starting Example BEAM application that uses watchdog...
Jan 16 14:46:02 test-ubuntu systemd[1]: Started Example BEAM application that uses watchdog.

We simulate reload by curl http://localhost:4000/reload:

hauleth@test-ubuntu:~/plug$ curl http://localhost:4000/reload
hauleth@test-ubuntu:~/plug$ systemctl status example-watchdog.service
● example-watchdog.service - Example BEAM application that uses watchdog
   Loaded: loaded (/usr/local/lib/systemd/system/example-watchdog.service; enabled; vendor preset: enabled)
   Active: reloading (reload) since Thu 2020-01-16 14:46:02 UTC; 2min 10s ago
 Main PID: 1662 (beam.smp)
    Tasks: 20 (limit: 1108)
   CGroup: /system.slice/example-watchdog.service
           ├─1662 /home/hauleth/plug/_build/prod/rel/plug_systemd_example/erts-10.6.2/bin/beam.smp -- -root /home/hauleth/plug/_build/prod/rel/plug_systemd_example -progname erl
           ├─1701 /home/hauleth/plug/_build/prod/rel/plug_systemd_example/erts-10.6.2/bin/epmd -daemon
           └─1704 erl_child_setup 1024

Jan 16 14:46:01 test-ubuntu systemd[1]: Starting Example BEAM application that uses watchdog...
Jan 16 14:46:02 test-ubuntu systemd[1]: Started Example BEAM application that uses watchdog.

And after 10 seconds it is “up” again:

hauleth@test-ubuntu:~/plug$ systemctl status example-watchdog.service
● example-watchdog.service - Example BEAM application that uses watchdog
   Loaded: loaded (/usr/local/lib/systemd/system/example-watchdog.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-16 14:46:02 UTC; 2min 27s ago
 Main PID: 1662 (beam.smp)
    Tasks: 20 (limit: 1108)
   CGroup: /system.slice/example-watchdog.service
           ├─1662 /home/hauleth/plug/_build/prod/rel/plug_systemd_example/erts-10.6.2/bin/beam.smp -- -root /home/hauleth/plug/_build/prod/rel/plug_systemd_example -progname erl
           ├─1701 /home/hauleth/plug/_build/prod/rel/plug_systemd_example/erts-10.6.2/bin/epmd -daemon
           └─1704 erl_child_setup 1024

Jan 16 14:46:01 test-ubuntu systemd[1]: Starting Example BEAM application that uses watchdog...
Jan 16 14:46:02 test-ubuntu systemd[1]: Started Example BEAM application that uses watchdog.

We can also force it to restart by curl http://localhost:4000/stop which will trigger Watchdog to restart application:

hauleth@test-ubuntu:~/plug$ curl http://localhost:4000/
1760
hauleth@test-ubuntu:~/plug$ curl http://localhost:4000/stop
hauleth@test-ubuntu:~/plug$ curl http://localhost:4000/
1846

Requests on / return current PID of the VM, so we clearly see that the restart happened in the meantime.

If you are using systemd for supervising your application, then I highly suggest you to try it and see if this can improve your experience.

26 Likes
2

Version 0.3.0 released:

This update adds support for systemd’s journal, which makes library feature complete for now. Now it needs a little polishing and real-world usage before I mark it as a 1.0.0.

Again - feel free to test this out and to submit issues if any. If someone needs any additional feature, then also feel free to submit it, however for me it is (as said above) feature complete.

4 Likes
3

I have also prepared simple Elixir application that uses this library:

Unfortunately, as mentioned in original post, it cannot use socket activation (yet), but rest of features is implemented there.

8 Likes
4

Ooo, this looks useful!! Thanks for it!!

5

There are still some bugs there and here (mostly in biggest part of the repo which is systemd_journal_formatter) with fixes on master (I will publish 0.3.3 soon).

Additionally I have finally got socket activation working. Unfortunately due to ERL-1138 the passed FDs can be used only in UNIX sockets (as these do not bind(3) and this is the culprit there), so I just have done some “socket jumping” and translate TCP port to UNIX Stream socket and then to the Cowboy.

So in other words there are 5 systemd units:

  • example-application.service - which is THE application service file, it will just simply run the application when started
  • example-application.socket - UNIX Stream socket definition that will be handled by systemd, this is required due to the above bug; this will start example-application.service on any connection incoming
  • example-activation.service - service that will take the systemd activated FD and forward all it all to another socket provided as an argument, think. of it as an alternative implementation of socat
  • example-activation.socket - definition of TCP socket that systemd will listen on and will pass it down when there will be any request incoming
  • example.target - simple target for ease of use

Within the application the magic happens there:

It creates new gen_tcp socket from the provided file descriptor and then pass it to the Ranch (via Cowboy). The only problem is that currently each first request (when service is down) will fail with no response sent, but all consecutive requests will work as expected.

2 Likes
6

Hi,

I use systemd and saw this but could you expand on the normal use case where you would use this and how? My interpretation is that instead of running the usual systemctl commands at the OS level you run them from inside the VM itself?

1 Like
7

It depends, because it provides few features. I tried to describe them the best I could in the docs (if you find something unclear or missing, then open the issue, I will try my best to fix it).

The features are as follows:

Notify socket

This is used to inform systemd about state of your application. So if you have NotifyAccess=main or NotifyAccess=all then you can for example notify systemd when your application is truly ready (especially when you will also use Type=notify). Erlang applications rarely are ready to work just when VM start and there is need for some setup which happens in the supervisor (DB connection, HTTP server setup, etc.) and this can take some time as well. So with this library you can just call :systemd.notify(:ready) when everything is done and systemctl will block until it receives that message. So when systemctl start my_app.service ends you are sure that your app is actually ready.

Additionally to that also will receive information when the application will be shutting down, but not yet completely dead. So for example if you use Plug.Cowboy.Drainer in your application, which will hold the application from shutting down until all HTTP requests will not be served, then you will have information about application shutting down in systemctl status my_app.service, like this:

https://asciinema.org/a/jqTbUdgFkc7206vFK4AScqq5p

You can also send other notifications but I am not truly sure how these are handled.

In future, when ERL-1139 and ERL-1138 will land, it will also be possible to “save” sockets between restarts, so your application will be able to restart without closing existing connections.

Watchdog

Watchdog is heart-beat systemd subsystem that expects to receive special messages within specified timespan to make sure that application is alive. This is controlled by WatchdogSec= option in [Service] section of systemd unit. This option will be handled automatically for you (so the pinging process is automatically handled), but you still have an option to call :systemd.watchdog(:trigger) which will send special triggering message which will work like there would be no keep-alive message sent within requested timespan. This will trigger action that depends on Restart= option. More can be found in systemd.service(5) and sd_watchdog_enabled(3).

Listen FDs

This is just the function that will return all file descriptors passed from systemd to the service. This is useful for socket activation, and in future may be used for keeping sockets open between restarts. It is simply call :systemd.listen_fds(false) that will return list of descriptors in form of integer() | {integer(), charlist()} where 2nd entry in tuple is descriptor name.

Journal

This is set of 3 modules for Erlang’s logger utility (usable as well with Elixir 1.10+):

  • systemd_stderr_formatter - this is formatter that will append special severity (log level) tag to messages passed to stdout/stderr which will allow journal to assign required severity level. Configuration is simple, it takes map with one additional field :parent that should contain the formatter module where all other options will be passed, defaults to logger_formatter.
  • systemd_journal_h and systemd_journal_formatter - these two SHOULD always be used together unless you really know what you are doing, this will use systemd journal datagram socket to send log messages which will allow for:
    • structured logging with metadata extracted, so you will be able to filter messages in the journalctl command, by default these metadata fields are:
      • SYSLOG_PRIORITY - log level in syslog integer format
      • SYSLOG_TIMESTAMP - timestamp of log entry
      • SYSLOG_PID - OS PID of the VM
      • ERL_PID - PID of the Erlang process that sent message
      • CODE_LINE
      • CODE_FILE
      • CODE_MFA - module, function, and arity of the function in the Erlang format Module:Function/Arity

While controlling systemd overall would in theory possible via DBus, but IMHO this is out of scope for this library, and should be implemented as a separate one as this is not that needed in Erlang applications.

5 Likes
8

@hauleth thank you very much for such an extensive overview.

So this is to be used with a systemd service/task that is set to take advantage of the functionality you expose right? (like the NotifyAccess, Type, etc, settings)

9

Yes. This isn’t for interfacing with systemd, this could be written as a separate library that will implement DBus, this is for integrating of “operating” and “observability” features.

10

Version 0.4.0 released.

The minor version bump was due the fact that I have removed function systemd:listen_fds/1 which was taking boolean argument that defined whether remove environment variables after fetching them. Now this is replaced by calling systemd:unset_env(listen_fds). Rest is left as is, however now user can decide that library should not automatically clear environment variables on startup (however it is discouraged to do so).

1 Like
11

Version 0.5.0 released

This introduced few breaking changes:

  • Removal of systemd_journal_formatter - now everything happenes in the systemd_journal_h and do not require user to remember to use one particular formatter
  • systemd_stderr_formatter was renamed to systemd_kmsg_formatter as it can be used with other output streams, not only standard_error, new name is derivation of format that the journal uses which is based on Linux kernel messages known as kmsg

Other notable changes:

  • systemd:notify/2 was removed
  • systemd:notify/1 can now take list of messages to be sent at once
  • systemd_kmsg_formatter will be used automatically for all handlers that use logger_std_h with type standard_io or standard_error if these are attached to the journal stream (wich is checked via JOURNAL_STREAM env variable set automatically by systemd)
  • systemd:ready/0 helper function that returns child spec for temporary child that will send ready notification, useful for adding it in the supervisor children list
  • Watchdog can now be manually forced to run, even if the PID mismatch
2 Likes
12

On Slack I have received feedback, that Elixir documentation is lacking, so right now I am trying to improve that a little.

So if someone would be so keen to check out if this is clear enough as I am biased

2 Likes
13

Updated example Elixir project to use newest version of systemd library

Now it do not longer need to hop via the additional UNIX socket, it uses the passed socket directly. Additionally this resolved the problem with dropping the first incoming connection and now works fast and reliably each time. It also manages EPMD via systemd (requires compiling Erlang with --enable-systemd flag).

6 Likes
14

Examples are now merged into main projects, so it should make them more up-to-date with the project implementation.

There is also new example that implements RFC 862 Echo Server in Erlang for both TCP and UDP listening on port 7. The reason for that was to show how to run service that is listening on 2 different sockets passed from the systemd.

5 Likes
15

This is a really cool library I have to say. ^.^

2 Likes
16

Thanks. I try to make it as seamless to integrate as possible. Additionally I want make documentation clear to increase awareness that you can achieve 90% of what Docker is doing by using tools that are already on your machine.

9 Likes
17

I’m very pro systemd, I know a lot of people aren’t because “monolithic”, but that gains a lot of abilities, like the thing of being able to pre-allocate but not initialize sockets to pass into programs for faster loading as one of many many things. There are a significant amount of capabilities with systemd, and combined with cgroups can give you a lot of safety you just can’t get any other way as easily.

4 Likes
18

systemd 0.6.2 released:

The only important change is new documentation that uses ExDoc - systemd — systemd v0.6.2

1 Like
19

First article of a series about integrating Elixir applications with systemd

10 Likes
20

Thanks for your work.

Looking forward to the next two articles. :wink:

2 Likes