Socket activation with systemd? (to reduce downtime on restart)

It’s possible to pass fd option to :gen_tcp.listen: gen_tcp — OTP 29.0.2 (kernel 11.0.2). So something like

# not sure about this ...
config :app, App.Endpoint,
  http: [transport_options: %{socket_opts: [fd: SYSTEMD_FIRST_SOCKET_FD]}]

might work.

So working from bottom up,

• :gen_tcp’s fd option would probably make it work
• ranch accepts socket options via :socket_opts
• plug uses :transport_options to configure that part of :ranch
• phoenix uses :http option to configure that part of plug

This looks like a solid start - I will experiment, many thanks!

Very interested in hearing if you get it working. I tried once briefly and failed to get it to work.

I tried with {fd, 3} as an option to gen_tcp:listen.

I’ll have to dig it back up to try again and see where it went wrong, I remember it was a hassle to get the systemd service and socket working properly to even test the Erlang code and that is likely the reason it didn’t work, and not an issue with Erlang.

I know @garazdawi said he was interested as well.

I will experiment the coming week, on a client setup for which I need to do maintenance (Ansible + Elixir). Will report back with my findings!

I can’t help with the socket activation stuff, however I have been using a docker solution for zero downtime deploys, and it has been working great for me.

In a nutshell, it uses docker swarm which will start a new instance of the app, wait for it to become ready, move traffic over, and finally remove the old one. It’s surprisingly simple and works great on single server setups.

I can write up more details if people are interested.

While I won’t be able to use that technique on the projects I’m managing at the moment, I’d say it’s interesting to share anyway (if that’s not too much work on your side!).

An update on this: so far I’m not able to start the app, but here are bits of feedback in case someone else also tackles this (I will get back to this later).

I’ve created a file under /etc/systemd/system/my_app.socket, with content:

[Unit]
Description=My App Socket
PartOf=my_app.service

[Socket]
# this is the Phoenix port
ListenStream=4000
BindIPv6Only=both

[Install]
WantedBy=sockets.target

The documentation for ListenStream etc is at Ubuntu Manpage: systemd.socket - Socket unit configuration.

After a restart with reload systemctl_daemon, I can see an active unit for the socket with sudo systemctl status my_app.socket.

I’ve also modified my app service file under /etc/systemd/system/my_app.service, with (extract, cannot share in full):

[Unit]
# added my_app.socket
After=syslog.target network.target postgresql.service my_app.socket
Requires=my_app.socket

[Service]
Type=simple
User={{ item.user }}
# SNIP

I’ve (trial and error) updated the prod.exs configuration used to build my releases (via Distillery) with:

http: [:inet6, port: port, transport_options: [socket_opts: [fd: 3]]],

Ultimately though, my logs show the following error:

[error] Failed to start Ranch listener MyAppWeb.Endpoint.HTTP in
:ranch_tcp:listen([{:cacerts, :...}, {:key, :...}, {:cert, :...}, {:fd, 3}, :inet6,
{:port, 4000}]) for reason :einval (invalid argument)

This is already progress, because I had a lot of other errors before earlier, but at this point I must dive deeper into Ranch to figure out what is happening.

I’d suggest starting with just a gen_tcp module and not involve ranch or anything else.

Getting the same error {:error, :einval} with :gen_tcp.listen(5000, fd: 3) and the following systemd units:

[Unit]
Description=My App Socket
PartOf=my_app.service

[Socket]
ListenStream=5000
BindIPv6Only=both

[Install]
WantedBy=sockets.target

[Unit]
Description=My app daemon
After=syslog.target network.target my_app.socket
Requires=my_app.socket

[Service]
Type=simple
User=ubuntu
Group=ubuntu
Restart=on-failure
WorkingDirectory=/opt/socket_activation_demo
ExecStart=/opt/socket_activation_demo/bin/socket_activation_demo foreground

[Install]
WantedBy=multi-user.target

Just in case, here’s the env vars set for the service

iex(socket_activation_demo@127.0.0.1)15> System.get_env |> Enum.each(&IO.inspect/1)
{"HOME", "/home/ubuntu"}
{"TERM", "xterm"}
{"ERTS_VSN", "10.3.1"}
{"ERTS_DIR", "/opt/socket_activation_demo/erts-10.3.1"}
{"POST_CONFIGURE_HOOKS", "/opt/socket_activation_demo/releases/0.1.0/hooks/post_configure.d"}
{"SRC_SYS_CONFIG_PATH", "/opt/socket_activation_demo/releases/0.1.0/sys.config"}
{"LANG", "C.UTF-8"}
{"SHLVL", "0"}
{"NAME_ARG", "-name socket_activation_demo@127.0.0.1"}
{"DISTILLERY_VSN", "2.1.1"}
{"PRE_UPGRADE_HOOKS", "/opt/socket_activation_demo/releases/0.1.0/hooks/pre_upgrade.d"}
{"RELEASES_DIR", "/opt/socket_activation_demo/releases"}
{"PATH", "/opt/socket_activation_demo/erts-10.3.1/bin:/opt/socket_activation_demo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}
{"CONSOLIDATED_DIR", "/opt/socket_activation_demo/lib/socket_activation_demo-0.1.0/consolidated"}
{"REL_DIR", "/opt/socket_activation_demo/releases/0.1.0"}
{"RELEASE_CONFIG_DIR", "/opt/socket_activation_demo"}
{"POST_UPGRADE_HOOKS", "/opt/socket_activation_demo/releases/0.1.0/hooks/post_upgrade.d"}
{"REL_NAME", "socket_activation_demo"}
{"NAME", "socket_activation_demo@127.0.0.1"}
{"ROOTDIR", "/opt/socket_activation_demo"}
{"DEST_VMARGS_PATH", "/opt/socket_activation_demo/var/vm.args"}
{"RELEASE_ROOT_DIR", "/opt/socket_activation_demo"}
{"OLDPWD", "/opt"}
{"RELEASE_MUTABLE_DIR", "/opt/socket_activation_demo/var"}
{"PRE_CONFIGURE_HOOKS", "/opt/socket_activation_demo/releases/0.1.0/hooks/pre_configure.d"}
{"ERL_OPTS", ""}
{"NAME_TYPE", "-name"}
{"DEST_SYS_CONFIG_PATH", "/opt/socket_activation_demo/var/sys.config"}
{"LISTEN_FDNAMES", "my_app.socket"}
{"SRC_VMARGS_PATH", "/opt/socket_activation_demo/releases/0.1.0/vm.args"}
{"ESCRIPT_NAME", "/opt/socket_activation_demo/releases/0.1.0/socket_activation_demo.sh"}
{"SHELL", "/bin/bash"}
{"LD_LIBRARY_PATH", "/opt/socket_activation_demo/erts-10.3.1/lib:"}
{"PRE_START_HOOKS", "/opt/socket_activation_demo/releases/0.1.0/hooks/pre_start.d"}
{"BINDIR", "/opt/socket_activation_demo/erts-10.3.1/bin"}
{"DISTILLERY_TASK", "foreground"}
{"LISTEN_PID", "5248"}
{"POST_STOP_HOOKS", "/opt/socket_activation_demo/releases/0.1.0/hooks/post_stop.d"}
{"PWD", "/opt/socket_activation_demo"}
{"ERTS_LIB_DIR", "/opt/socket_activation_demo/lib"}
{"SYS_CONFIG_PATH", "/opt/socket_activation_demo/var/sys.config"}
{"PROGNAME", "opt/socket_activation_demo/releases/0.1.0/socket_activation_demo.sh"}
{"LISTEN_FDS", "1"}
{"VMARGS_PATH", "/opt/socket_activation_demo/var/vm.args"}
{"PRE_STOP_HOOKS", "/opt/socket_activation_demo/releases/0.1.0/hooks/pre_stop.d"}
{"REL_VSN", "0.1.0"}
{"EMU", "beam"}
{"POST_START_HOOKS", "/opt/socket_activation_demo/releases/0.1.0/hooks/post_start.d"}
{"LOGNAME", "ubuntu"}
{"USER", "ubuntu"}
{"START_ERL_DATA", "/opt/socket_activation_demo/var/start_erl.data"}
{"JOURNAL_STREAM", "9:98738"}
{"RUN_ERL_ENV", ""}
{"INVOCATION_ID", "3f625ea198164b3094464e146ff67c0b"}
:ok

the relevant ones are probably:

{"LISTEN_FDS", "1"}
{"LISTEN_FDNAMES", "my_app.socket"}

Can connect to it so it works …

iex(socket_activation_demo@127.0.0.1)12> :gen_tcp.connect({0,0,0,0}, 5000, [:binary])
{:ok, #Port<0.55>}

But cannot listen

Yea, I think that is what I got when I tried as well and never had the time to debug further.