Systemd can't shutdown my foreground app cleanly

Questions / Help
, ,
1

Hello, I have an application that is built using distillery.
I have a systemd unit file that starts the app in foreground mode (logs through rsyslog, direct systemd control and other advantages).
Unfortunately, no matter what I try, I can’t find a way to terminate the application in foreground in a clean way.

The main problem is that the bash script trapping SIGTERM dies when the app starts (but not the rest), so there is only beam.smp catching the signal, which causes exiting with code 1. I’m on otp 20 and elixir 1.6.5.

Anyone with ideas on how to achieve a clean shutdown?

The systemd file:

[Unit]
Description=Winex Phoenix Server
After=local-fs.target network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=winex
Group=winexweb
WorkingDirectory=/opt/winex
ExecStart=/opt/winex/bin/winex foreground
ExecStop=/opt/winex/bin/winex stop
Environment=LANG=en_US.UTF-8
Environment=MIX_ENV=prod
Environment=RELEASE_MUTABLE_DIR=/var/opt/winex
Environment=ERL_CRASH_DUMP=/var/opt/winex/erl_crash.dump
LimitNOFILE=65536
UMask=0027
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=winex
Restart=always
RestartSec=10s
TimeoutStartSec=10s
TimeoutStopSec=5s

[Install]
WantedBy=multi-user.target

When the process is started, the following things are running:

● winex.service - Winex Phoenix Server
   Loaded: loaded (/lib/systemd/system/winex.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-06-07 04:26:04 UTC; 7min ago
 Main PID: 13698 (beam.smp)
    Tasks: 31 (limit: 1152)
   CGroup: /system.slice/winex.service
           ├─13698 /opt/winex/erts-9.3.1/bin/beam.smp -Bd -- -root /opt/winex -progname opt/winex/releases/0.0.17/winex.sh -- 
           ├─13959 /opt/winex/erts-9.3.1/bin/epmd -daemon
           ├─13970 erl_child_setup 65536
           ├─13989 inet_gethost 4
           └─13990 inet_gethost 4

When is stopped, I get:

● winex.service - Winex Phoenix Server
   Loaded: loaded (/lib/systemd/system/winex.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2018-06-07 03:58:25 UTC; 1s ago
  Process: 12492 ExecStop=/opt/winex/cmds/stop.sh (code=exited, status=0/SUCCESS)
  Process: 12210 ExecStart=/opt/winex/bin/winex foreground (code=exited, status=1/FAILURE)
 Main PID: 12210 (code=exited, status=1/FAILURE)

Jun 07 03:57:51 winehost1 systemd[1]: Started Winex Phoenix Server.
Jun 07 03:57:53 winehost1 winex[12210]: 03:57:53.540 [info] Running Winex.Web.Endpoint with Cowboy using http://127.0.0.1:4000
Jun 07 03:58:25 winehost1 systemd[1]: Stopping Winex Phoenix Server...
Jun 07 03:58:25 winehost1 winex[12210]: 03:58:25.455 [info] SIGTERM received - shutting down
Jun 07 03:58:25 winehost1 winex[12210]: erl_child_setup closed
Jun 07 03:58:25 winehost1 winex[12210]: [1B blob data]
Jun 07 03:58:25 winehost1 winex[12210]: Crash dump is being written to: /var/opt/winex/erl_crash.dump...done
Jun 07 03:58:25 winehost1 systemd[1]: winex.service: Main process exited, code=exited, status=1/FAILURE
Jun 07 03:58:25 winehost1 systemd[1]: winex.service: Failed with result 'exit-code'.
Jun 07 03:58:25 winehost1 systemd[1]: Stopped Winex Phoenix Server.

Any suggestions?
Right now my temporary solution is to avoid ExecStop and instead use KillMode=mixed which kills the main process with SIGTERM and the rest with SIGKILL.

2

You really shouldn’t be using a foreground runner, rather do it like how I detail at:

As for logging to the system syslog, you should add in a syslog logger as it will give you substantially better control and won’t just be plain string munging output:

:slight_smile:

2 Likes
3

Thanks for the input, seems interesting!
I would still like to find a solution when using foreground, especially for future reference.

For small apps, not needing hex packages and just write to STDOUT and STDERR (which gives you a lot of control), without also having to deal with the pidfile greatly simplifies the whole operation part.

As I said, the entire thing is working correctly, it just needs a clean shutdown.

4

systemd managing foregrounded apps expect them to behave in a certain way, it’s mostly a thing to manage broken apps and should not be used for anything that follows proper Unix style daemon’s, which Beam backgrounded mode does.

But this is the issue, systemd expects foregrounded apps to die when their pipe closes or during certain signals, the Beam is not built for that, it is built to be a normal standard unix-style daemon. You are fighting it as you see that you currently are if you want to try it this way.

However, you could write a wrapper script that handles communication between systemd and the beam and handles the signals, just when you get a shutdown indicator then just be sure to safely and properly shut down the beam before returning, and make sure it doesn’t get brutal killed before it’s done.

It is better to follow standard daemon procedures when dealing with systemd, regardless of the type of server. ^.^;

1 Like
5

Mh, ok you convinced me, makes sense. I didn’t consider that actually beam needs to run “forever” (as a daemon), thanks! :wink:

1 Like