Also, if it helps, I use this overlay to generate (via distillary) a systemd service file using this PID library:

[Unit]
Description=<%= description %>
After=network.target

[Service]<% full_dir = Path.absname(output_dir)%>
Type=forking
User=<%= deploy_user %>
Group=<%= deploy_group %>
WorkingDirectory=<%= full_dir %>
ExecStart=<%= full_dir %>/bin/<%= p_name %> start
ExecReload=<%= full_dir %>/bin/<%= p_name %> reload_config
ExecStop=<%= full_dir %>/bin/<%= p_name %> stop
PIDFile=<%= full_dir %>/<%= p_name %>.prod.pid
Restart=always
RestartSec=5
Environment=PORT=3000
Environment=LANG=en_US.UTF-8
SyslogIdentifier=<%= p_name %>

[Install]
WantedBy=multi-user.target

Been using it in production ever since migrated the server from Windows 2008 to redhat, so a couple weeks now without issue. This follows the proper systemd model using a PID file with proper restarts (tested it by manually telling the server to stop, killing it, kill -9’ing it, comes back up in 5 seconds every time).

Thanks for this! I’ve been trying to get systemd to restart my app if it crashes. I used your :pid_file package and it seems to be managing the pidfile correctly. I added the path to the pidfile to my systemd unit file. I can start and stop the app via systemd and the app starts automatically when the server starts but if I kill the app it doesn’t restart. Wondering if you have any suggestions on this. I start my app using systemd, then check the pidfile to get the pid (in this case it was 1645). then run kill 1645 the app stops and is not restarted. I checked the systemd logs using journalctl -u my_app.service and got the following output. Any thoughts? Thanks again!

Aug 22 11:01:25 ip-172-31-4-169 my_app[1682]: Starting up
Aug 22 11:01:26 ip-172-31-4-169 systemd[1]: my_app.service: Supervising process 1645 which is not our child. We'll most likely not notice when it exits.
Aug 22 11:01:26 ip-172-31-4-169 systemd[1]: Started my_app.
Aug 22 11:02:03 ip-172-31-4-169 run_erl[1644]: Erlang closed the connection.

I noticed in the last line of the log messages run_erl[1644], note 1644 is the app’s pid - 1. I also noticed the message “Supervising process 1645 which is not our child. We’ll most likely not notice when it exits”. I think this may be a clue. Is run_erl a process that “wraps” the app process? Maybe systemd interacts with run_erl rather than our app directly and therefore doesn’t notice if the app exits.

Hmm I just kill -9 beam.smp here and it started back up after 5 seconds. Though I use the systemd service config file I displayed above.

Can you post the systemd service config file that you are using? And I bet the run_erl is pre-forking or something, as you can see I’m calling the release scripts directly.

Thanks for this. Yea I’m not too sure what the issue is. My server is ubuntu so I couldn’t use the exact command you used to kill it but I tried pkill -9 beam.smp and got the same results. Systemd reported that “Erlang closed the connection” and didn’t appear to attempt a restart. This is my conf file. Maybe I messed something up.

[Unit]
Description=My App
After=network.target

[Service]
Type=forking
User=my_app_deployer
Group=my_app_deployer
WorkingDirectory=/home/my_app_deployer/my_app/staging/my_app
ExecStart=/home/my_app_deployer/my_app/staging/my_app/bin/my_app start
ExecStop=/home/my_app_deployer/my_app/staging/my_app/bin/my_app stop
PIDFile=/home/my_app_deployer/my_app/staging/my_app/my_app.pid
Restart=always
RestartSec=5
EnvironmentFile=/home/my_app_deployer/.my_app_env
SyslogIdentifier=my_app
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Thanks again!

Well first of all that should not be there. That means ‘pretend everything stays working even when the application closes’. Why do you have that there?! o.O

Nice. Thanks for your help @OvermindDL1. That was my problem. Removed that line and it’s working

I had it in there because i saw in the Distillery docs (https://hexdocs.pm/distillery/use-with-systemd.html#content) that “It’s important that you have RemainAfterExit=yes set, or you will get an error trying to start the service.” I didn’t give any thought to the effect that line would have. And contrary to the note in the docs I haven’t had trouble starting the service since removing that line.

Thanks again!

Distillery does it wrong, I don’t know why they don’t do it right… ^.^;
Using Distillery’s instructions your service will not get restarted, which seems to defeat the point to me? ^.^

You are welcome! Nice to see others are using this too.

@OvermindDL1: Just looked at your code - it’s simple and useful.
I don’t understand why your are calling:

:os.getpid() # charlist
|> to_string()
|> String.to_integer()
|> to_string()

i.e. update_pid/1 calls get_pid/0 that returns integer (from string) and you are changing it back to string. I could understand if it’s used in another function, but it’s not, so I’m little curious why you did it.

:os.getpid() # charlist
|> to_string() # Convert charlist to string
|> String.to_integer() # convert string to integer to make sure it really is an integer or it throws, I needed this in some cases but don't recall where
|> to_string() # Convert it back to a string for later writing out

So yeah, I needed to ensure it was an integer at one point but don’t recall where, this code is old… >.>

Probably it is worth mentioning in this thread that it is common to supervise the VM via the built-in heart mechanism: heart — OTP 29.0.2 (kernel 11.0.2)

It works by starting a separate process called heart that not only checks if the beam process is still alive but also sends heartbeat messages to it (hence the name).

Of course if you use heart other restart mechanisms should not be in place. Also if you want to manually kill the VM you need to kill the heart process first, then the VM. Better option to stop the VM would be to call init:stop() from a remote node as described here for example: [erlang-questions] How to stop a detached OTP app?

The main advantage of the heart method is that it is OS independent.