Elixir apps as systemd services - info & wiki

wiki
deployment
linux

#1

Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for any Debian derivative and shouldn’t need much tweaking for other distros.

Example systemd service

[Unit]
Description=My app daemon

[Service]
Type=simple
User=username
Group=groupname
Restart=on-failure
Environment=MIX_ENV=prod "PORT=4000"
Environment=LANG=en_US.UTF-8

WorkingDirectory=/var/apps/myapp
ExecStart=/usr/local/bin/mix phoenix.server

[Install]
WantedBy=multi-user.target

must be put it in /lib/systemd/system/myapp.service, also note the use of absolute paths and extra verbosity for utf8 support. Control it using systemctl, e.g. sudo systemctl status myapp.service, sudo systemctl restart myapp.service etc.

After setting up the service must be enabled by running systemctl enable myapp.service, this must only be done once so that the system creates symlinks to the service file. Without this step everything will work fine but will not be restarted on boot.

For much more feature rich description s. https://www.freedesktop.org/software/systemd/man/systemd.service.html

View logs

Default Phoenix behavior (logging to standard output) plays well with systemd, so you can use journalctl to manage logs, few examples:

journalctl -u myapp.service --since today
journalctl -u myapp.service --since 09:00 --until "1 hour ago"
journalctl -u myapp.service --since "2016-11-10 12:00" --until "2016-11-10 13:00"

s. https://www.freedesktop.org/software/systemd/man/journalctl.html for more.

Restart on deploy

To allow for automatic restart of the service, e.g. as part of automatic deployment adjust the sudoers file by running sudo visudo and add something like that username ALL = NOPASSWD: /bin/systemctl restart myapp.service so that the user with the name username can restart the service without entering root password

“Old school” logs

Write data to a file (or use off the shelf solution like https://github.com/onkel-dirtus/logger_file_backend) and manage logs using Linux lorgotate, here is an example config

/var/apps/myapp/log/*.log {
daily
missingok
rotate 18
compress
delaycompress
notifempty
create 660 username groupname    
dateext
dateformat -%Y-%m-%d-%s    
su username groupname
}

must be put in /etc/logrotate.d/myapp, dry run / debug sudo logrotate -d /etc/logrotate.d/myapp, force rotation (for testing) sudo logrotate --force /etc/logrotate.d/myapp. More on logrotate http://www.linuxcommand.org/man_pages/logrotate8.html

Example upstart service

Relevant if using Ubuntu before 16.04 (not sure how it goes between LTS’), this example uses exrm releases:

description "My app daemon"

setuid username
setgid groupname

start on runlevel [2345]
stop on runlevel [016]

expect stop
respawn

env MIX_ENV=prod
export MIX_ENV

env PORT=4000
export PORT

env HOME=/var/apps/myapp
export HOME


pre-start exec /bin/sh /var/apps/myapp/bin/myapp start

post-stop exec /bin/sh /var/apps/myapp/bin/myapp stop

Example deployment script

To run automatically after deployment for rolling update of a phoenix project with no DB (hence no ecto migrate) in staging environment (hence tests):

cd /var/apps/myapp
MIX_ENV=prod mix deps.get
brunch build --production
MIX_ENV=prod mix phoenix.digest
MIX_ENV=prod mix compile
sudo service myapp restart    
mix test

can be triggered by a commit into a specific branch or using web UI, if anything goes wrong the whole thing exits with a non zero status and lets you know about the problem. Makes a nice simple alternative to CI.


Phoenix server crashing randomly
Rotating logs in OTP releases
Creating a systemd Service for Monitoring
Running phoenix server in background
#2

Type=simple might not be a correct type https://www.lucas-nussbaum.net/blog/?p=877


#3

Do you get any issues cause of the type? I don’t think I do, a startup error (as far as I remember) triggers visible error, starting and restarting the service works fine except erlang sees it as crush cause of the child killing business :slight_smile:


#4

I had an issue with node restart on an application crash and overcame it with RestartSec=5 but Type=forking is a cleaner solution.


#5

Do you have a well tested service with forking type? If not I’ll give it a try next time I have to set up an environment.


#6

We switched to it couple weeks ago and so far the flight is normal :wink:


#7

care to post the config?


#8

I’m using releases, systemd and a service type of ‘forking’. I haven’t tested restarting on crash yet. (What would be a good way of doing that?)

My unit file:

[Unit]
Description=My App
After=network.target
Requires=network.target

[Service]
Type=forking
WorkingDirectory=/home/appuser/app
User=appuser
Group=appuser
Restart=on-failure
RestartSec=5
EnvironmentFile=/home/appuser/env_vars
ExecStart=/bin/bash -c 'PATH=/home/appuser/.asdf/shims:$PATH exec /home/appuser/app/bin/my_app start'
ExecStop=/bin/bash -c 'PATH=/home/appuser/.asdf/shims:$PATH exec /home/appuser/app/bin/my_app stop'

[Install]
WantedBy=multi-user.target

(I’ve installed erlang on the server via asdf so that my releases don’t have to include_erts, and found that I needed to add its shims to PATH.)

Is this “child killing business” why systemd reports a failed state on stop? Is there any way round that?


Relevant commit and discussion at Distillery:


#9

you can try killing the process so that it gets restarted

I think it is. There might be a cleaner way to stop the vm, from your config it seems both changing the type to forking and the nice ExecStop do not work. I didn’t get to that yet, please let me know if you find the settings that kill the vm softly :slight_smile:

The comment from bitwalker in the linked issue seems to make sense for both releases and mix services: “I suspect that maybe it’s difficult for systemd to understand which pid is the actual pid it should care about - guessing the main pid when starting is probably pretty easy, but stopping is maybe getting tripped up because even though the daemon is stopped, the shell process is still executing the post_stop hooks afterward.”