yurko

Elixir apps as systemd services - info & wiki

Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for any Debian derivative and shouldn’t need much tweaking for other distros.

Example systemd service

[Unit]
Description=My app daemon

[Service]
Type=simple
User=username
Group=groupname
Restart=on-failure
Environment=MIX_ENV=prod "PORT=4000"
Environment=LANG=en_US.UTF-8

WorkingDirectory=/var/apps/myapp
ExecStart=/usr/local/bin/mix phoenix.server

[Install]
WantedBy=multi-user.target

must be put it in /lib/systemd/system/myapp.service, also note the use of absolute paths and extra verbosity for utf8 support. Control it using systemctl, e.g. sudo systemctl status myapp.service, sudo systemctl restart myapp.service etc.

After setting up the service must be enabled by running systemctl enable myapp.service, this must only be done once so that the system creates symlinks to the service file. Without this step everything will work fine but will not be restarted on boot.

For much more feature rich description s. https://www.freedesktop.org/software/systemd/man/systemd.service.html

1.9 and up: using releases

Since 1.9 there is no reason not to use releases, the transition from mentioned service is also pretty easy. In my experience simple service type works fine (and since it’s a recommended way unless something special needed I use it).

An extra step required when deploying - to build a release. The simplest possible case would be to do it on the same server, something like MIX_ENV=prod PORT=80 mix release release-name --overwrite. S. mix release — Mix v1.20.2 for info about release command and release configs.

The resulting systemd service is not much different:

[Unit]
Description=My app daemon
 
[Service]
Type=simple
User=username
Group=groupname
Restart=on-failure
Environment=MIX_ENV=prod
Environment=LANG=en_US.UTF-8
Environment=PORT=80
 
WorkingDirectory=/var/apps/myapp

ExecStart=/var/apps/myapp/_build/prod/rel/release-name/bin/live start
ExecStop=/var/apps/myapp/_build/prod/rel/release-name/bin/live stop

[Install]
WantedBy=multi-user.target

Side note: To be able to use port 80 as in the above example without running as root CAP_NET_BIND_SERVICE Linux capability can be used on packaged runtime, here is a command that achieves it (version number may change so after upgrading Erlang this command must be run again with new path): sudo setcap 'cap_net_bind_service=+ep' /var/apps/myapp/_build/prod/rel/release-name/erts-10.4.3/bin/beam.smp more info about Linux capabilities https://linux.die.net/man/7/capabilities

UPD: Instead of setting capabilities on executable level it is also possible to do that on service level, this way it would not care about runtime version. It is done with following line in the service file: CapabilityBoundingSet=CAP_NET_BIND_SERVICE. There is a quirk though (as it often the case with systemd): this would not work unless ambient capabilities are set as well, so the whole following part is needed:

# Add capability to be able to bind on port 80, 
# doing it here means we don't care about runtime version and location
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

View logs

Default Phoenix behavior (logging to standard output) plays well with systemd, so you can use journalctl to manage logs, few examples:

journalctl -u myapp.service --since today
journalctl -u myapp.service --since 09:00 --until "1 hour ago"
journalctl -u myapp.service --since "2016-11-10 12:00" --until "2016-11-10 13:00"

s. https://www.freedesktop.org/software/systemd/man/journalctl.html for more.

Restart on deploy

To allow for automatic restart of the service, e.g. as part of automatic deployment adjust the sudoers file by running sudo visudo and add something like that username ALL = NOPASSWD: /bin/systemctl restart myapp.service so that the user with the name username can restart the service without entering root password

“Old school” logs

Write data to a file (or use off the shelf solution like GitHub - onkel-dirtus/logger_file_backend · GitHub) and manage logs using Linux lorgotate, here is an example config

/var/apps/myapp/log/*.log {
daily
missingok
rotate 18
compress
delaycompress
notifempty
create 660 username groupname    
dateext
dateformat -%Y-%m-%d-%s    
su username groupname
}

must be put in /etc/logrotate.d/myapp, dry run / debug sudo logrotate -d /etc/logrotate.d/myapp, force rotation (for testing) sudo logrotate --force /etc/logrotate.d/myapp. More on logrotate http://www.linuxcommand.org/man_pages/logrotate8.html

Example upstart service

Relevant if using Ubuntu before 16.04 (not sure how it goes between LTS’), this example uses exrm releases:

description "My app daemon"

setuid username
setgid groupname

start on runlevel [2345]
stop on runlevel [016]

expect stop
respawn

env MIX_ENV=prod
export MIX_ENV

env PORT=4000
export PORT

env HOME=/var/apps/myapp
export HOME


pre-start exec /bin/sh /var/apps/myapp/bin/myapp start

post-stop exec /bin/sh /var/apps/myapp/bin/myapp stop

Example deployment script

To run automatically after deployment for rolling update of a phoenix project with no DB (hence no ecto migrate) in staging environment (hence tests):

cd /var/apps/myapp
MIX_ENV=prod mix deps.get
brunch build --production
MIX_ENV=prod mix phoenix.digest
MIX_ENV=prod mix compile
sudo service myapp restart    
mix test

can be triggered by a commit into a specific branch or using web UI, if anything goes wrong the whole thing exits with a non zero status and lets you know about the problem. Makes a nice simple alternative to CI.

25 comments

#wiki #deployment #linux

73 19755 25

2024-07-18 09:10:18 UTC

Most Liked

blackham

I used mix in systemd to launch phx apps for a couple years. But it’s time to replace the simple service with a forked daemon. Here is where I’ll save my notes so when I have to do it again in 3 months and I google “elixir systemd service” I’ll find them again. Oh, and I guess if it helps someone else out, oh well. That’s the price I pay to be lazy.

Server: Ubuntu 20.04.5 LTS
Version: elixir 1.14.3 (compiled with Erlang/OTP 25) - with asdf
username: core (replace as you need)
project name: court (or court_api, replace as you need)

Build the release version in /opt/court_api

MIX_ENV=prod mix compile
MIX_ENV=prod mix assets.deploy
mix phx.gen.release
mix release

I like to load my entire environment from file rather than set each value in systemd’s config. These settings are all basic stuff except the PHX_SERVER=true is needed to tell the elixir daemon to launch the phoenix server. Also since I’m using asdf I include it’s shims and bin folder in the PATH.

vi /etc/environment-court

PATH="/home/core/.asdf/shims:/home/core/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
SECRET_KEY_BASE=RlfalO4gogetyourownkeyzdNsLE
MIX_ENV=prod
DATABASE_URL=ecto://mydbuser:mypassword@192.168.1.257/court
PORT=5428
PHX_SERVER=true

Now setup the systemd service. Note I’m running it as daemon and not start. This forks it in the background or something like that. The takeaway is it is now running in the background. (If I was in Bash, I would get my prompt back)

vi /lib/systemd/system/court.service

[Unit]
Description=Court
After=network.target
Requires=network.target

[Service]
Type=forking
WorkingDirectory=/opt/court_api
User=core
Group=core
Restart=always
RestartSec=5
EnvironmentFile=/etc/environment-court
ExecStart=/opt/court_api/_build/prod/rel/court_api/bin/court_api daemon
ExecStop=/opt/court_api/_build/prod/rel/court_api/bin/court_api stop

[Install]
WantedBy=multi-user.target

Now restart systemd and start the new court service. I use restart because lets be honest, I’ll typo something or want to add RuntimeMaxSec.

systemctl daemon-reload
systemctl restart court

NOTES - Logs
Bad news is now that phx is running in the background, stdout isn’t caught by systemd. TODO: Look for or add a phx setting that will route stdout to syslog. Until that day, find your logs here:

tail -f /opt/court_api/_build/prod/rel/court_api/tmp/log/erlang.log.1

Not sure if elixir will rotate that or not. If not then something like this (not tested)

vi /etc/logrotate.d/elixir-court

/opt/court_api/_build/prod/rel/court_api/tmp/log/*log* {
  daily
  missingok
  rotate 14
  compress
  delaycompress
  notifempty
  create 640 core core
  sharedscripts  
}

NOTES - console
Losing journalctl logs sucks the big one, but this makes up for it! The ability to jump into the console and break things directly!

/opt/court_api/_build/prod/rel/court_api/bin/court_api remote

I hope I find these notes useful in 3 months when I have to do this again. (I hope someone replies back with the news that elixir/phx now detects if it has been launched in daemon mode and redirects stdout to syslogd)

Post #22

blackham

I’m just saving some more notes. While my notes above work for full production systems, the following is easier and works great for some internal microservices. Rather than build it, just run the mix phx.server. This way the logs are sent to systemd so you can follow them with: journalctl -u example -f

I also figured out how to connect to the iex CLI after the process is started with systemd. Warning: I can do this safely because my services are running behind a firewall. I’d read up on the --name “example@127.0.0.1” part of iex/elixir. You are exposing a management port/connection which is super helpful and a little more dangerous if someone naughty has access to that port. Enough talk, here is how I set it up:

sudo vi /etc/environment-example

PATH="/home/core/.asdf/shims:/home/core/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
SECRET_KEY_BASE=RlfalO4gogetyourownkeyzdNsLE
MIX_ENV=prod
DATABASE_URL=ecto://mydbuser:mypassword@192.168.1.257/example
PORT=5428

sudo vi /lib/systemd/system/example.service

[Unit]
Description=Example
After=network.target auditd.service

[Service]
EnvironmentFile=/etc/environment-example
User=core
Group=core
Type=simple
WorkingDirectory=/opt/example
ExecStart=/home/core/.asdf/shims/elixir --name "example@127.0.0.1" -S mix  phx.server
KillMode=process
TimeoutSec=0
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Here is how to connect to the iex/CLI of the running service:

mkdir ~/bin
vi ~/bin/example.sh

#!/bin/sh
exec iex --name console-example@127.0.0.1 --remsh example@127.0.0.1

chmod +x example.sh
~/bin/example.sh

Post #24

yurko

@kif that’s an interesting issue, I have checked a production app that runs this way and can confirm that I have stack trace, the error is one that is raised manually with raise and not one that happened “on itself”.

We use exception tracking via AppSignal so I must have missed the issue with consistency between real exceptions and what we get in logs.

I also searched for a more “natural” error and have not found one, though I did find ActionClauseError of Phoenix which was reported properly in the monitoring (stack trace included), but in the log I have only seen the “Sent 400 in 6ms”, so that might be your case: Phoenix handles the error in some fixed way which makes it normal case and not an exception which loses the error info for the resulting output.

I would assume you have the same situation, you can test it by adding a manual raise call and see what you get.

In any case I am pretty sure the issue has nothing to do with OS but rather with application environment and configuration.

Post #17