Works fine with type simple and releases, seems easy enough. S. " 1.9 and up: using releases" in the above article.
yurko
jihantoro
WORKING STEP TO DEPLOY PHOENIX WITH SYSTEMD
- make sure
Can't run phoenix with elixir 1.9.0 release - #3 by jihantoro - do mix release, for example
SECRET_KEY_BASE='MBwDNPWb9nw5K/Cm/QJ62dgKeU7OM76hH7hVz9HMo7f2fasurhNqGNaMsFh3Ll' DATABASE_URL='ecto://user:password@host/database' PORT=4002 MIX_ENV=prod mix release - Change this systemd config file
[Unit]
Description=My Phoenix App
After=network.target
[Install]
WantedBy=multi-user.target
[Service]
Environment="HOME=/var/app/my-phoenix-app"
ExecStart=/var/app/my-phoenix-app/_build/prod/rel/my-phoenix-app/bin/my-phoenix-app start
ExecStop=/var/app/my-phoenix-app/_build/prod/rel/my-phoenix-app/bin/my-phoenix-app stop
SyslogIdentifier=simple
Restart=always
# 'StartLimitInterval' must be greater than 'RestartSec * StartLimitBurst' otherwise the service will be restarted indefinitely.
# https://serverfault.com/a/800631
RestartSec=5
StartLimitBurst=3
StartLimitInterval=10
0
kif
@yurko thank you for making this post! This is great info. I’m wondering if you get stack trace logs when exception happen in your app? I’m having trouble getting that errors with journalctl -u <app>. Linking my original post here.
1
yurko
@kif that’s an interesting issue, I have checked a production app that runs this way and can confirm that I have stack trace, the error is one that is raised manually with raise and not one that happened “on itself”.
We use exception tracking via AppSignal so I must have missed the issue with consistency between real exceptions and what we get in logs.
I also searched for a more “natural” error and have not found one, though I did find ActionClauseError of Phoenix which was reported properly in the monitoring (stack trace included), but in the log I have only seen the “Sent 400 in 6ms”, so that might be your case: Phoenix handles the error in some fixed way which makes it normal case and not an exception which loses the error info for the resulting output.
I would assume you have the same situation, you can test it by adding a manual raise call and see what you get.
In any case I am pretty sure the issue has nothing to do with OS but rather with application environment and configuration.
3
kif
Thank you for the reply!
You right, it has nothing to do with with systemd. I posted some update to the issue.
What you’re saying is interesting. “Phoenix handles the error in some fixed way which makes it normal case and not an exception”
Why do you think this is implemented this way? Isn’t that crucial to get the root cause of failure in production log with nice stack trace and not just “Sent 400 in 6ms”? 
0
yurko
Isn’t that crucial to get the root cause of failure in production log with nice stack trace and not just “Sent 400 in 6ms”?
Well, as I mentioned - I see that info in the monitoring system, so it depends on your error handling - it’s not lost, just unused basically. Even if you don’t handle errors explicitly, you still do using defaults and if these defaults don’t work for you you can change them.
It’s about what your app considers important / exceptional and what is normal (for your app). Say a request to a non existing route or anything similar that is converted to a 4xx falls into the “not exceptional” category so you get no stack trace and if you raise an exception manually you will see it (probably, I did - see above).
If you want to know exactly what is happening and why, you can look at the source and related issues, I was not able to pinpoint the current state and how we got there quickly enough, but here are few issues to get you started:
Error handling in Phoenix · Issue #482 · phoenixframework/phoenix · GitHub that’s more about initial reasoning
Don't log an ErlangError if we're handling the error · Issue #340 · phoenixframework/phoenix · GitHub this one is closer to your issue
The change seems to remove the logic which kind of did the “view hack”, just not in the view and not in a hacky way Do not log errors if handled by user. Close #340 · phoenixframework/phoenix@701628b · GitHub
PS since it’s not related to OS in any way, we got a bit off topic here.
0
kif
What monitoring system are you using? I’m basically using the same systemd setup that you outlined in this topic.
I looked at you links. There are good bits of information in there. It would be great if I have more experience with Elixir for comprehending the source code. I started to learn Elixir and Phoenix 2 month ago.
Thank you for clarification on my questions. Yes we went a bit of topic.
0
yurko
What monitoring system are you using?
We use AppSignal Application Monitoring for Elixir applications | AppSignal and there I also see stack traces for errors that are not in my log, at least in case of mentioned ActionClauseError.
1
blackham
I used mix in systemd to launch phx apps for a couple years. But it’s time to replace the simple service with a forked daemon. Here is where I’ll save my notes so when I have to do it again in 3 months and I google “elixir systemd service” I’ll find them again. Oh, and I guess if it helps someone else out, oh well. That’s the price I pay to be lazy.
Server: Ubuntu 20.04.5 LTS
Version: elixir 1.14.3 (compiled with Erlang/OTP 25) - with asdf
username: core (replace as you need)
project name: court (or court_api, replace as you need)
Build the release version in /opt/court_api
MIX_ENV=prod mix compile
MIX_ENV=prod mix assets.deploy
mix phx.gen.release
mix release
I like to load my entire environment from file rather than set each value in systemd’s config. These settings are all basic stuff except the PHX_SERVER=true is needed to tell the elixir daemon to launch the phoenix server. Also since I’m using asdf I include it’s shims and bin folder in the PATH.
vi /etc/environment-court
PATH="/home/core/.asdf/shims:/home/core/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
SECRET_KEY_BASE=RlfalO4gogetyourownkeyzdNsLE
MIX_ENV=prod
DATABASE_URL=ecto://mydbuser:mypassword@192.168.1.257/court
PORT=5428
PHX_SERVER=true
Now setup the systemd service. Note I’m running it as daemon and not start. This forks it in the background or something like that. The takeaway is it is now running in the background. (If I was in Bash, I would get my prompt back)
vi /lib/systemd/system/court.service
[Unit]
Description=Court
After=network.target
Requires=network.target
[Service]
Type=forking
WorkingDirectory=/opt/court_api
User=core
Group=core
Restart=always
RestartSec=5
EnvironmentFile=/etc/environment-court
ExecStart=/opt/court_api/_build/prod/rel/court_api/bin/court_api daemon
ExecStop=/opt/court_api/_build/prod/rel/court_api/bin/court_api stop
[Install]
WantedBy=multi-user.target
Now restart systemd and start the new court service. I use restart because lets be honest, I’ll typo something or want to add RuntimeMaxSec.
systemctl daemon-reload
systemctl restart court
NOTES - Logs
Bad news is now that phx is running in the background, stdout isn’t caught by systemd. TODO: Look for or add a phx setting that will route stdout to syslog. Until that day, find your logs here:
tail -f /opt/court_api/_build/prod/rel/court_api/tmp/log/erlang.log.1
Not sure if elixir will rotate that or not. If not then something like this (not tested)
vi /etc/logrotate.d/elixir-court
/opt/court_api/_build/prod/rel/court_api/tmp/log/*log* {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 640 core core
sharedscripts
}
NOTES - console
Losing journalctl logs sucks the big one, but this makes up for it! The ability to jump into the console and break things directly!
/opt/court_api/_build/prod/rel/court_api/bin/court_api remote
I hope I find these notes useful in 3 months when I have to do this again. (I hope someone replies back with the news that elixir/phx now detects if it has been launched in daemon mode and redirects stdout to syslogd)
4
axelson
Scenic Core Team
Have you tried using GitHub - hauleth/erlang-systemd: systemd utilities for Erlang applications · GitHub by @hauleth to help with the logs?
3