Use an absolute path, just to be sure.
Also you haven’t shown your configuration of the pid_file-application.
1 Like
@NobbZ thank you for taking time to look at this.
Somebody pointed to me that in the systemd config file it shouldn’t be Type=forked but Type=forking.
I changed that and used absolute path. Now I get the following error
ubject: Unit union.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit union.service has finished starting up.
--
-- The start-up result is done.
Jun 08 14:58:45 union-staging run_erl[31042]: Erlang closed the connection.
Jun 08 14:58:46 union-staging union[31472]: ==> Generated sys.config in /test/union/var
Jun 08 14:58:46 union-staging systemd[1]: union.service: control process exited, code=exited status=1
Jun 08 14:58:46 union-staging union[31472]: Node union@127.0.0.1 is not running!
Jun 08 14:58:46 union-staging systemd[1]: Unit union.service entered failed state.
Jun 08 14:58:46 union-staging systemd[1]: union.service failed.
Jun 08 14:58:51 union-staging systemd[1]: union.service holdoff time over, scheduling restart.
Jun 08 14:58:51 union-staging systemd[1]: Starting Union Servers...
-- Subject: Unit union.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit union.service has begun starting up.
Jun 08 14:58:53 union-staging union[31766]: ==> Generated sys.config in /test/union/var
Jun 08 14:58:53 union-staging systemd[1]: PID file /test/union/union.prod.pid not readable (yet?) after start.
Did anybody ever encounter that not readable yet pid file ?
for the pid_file application I use
config :pid_file, file: System.get_env("PIDFILE")
and use conform for environment variables with PIDFILE="/test/union/union.prod.pid"
I forgot to mention that if I use the bin script directly, then it works like a charm
/test/union/bin/union start
Is $PIDFILE set during compilation? Please use the form as exampled in the documentation of :pid_file:
config :pid_file, file: {:SYSTEM, "PIDFILE"}
Continueing what @NobbZ last said, if you are using the :SYSTEM tuple then be sure to set the PIDFILE in the ENVIRONMENT in the systemd file too as it will grab that at load time.
1 Like
so looking under the release_folder/var/logs, I find that when starting with systemctl, the database variable seem to be missing
** (RuntimeError) Connect raised a KeyError error. The exception details are hidden, as^M
they may contain sensitive data such as database credentials.^M
^M
(elixir) lib/keyword.ex:377: Keyword.fetch!/2^M
(postgrex) lib/postgrex/protocol.ex:98: Postgrex.Protocol.connect/1^M
(db_connection) lib/db_connection/connection.ex:135: DBConnection.Connection.connect/2^M
(connection) lib/connection.ex:622: Connection.enter_connect/5^M
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3^M
I don’t understand why I am able to start the service with the bin script start like
/test/union/bin/union start and why when I do it with systemctl the db variable is missing.
when I check the sys.config file (file generated after every start by conform https://github.com/bitwalker/conform), I can see that all my environment variables are there.
I’m using conform for environment variable. That is why I haven’t used the system tuple instead of the simple system.get_env. Isn’t conform for not having to use the system tuple ?
I’m guessing that both of you don’t use conform and recommend going with the system tuple everywhere in the app ? (just wondering what’s the best way to do this)
If I use the foreground bin script for execstart then it starts, but fails after the starttimeout (whether I set RemainAfterExit to yes or to no
I had one more question over the benefit of using systemctl. The benefit in this case is to restart in case the erlang VM crashes, correct ? I thought that unless you do something seriously wrong (calling bad NIFs), the chance of the BEAM crashing are almost non existent ? (I could be very wrong on this, this is why I’m asking)
No, I don’t. I don’t even use distillery at the moment…
I’d say, its more to get a unified interface to have runtime configuration.
Not at all, to be honest. Its not available everywhere, used differently through various implementations (sometimes :system, sometimes :System, or even :SYSTEM, etc).
If I remember correctly, some kind of startup hook was introduced to set up configuration during application startup, but I can’t find its documentation right now.
Thats a benefit, not the.
Using systemd does ensure that services you depend on are started before yours (eg. postgres and network) if you write the unit-file correctly. It ensures that your service is restarted after a system reboot if the service is properly enabled.
And to be honest, how easy it is to crash the BEAM, depends on where in your supervision tree a irrecoverable crash occurs. Due to bubbling restarts, a long time outage of the database connection might bring your application down.
Having used heart in production for about 8 years I can safely say it has been more of a hassle than it’s worth - trying to persuade the heart+erlang runtime that it’s time to shut down cleanly without each of them restarting the other is always painful. I can count the number of times I’ve needed heart to restart a hung vm on the fingers of one hand.
1 Like
No clue about Conform, haven’t used it, but it absolutely should work with daemon releases, especially as the same author made both Conform and Distillery, might be a bug that needs to be reported to them? Try running the release backgrounded manually first?
How do you do that with Distillery? This looks like exactly what I should be doing instead of SCPing my .service file up to the server.
Via an overlay template. 
Essentially I just have these added to be release ... do ... end block:
set overlay_vars: [
deploy_user: user,
deploy_group: group
]
set overlays: [
{:template, "priv/templates/my_server.service.eex", "my_server.service"}
]
Still using it to this day. ^.^
3 Likes
@OvermindDL1 how does this work with restarting node through OTP scripts - bin/my_app restart, both Distillery and native releases wise?
I’ve just used it for mix release. It recreated the xxx.pid file, but systemd didn’t restart the process. Pid wasn’t changed either. Not sure if that’s the information you are asking for.
@DiodonHystrix Do you mean that bin/my_app restart restart the application and systemd somehow detected that and recreated pid file?
The part where pid doesn’t change is a bit weird - did new application process had a same pid again after restart? Or it was an error on the systemd side? Why then it recreated pid file?
Was systemd able to pickup application output after restart? I.e. were logs accessible through journalctl?
The library :pid_file recreated the file after the restart.
PID belongs to beam.smp, which is not restarted it seems.
I don’t have syslog connected to app (which is run as daemon), so there is nothing in journalctl. But I do have new logs in erlang.log.1 after the app was restarted.
1 Like
This sums it up when you do a beam restart, it just reloads all the code and restarts the system so the pid file gets overwritten with the same pid, so no change.
1 Like
@OvermindDL1 did you ever bother with allowing to shutdown application through Distillery scripts?
I thought that using proper Restart= option may help to instruct Systemd to not restart application that was shutdown through ./bin/my_app stop but setting on-failure doesn’t seem to work :\
How does the on-failure not seem to work, is your BEAM process not actually going down?
@OvermindDL1 can’t recover the context now, but I do have another issue - epmd process is still running after I stopped the app through systemctl
$systemctl stop sidecar
$ps aux | grep epm
xxxx 2715 0.0 0.0 3916 1592 ? S May16 0:04 /srv/sidecar/erts-11.2/bin/epmd -daemon
My unit file
[Unit]
Description=sidecar service
After=network.target
[Service]
Type=forking
User=xxx
Group=xxx
WorkingDirectory= /srv/sidecar
ExecStart=authbind --deep /srv/sidecar/bin/sidecar start
ExecReload=/srv/sidecar/bin/sidecar reload_config
ExecStop=/srv/sidecar/bin/sidecar stop
PIDFile=/srv/sidecar/sidecar.prod.pid
Restart=always
RestartSec=5
EnvironmentFile=/etc/default/sidecar.env
SyslogIdentifier=sidecar
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
Yep, epmd forks as its own daemon. If you want to control it via systemctl then you need to give it its own service and have the other BEAM based services (erlang/elixir/etc) depend on it so they start after it.
Actually it must be managed externally, as systemd by default starts process in the namespace group so if the process goes down, then all other spawned processes should go down with it.
@lessless check out if there is systemctl status epmd (if you have installed Erlang via system package manager, then there should be the EPMD systemd service created).
1 Like
