Help with Elixir 1.9 systemd unit file

Questions / Help
, ,
1

Just getting up to speed on Elixir 1.9 deployments. I’m super excited about not relying on Distillery and eDeliver and even Docker containers (since I’m currently just tackling a smaller personal project), but I’m still getting caught up with how to properly do a release that runs in the background, via systemd.

I added https://github.com/cogini/mix_systemd to my deps and generated my systmd unit file and placed it in /etc/systemd/system and did some tests, but can’t get my app to show up at mysite:4000. Keep in mind, I originally just tested the build executable and was able to successfully see my app at that URL so I know the problem is somewhere with how I’m doing my systemd.

Here’s a copy of my systemd unit file:

[Unit]
Description=dayoff service
After=local-fs.target network.target

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/home/deploy/build/dayoff_1.9/_build/prod/rel/dayoff
ExecStart=/home/deploy/build/dayoff_1.9/_build/prod/rel/dayoff/bin/dayoff start
ExecStop=/home/deploy/build/dayoff_1.9/_build/prod/rel/dayoff/bin/dayoff stop
Environment=LANG=en_US.utf8
Environment=MIX_ENV=prod

Environment=PORT=4000
LimitNOFILE=65535
UMask=0027
SyslogIdentifier=dayoff
Restart=always

[Install]
WantedBy=multi-user.target

If i run a status check on the systemd service I see that it is active, however it seems to be starting and stopping every couple seconds.

● dayoff.service - dayoff service
   Loaded: loaded (/etc/systemd/system/dayoff.service; disabled;
   Active: active (running) since Sun 2020-02-23 10:24:49 EST; 7
  Process: 20886 ExecStop=/home/deploy/build/dayoff_1.9/_build/p
 Main PID: 20944 (beam.smp)
    Tasks: 5
   Memory: 6.4M
      CPU: 48ms
   CGroup: /system.slice/dayoff.service
           ├─20944 /home/deploy/build/dayoff_1.9/_build/prod/rel
           └─20986 erl_child_setup 65535

Feb 23 10:24:49 seojeek systemd[1]: dayoff.service: Service hold
Feb 23 10:24:49 seojeek systemd[1]: Stopped dayoff service.
Feb 23 10:24:49 seojeek systemd[1]: Started dayoff service.

Another area of potential confusion is where exactly I place my env vars. I was able to build my release successfully using the vars stored on the server in ~/.profile which I think seems to be a fine solution.

However, in my prod.exs I have some config vars for mix_systemd:

config :mix_systemd,
  app_user: "deploy",
  app_group: "deploy",
  base_dir: "/home/deploy/build/dayoff_1.9/_build/prod/rel",
  current_dir: "/home/deploy/build/dayoff_1.9/_build/prod/rel",
  env_vars: [
    "PORT=4000",
  ]

So I’m not sure if I should replace my ~/.profile and place them in my systemd config?

So yeah, any help is appreciated. Again, if I take the ExecStart command from my systemd file it runs so I just want to set it and forget it but can’t get it to stay on.

Thanks.

2

While I like the idea of that tool I highly suggest You to learn how to write your own unit files. It is not that hard to achieve similar level to the one that is presented in Your snipped, but in the long run it gives you more flexibility and insights how to work with systemd.

Are you sure that you are listening on proper interface? Because if you listen on loopback address, then it will be natural that you cannot connect to that service.

Active service do not mean healthy. It just mean that the application started and that is it.

Such behaviour mean that the application cannot start in reality, you need to check logs for informations about failures in your application, as it seems that the application starts (from the viewpoint of systemd) then internally it cannot start some process, so it restarts until you rect supervisor restart limit, then it shuts down init process and whole process starts again. If you want better insights what is happening in your application you can check out my systemd library and example project where I use watchdog to monitor real state of the application.

2 Likes
3

I’m not sure what you mean here. If I run my app without systemd, it works fine on port 4000 so I’m assuming things are setup fine for that port.

I’ll try out your module. But here’s a question for you. Why am I relying on watchdog and not journalctl as seen here: GitHub - cogini/mix-deploy-example: Example Elixir app which uses mix_systemd and mix_deploy to deploy ?

Also, here’s an interesting comment hinting that maybe I shouldn’t rely on watchdog? linux watchdog and systemd watchdog - Unix & Linux Stack Exchange