Advice and guidance on operating phoenix in production

Questions / Help
1

Hi everyone,

I published elxsy as my first phoenix project and put it in production already. I like to live dangerously and learn from trial error, mistakes, what can I say :slight_smile: It is very early and if I am doing something wrong, I can pivot very easily.

Let me tell you about my setup.

I run on a VM that I look after and maintain. I build with the following setup on the production VM as my dev machine is completely different and not using any build machines, services as of now. From farm to the table!

0 to release - build.sh

git pull

# Initial setup
mix deps.get --only prod
MIX_ENV=prod mix compile

# Compile assets
npm install --prefix ./assets
npm run deploy --prefix ./assets
mix phx.digest

# Run migrations
MIX_ENV=prod mix ecto.migrate

# Build the release and overwrite the existing release directory
MIX_ENV=prod mix release --overwrite

patch.sh

git pull

# Initial setup
MIX_ENV=prod mix compile

# Build the release and overwrite the existing release directory
MIX_ENV=prod mix release --overwrite

service .... stop
service ... start

service systemd definition

[Service]
Type=simple
User=...
Group=...
WorkingDirectory={PATH}/_build/prod/rel/{NAME}
ExecStart={PATH}/_build/prod/rel/{NAME}/bin/{NAME} start
ExecStop={PATH}/_build/prod/rel/{NAME}/bin/{NAME} stop
...
Environment=MIX_ENV=prod
Environment=PORT=.....
LimitNOFILE=65535
UMask=0027
SyslogIdentifier=...
Restart=always


[Install]
WantedBy=multi-user.target

I run the service on a port and NGINX at the front as a reverse proxy.

Questions:

  1. I have had a bug and received many internal server errors but the syslogs were all OK! Just the request and session logs. I was monitoring the logs and triggering the error to spot the bug but I couldn’t. Is there something wrong with my setup that errors didn’t log?
  2. I would like to push non breaking, db free updates in blue-green deployment manner without downtimme. patch.sh is a workaround for this but still involves service stop and start. Ideally bring the new version up and then take down the old.
  3. I would like to push db breaking changes actively ideally if not minimum downtime with active, passive where possible.

I have some ideas, I know a bit about the nodes but my lack of experience and knowledge is bigger than what I know about running phoenix apps.

I wanted to check if I am reinventing the wheel here, solving an already solved problem and make use of your experiences before I attempt the work.

All ears for your recommendations :slight_smile:

Thank you

2 Likes
2

very quiet :confused:

Am I asking basic questions that could be found with research or questions that no one experienced in the forum :frowning: ?

3

Your setup is normal; however I don’t understand your questions. There is nothing in your setup about logs, so I’d assume you intent to log to stdout and capture those with systemd and redirected to syslog? Was it not happening? If so, you need to consult systemd’s manual.

For continuous deployment you have 2 choices: hot code update or just restart. Hot code update is quite tricky and I have no idea; my service restart in seconds so that’s what I do.

4

It is happening, I am seeing the logs in the journal filtered for the service and systemd.

My problem was when I had a bug in the template and it was throwing Internal server error, nothing was being captured in the logs for the ERROR.

I thought there might be a delay and waited waited but nothing came up, logs were resuming as normal with other user’s requests.

Yup that is what I am after :slight_smile: Thank you for the response

5

Well, your setup is quite simple.

For blue/green I like to have the app as stateless as possible, and put two instances of a release on different ports behind nginx. Since all state is in the DB, I start the second release, switch the exposed release with nginx, and stop the first release. Of course this is scripted to minimize manual work. A friend made a whole ansible configuration for that. A CI/CD system could also do that.

This setup made my life easier, but again my apps do not contain state that isn’t backed by the DB. I have some Agents, some GenServers, but they’re writing to the DB and reading from it in the background. We also run postgres, nginx and elixir on the same machine, it’d be a bit more setup if it spanned multiple machines.

This way, my virtual servers are almost disposable and only need nginx/postgres.

6

Maybe you did not capture both stdout and stderr? I am not familiar with systemd so I can’t tell you how.

7

That sounds like what I had in mind to do blue/green. Thank you for sharing.

I want to introduce CI/CD wayyyy later to be honest when contributors is more than me - if ever :smiley:

8

That would make sense! Let me dig into that, I am not that strong on systemd grounds either!

9

I reckon I see where you’re coming from on the introduce CI/CD way later, but, I’ll throw this here regardless (so you can keep it at the back of your mind).

2 Likes
10

Thank you bookmarked it :slight_smile:

11

It is capturing both by default but the problem is logging output is not matching systemd,

it is a nested log currently. journal can not differentiate the log.level so coming in as p info and above :frowning: