Unable to access Phoenix app after deployment to Ubuntu 18.04

[phollyer]

I have set up a new Ubuntu 18.04 server and have built and released my phoenix app to production without any problems.

I can ping it with edeliver, and the logs show it as being alive.

When I try and access it over HTTP I am receiving a 502 Bad Gateway error which I assume is something to do with either my Nginx setup or Ubuntu, but I really don’t know.

I initially posted in SO because I figured the problem was more related to Ubuntu or Nginx, the question is here: https://stackoverflow.com/questions/61808572/502-bad-gateway-elixir-phoenix, with more detailed information. I’m sure people around here have the process macked, so if someone can help me here, I would be truly grateful as I’m now at a dead end.

I have posted a Gist detailing all the steps I have taken here: https://gist.github.com/phollyer/cb3428e6c23b11fadc5105cea1379a7c

If anyone can help me I would greatly appreciate it, I will keep the SO question, the Gist, and this post in sync with each other wrt to any solution provided.

I can provide whatever additional information is required.

Thanks

[outlog]

what is in your nginx config - that would be my first guess…

[phollyer]

upstream phoenix {
  server 127.0.0.1:4013 max_fails=5 fail_timeout=60s;
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
  server_name _;
  listen 80;

  location / {
    allow all;

    # Proxy Headers
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Cluster-Client-Ip $remote_addr;

    # The Important Websocket Bits!
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_pass http://phoenix;
  }
}

I don’t yet have a domain name (will be transferred from another server), so am just using the IP address in the browser.

[sfusato]

Did you un-comment this line in prod.secret.exs?

config :myapp, MyAppWeb.Endpoint, server: true

[phollyer]

This is my prod.exs

config :app, AppWeb.Endpoint,
  load_from_system_env: false,
  url: [host: "127.0.0.1", port: 4013],
  cache_static_manifest: "priv/static/cache_manifest.json",
  check_origin: true,
  root: ".",
  version: Mix.Project.config[:version]

config :logger, level: :info

config :phoenix, :serve_endpoints, true

import_config "prod.secret.exs"

My prod.secret.exs only has the DB config and the secret_key_base

[sfusato]

Open up config/prod.secret.exs and you should find a section about “Using releases” with a configuration to set. Go ahead and uncomment that line or manually add the line below, adapted to your application names:

config :my_app, MyApp.Endpoint, server: true

This is from the Phoenix Guide about deploying with releases.

[phollyer]

OK, added the line, (changed for my app name), built a new release and deployed it, but no joy.

Same 502 error.

[outlog]

open up the port in ufw and see if phoenix is running - check the logs - I suspect csrf/origin issues since you have “127.0.0.1” as host in phoenix config (you can try host: nil to get going)

once phoenix is confirmed as running properly the next step is nginx

I’m not up to date on nginx - but if it doesn’t work try making it look like https://gist.github.com/ziazek/e75fbe5ffa98febca3e78e087087f81b#file-deploy_phoenix from:

Deploy Early and Often: Deploying Phoenix with Edeliver and Distillery (Part...

[Ninigi]

Can you run a remote console? Does the phoenix log say anything? The nginx logs?

[NobbZ]

@phollyer

url: [host: "127.0.0.1", port: 4013],

This only sets how you generate URLs using the path-helpers.

It will not affect on which port tthe cowboy will listten for requests.

Pleasee use netstat -lnpt | grep beam to see on which port the BEAM is listening, if at all. Also please check the already mentioned option for configuring the endpoint to be started at all.

You probably want to set http: [port: 4013] as your endpoint config to listen on 4013. Also, you probably do not want your URL to include the :port field, or generated links will circumvent the RP in front of it. You also do not want to have 127.0.0.1 as :host, as that willl generate URLs pointing you to your local machine, rather than the server.

[phollyer]

@sfusato

config :my_app, MyApp.Endpoint, server: true

That appears to be a typo in the docs, or not updated in line with Phoenix, I’ve just started a new Phoenix 1.5.1 project and in prod.secret.exs it says:

config :hello, HelloWeb.Endpoint, server: true

I’m only mentioning it in case someone else comes across this thread, and doesn’t realise the Endpoint module has moved to the MyAppWeb dir.

[Ninigi]

@NobbZ

@phollyer

url: [host: "127.0.0.1", port: 4013],

This only sets how you generate URLs using the path-helpers.

That is not entirely true - I am guilty of writing code that uses this setting to determine certain base domains.

@NobbZ

It will not affect on which port tthe cowboy will listten for requests.

Which does not matter for nginx - it will only listen on ports you tell it to listen to, and that is the port you tell phoenix to listen to. I’d be extremely surprised if this was the root of the problem.

@NobbZ

Pleasee use netstat -lnpt | grep beam to see on which port the BEAM is listening, if at all. Also please check the already mentioned option for configuring the endpoint to be started at all.

Don’t do this please unless you know what you are doing. Why would you want to tell nginx a port “the BEAM is listening” to? The prod config for your phoenix server should set the port, if you have to do a workaround like this, something is very, very wrong!

@NobbZ

You also do not want to have 127.0.0.1 as :host , as that willl generate URLs pointing you to your local machine, rather than the server.

This is true as far as I know (not an nginx expert…) All I can tell you is that I am usually using the domain the app is hosted on as the :host option, so if you have a domain bara.com and a DNS entry to haha.bara.com try to change the :host to haha.bara.com.

[phollyer]

@Ninigi

Don’t do this please unless you know what you are doing.

Aggghhh, that was the first thing I did. I should probably say that I’m a hobbyist, having used Rails and Phoenix in the past for my business. I tend to learn ‘just what I need to’ to get a job done, so as far as netstat goes, I’d say I definitely don’t know what I’m doing

The Phoenix app in question is one that I updated from an earlier 1.3 app, following the guide to do so, but didn’t have any problems with it on my Centos server, but I’ve messed that server up the other day. I can’t build a release on it, so I started a new Ubuntu box because it’s packages seem to be more up to date wrt to Elixir.

I’ve taken a quick look around the base Hello app generated from scratch and there’s a few things which look new so I guess I’ve got some reading to do.

I’ll address all the other suggestions shortly.

[NobbZ]

@Ninigi

Which does not matter for nginx - it will only listen on ports you tell it to listen to, and that is the port you tell phoenix to listen to. I’d be extremely surprised if this was the root of the problem.

Yes, NGINX listens on ports it was told to listen to, and forward requests to phoenix, cowboy, beam, whomever it was configured to forward to. In this particulart case it was configured to forward localhost:4013, but phoenix wasn’t told to listen on that port.

So, indeed, it does matter. And phoenix not beeing available were NGinx expects it, is the root of the problem.

@Ninigi

Don’t do this please unless you know what you are doing.

As I said, this is just to check if some BEAM process is listening at all on a TCP port. Of course, that might or might not be the web application we want to get up and running. But this is part of the tedious research whether the app is up and running as a server or not.

[phollyer]

@NobbZ

Pleasee use netstat -lnpt | grep beam to see on which port the BEAM is listening, if at all. Also please check the already mentioned option for configuring the endpoint to be started at all.

If this is supposed to provide an output with some helpful information, it didn’t. Nothing was output. So does that mean BEAM isn’t listening on a port?

[NobbZ]

That means that no beam related process were listening on any TCP port.

[Ninigi]

@phollyer

url: [host: “127.0.0.1”, port: 4013]

@NobbZ

Yes, NGINX listens on ports it was told to listen to, and forward requests to phoenix, cowboy, beam, whomever it was configured to forward to. In this particulart case it was configured to forward localhost:4013, but phoenix wasn’t told to listen on that port.

OK?

@NobbZ

As I said, this is just to check if some BEAM process is listening at all on a TCP port. Of course, that might or might not be the web application we want to get up and running. But this is part of the tedious research whether the app is up and running as a server or not.

Yes, a lot easier than trying to start remote_console, which will tell you if the node is running or not.

Sorry I am so smug about this, but what you are saying does not make sense to me - and you like to be very smug about your replies.

I’ll stop being such a snug a*hole from this post on.

[NobbZ]

Not sure what you mean by “snug”.

But, yes, we need to know both is the application running (can be verified by a remote console) and is it listening at all or even on the port we need it to listen on.

We now have verified, that the latter is not the case. Still it would be nice to know if its running at all.

Therefore I have to agree, not all questions asked from this thread have been answered.

For now, from all informations available in this thread we have to assume that the application is running, but not starting the HTTP server and therefore nginx can’t forward requests and replies with a 502.

Also we have to assume that even if the endpoint gets started, it doesn’t get started at the port we want it to have, as no port to listen on is specified in any of the configs we can see. I do not remember which port was used by default, but I’m pretty sure, it is not 4013, but 4013 is expected by nginx, so it will still reply with 502.

So, as already has been mentioned, adding not only the config line for starting the server at all, but also adding http: [port: 4013] to the endpoint config, would help.

Also port: 4013 should be removed from the :url endpoint config, as it will affect how URLs will be created. And as there is an nginx listen in front for RPing the traffic, it should be left empty, such that phoenix will use the default port for each schema when generating links.

[Ninigi]

On Ubuntu server, you could use htop which should show your app as (at least) one process. If it is not, then your app is probably not running. But once again, I don’t see how that gives you any information over your_app/bin/your_app remote_console

[phollyer]

@NobbZ

Therefore I have to agree, not all questions asked from this thread have been answered.

I will do my best, I’ve just been watching you guys chatting it about it.

your_app/bin/your_app remote_console starts a console.

I’ve checked the logs, and the app is logging itself as alive every 15 mins, but nothing more.

The last 4 lines of the Nginx logs:

2020/05/15 13:39:32 [error] 30363#30363: *30 connect() failed (111: Connection refused) while connecting to upstream, client: 198.108.66.214, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4013/", host: "109.228.46.36"
2020/05/15 13:48:01 [error] 30363#30363: *33 connect() failed (111: Connection refused) while connecting to upstream, client: 2.179.118.158, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4013/", host: "109.228.46.36:80"
2020/05/15 14:19:52 [error] 30363#30363: *35 connect() failed (111: Connection refused) while connecting to upstream, client: 142.93.183.128, server: _, request: "GET / HTTP/1.0", upstream: "http://127.0.0.1:4013/"
2020/05/15 14:21:46 [error] 30363#30363: *37 connect() failed (111: Connection refused) while connecting to upstream, client: 49.233.216.230, server: _, request: "GET / HTTP/1.0", upstream: "http://127.0.0.1:4013/"