Phoenix app runs but nginx returns 502 error

Phoenix Forum Questions / Help
, ,
1

I am running my app with PORT=4001 MIX_ENV=prod mix phx.server. Visiting the app at myserver.com:4001 works fine. However, visiting myserver.com itself returns a 502 Bad Gateway error. It seems that nginx failed to reach the server instance somehow.

The following is my nginx config (I try to keep it as minimal as possible, though I still included the default sections):

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    #include /etc/nginx/conf.d/*.conf;

    upstream my_app {
        server  0.0.0.0:4001;
    }

    server{
        listen 80;
        # Do we really need to add the dot before `myserver`?
        server_name .myserver.com;

        location / {
          proxy_redirect off;
          proxy_pass http://my_app;
        }
    }
}

The prod.exs:

config :app, APP.Endpoint,
  url: [host: "myserver.com"],
  http: [port: 4001],
  cache_static_manifest: "priv/static/cache_manifest.json"

I’m not sure if it has something to do with the iptables setting. I’m not fully familiar with it and it came with some options preconfigured:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
# I added this line
-A INPUT -p tcp -m multiport --dports 80,443,4001 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

I’m not sure if it is causing the problem though I doubt it, since when directly visited from port 4001, the app is able to communicate with the postgres instance on 5432 and perform DB operations totally fine.

I’m not sure what could be the cause here.

2

Try having your upstream point to 127.0.0.1:4001. It’s now pointing to 0.0.0.0 which will never be routable, I think, so Nginx is not going to find your app server.

3

For reference, here is the config I use:

# This block just redirects all requests to HTTPS
server {
  listen 80;
  listen    [::]:80;
  server_name codestats.net;
  location / {
    rewrite ^ https://$server_name$request_uri permanent;
  }

  include generic/sites-general.conf;
}

server {
  listen 443 ssl http2;
  listen   [::]:443 ssl http2;

  root /path/to/codestats/priv/static;
  server_name codestats.net;

  # HTTPS
  ssl_certificate /etc/simp_le/sites/codestats.net/fullchain.pem;
  ssl_certificate_key /etc/simp_le/sites/codestats.net/key.pem;

  include generic/sites-general.conf;
  include generic/sites-https.conf;

  location / {
    try_files $uri @phoenix_codestats;
  }

  location /live_update_socket/ {
    proxy_pass http://127.0.0.1:13370;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location @phoenix_codestats {
     proxy_pass http://127.0.0.1:13370;
  }
}
1 Like
4

Change your upstream to point to 127.0.0.1:4001 and you maybe need to set the ip in your endpoint config too:

config :app, APP.Endpoint,
  url: [host: "myserver.com"],
  http: [ip: {127, 0, 0, 1}, port: 4001],
  cache_static_manifest: "priv/static/cache_manifest.json"
1 Like
5

That didn’t work. And when I change the ip in the endpoint configuration somehow I am not able to directly connect via the port either. I simply get “connection refused” on port 4001 even though nmap localhost does show that 4001/tcp is open…

6

I tried to adapt your config but got the same 502 error… You were running the Phoenix app on 0.0.0.0 right? (i.e. didn’t modify the http field in the endpoint config.)

7

What does your nginx and Phoenix config look like now? My http config looks like this: http: [port: {:system, "PORT"}]. So I don’t touch the bind, but I guess I should bind it to 127.0.0.1.

8

I just modified your configuration minus the SSL part. However it just didn’t work for some reason. Anyways I switched to HAProxy and it worked. No idea why Nginx didn’t. I guess I messed up my Nginx installation in some way but now I’ll just go with HAProxy.

9

It’s impossible to say without seeing your Nginx config and what is printed to the Nginx error log.

10

I think Phoenix needs to listen to 0.0.0.0 and Nginx to proxy_pass to 127.0.0.1

1 Like
11

If Phoenix is not open to the outside, it can just listen on 127.0.0.1 (and it’s better to avoid too “wide” listen binds).

1 Like
12

Turns out I didn’t properly read the error log (They have suffixes such as 06-28 and I just tried to read the most recent one which contained nothing). The error is “Permission Denied” when Nginx was trying to connect to the Cowboy server, which was already asked here: https://stackoverflow.com/questions/23948527/13-permission-denied-while-connecting-to-upstreamnginx. The solution is to change httpd setting in SELinux: setsebool -P httpd_can_network_connect 1. HAProxy isn’t restricted by this setting on httpd, apparently.

2 Likes
13

That sounds unsafe…

But yeah, if you have selinux enabled then it is pretty obvious that you need to allow the connection… ^.^;

A quick set of commands that would generally work in this case would be something like:

# See what it will enable:
sudo cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -m nginxlocalconf > nginxlocalconf.te
view nginxlocalconf.te

# If the above is good then commit the changes:
sudo cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M nginxlocalconf
sudo semodule -i nginxlocalconf.pp

# See what is allowed to httpd by:
sudo semanage port -l | grep http

# You can add a specific port, like say 6080, by something like:
sudo semanage port -a -t http_port_t -p tcp 6080

In essence, if one does not know selinux decently, don’t use it unless required (though it is worth learning, it is very powerful).

2 Likes