Can't upstream https requests to cowboy 2.5 via nginx

Questions / Help
, , , ,
1

Been having a really hard time trying to stand up a server with nginx https and cowboy 2.5.

Before I get in to the errors I’m seeing heres my configs

Nginx Site config.

upstream polymorphic_productions{
	server 127.0.0.1:4000;
}

map $http_upgrade $connection_upgrade {
	default upgrade;
	'' close;
}

server {
	listen 80 default_server;
	listen [::]:80 default_server;
	server_name polymorphic.productions www.polymorphic.productions;
	return 301 https://$server_name$request_uri;
}

server {
	listen [::]:443 ssl default_server ipv6only=on; # managed by Certbot
		listen 443 ssl default_server;
		ssl_certificate     /etc/letsencrypt/live/polymorphic.productions/fullchain.pem; # managed by Certbot
		ssl_certificate_key /etc/letsencrypt/live/polymorphic.productions/privkey.pem; # managed by Certbot
		include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
		ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


	server_name polymorphic.productions www.polymorphic.productions;

	location / {
		try_files $uri @proxy;
	}

	location @proxy {
		include proxy_params;
		proxy_redirect off;
		proxy_pass http://polymorphic_productions;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header Host $host;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";

	}
}

Heres my endpoint config

config :polymorphic_productions, PolymorphicProductionsWeb.Endpoint,
  http: [
    port: 4000
    # cipher_suite: :strong,
    # keyfile: "/etc/letsencrypt/live/polymorphic.productions/privkey.pem",
    # certfile: "/etc/letsencrypt/live/polymorphic.productions/cert.pem",
    # cacertfile: "/etc/letsencrypt/live/polymorphic.productions/chain.pem"
  ],
  url: [host: "polymorphic.productions", port: 4000],
  cache_static_manifest: "priv/static/cache_manifest.json",
  server: true

I can confirm the app will run on my server

deploy@polymorhpic:~$ ./app/polymorphic_productions/bin/polymorphic_productions foreground
[Server IP] [info] Running PolymorphicProductionsWeb.Endpoint with cowboy 2.5.0 at http://polymorphic.productions:4000

in another term

deploy@polymorhpic:~$ curl localhost:4000
<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="utf-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=edge" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, shrink-to-fit=no">
  <meta name="description" content="Portland's Digital Curator">
  <meta name="author" content="Josh Chernoff <jchernoff@polymorhpic.productions>">

  <title>Polymorphic Productions</title>
  <link rel="stylesheet" href="/css/app-005b180feb294658404a412667531133.css?vsn=d"/>
  <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.2/css/all.css" ......

also my foreground logs show

[Server IP] request_id=2lhmfja4sd16m1u03k00007h [info] GET /
[Server IP]  request_id=2lhmfja4sd16m1u03k00007h [info] user=nil message="anonymous user"
[Server IP]  request_id=2lhmfja4sd16m1u03k00007h [info] Sent 200 in 1ms

Now from the nginx side.
tailing my access logs I see the following when I attempt to hit the server via the url.

[MY IP] - - [02/Nov/2018:20:15:46 +0000] "GET / HTTP/1.1" 400 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36"```

Nothing shows up under the nginx error logs nor under my foreground logs nor in my erlang logs

Any idea what I could be doing wrong? The only error information I get is in the browser via:

Request URL: https://polymorphic.productions/
Request Method: GET
Status Code: 400 Bad Request
Remote Address: ....
Referrer Policy: no-referrer-when-downgrade

I’d also like to add, from the perspective of my app, I can run cowboy in place of nginx and it will even work with http2 requests. The problem I believe I’m seeing is in the upstream of the request from nginx to cowboy.

1 Like
2

Sorry for being off topic, but port: 4000 here would probably make your absolute links look like

<a href="https://polymorphic.productions:4000/<...>"></a>

You might want to replace it with 443 or 80 instead.

3

Yeah I figured that much, I forgot to change it back after I was testing different ideas. I just wanted to rule out that causing the requests to be ignored.

4

As far as I understand https://trac.nginx.org/nginx/ticket/923, you can’t RP HTTP2.

5

Correct me if I’m wrong but I don’t think I’m passing http2 requests.

[MY IP] - - [02/Nov/2018:20:15:46 +0000] "GET / HTTP/1.1" 400 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36"```

Also I did just write this and this current ticket is my attempt to rollback from http2 but I’m seeing the same issues I had then.

6

Hmmm, I thought the upgrade stuff was HTTP2 related in your config. My failure then.

I’m wondering why nginx claims to send a 200 but phoenix and your browser state a 400…

7

Good question, I dont know. I’ve been trying to figure that out my self.

Something to note: Phoenix never gets the request as far as I can tell, it’s simply the browser and nginx.

Also I’ve noticed that if I turn off the phoenix app I do get a 502 from nginx so its possible the request is making it to phoenix, but if so it never logs the request

8

I thought this was from your phoenix log…

9

nope thats the access logs from nginx, also as noted there are no error logs on nginx and no logs what so ever coming from erlang

10

400 usually means that the clients request was malformed in a away. Can you try curl with verbose output against nginx?

11

And I took another look at your config. And compared it to a reverse proxy we have in production. It seams as if you were trying to upgrade every incoming request to a websicket connection. You need to proxy_oass regular HTTP and websicket separately.

2 Likes
12

Hard to say just what it was but I started over and tried a different approach and now its working.

I mostly followed the example found here

thanks for the help.