The hard 3 day lesson I learned trying to use http2 with Phoenix, Cowboy and Nginx

phoenix
cowboy
nginx
http2

#1

So I’ve been banging my head to the wall trying to figure out why my upstream was not being handled by phoenix.

I first checked my nginx configs.
Upstream set to right host and post of 4000 :white_check_mark:
Proxy headers all set :white_check_mark:
Cert via lets encrypt :white_check_mark:
Phoenix app confirmed running via logs and curl on localhost:4000 on server :white_check_mark::white_check_mark::white_check_mark::white_check_mark:
Http2 request coming in on 443 proxy up to phoenix :fire::fire::fire::fire::fire::fire::fire::fire::fire::fire::fire::fire::fire:

So something got me thinking, does this request fall back to http1.1 if htt2 does not work?
That lead me to look into the qa section of nginx where I found this little nugget.

Q&A
Q: Will you support HTTP/2 on the upstream side as well, or only support HTTP/2 on the client side?
A: At the moment, we only support HTTP/2 on the client side. You can’t configure HTTP/2 with proxy_pass. [Editor – In the original version of this post, this sentence was incorrectly transcribed as “You can configure HTTP/2 with proxy_pass.” We apologize for any confusion this may have caused.]

I maybe reading this wrong, but if I’m correct this is saying you can’t using http2 requests with upstreaming your request via proxy.

I than cut out my nginx stack from my app and ran cowboy as my primary http server and all is working fine now. In the end the only indiction my request even hit my server was in my nginx error logs where it simply said

upstream prematurely closed connection while reading response header from upstream

:poop:

well any how I thought I would share my lesson in hopes it saves someone else a few days of their life


Elixir Forum 2019 Update!
Can't upstream https requests to cowboy 2.5 via nginx
#2

It is worth noting that this is the same with Application Load Balancers when using AWS https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html

Application Load Balancers provide native support for HTTP/2 with HTTPS listeners. You can send up to 128 requests in parallel using one HTTP/2 connection. The load balancer converts these to individual HTTP/1.1 requests and distributes them across the healthy targets in the target group. Because HTTP/2 uses front-end connections more efficiently, you might notice fewer connections between clients and the load balancer. You can’t use the server-push feature of HTTP/2.


#3

Those “subtle” things that may just cause a lot of debugging fun is the reason I try to avoid any moving parts that are not absolutely necessary (not using nginx with phoenix for this reason). Thanks for sharing!


#4

I actually ran in to this issue recently without knowing that it wouldn’t work. I decided to drop Nginx completely and let Phoenix handle the certs and http/2 requests. I wrote about the steps I need to do to make that transition.