The hard 3 day lesson I learned trying to use http2 with Phoenix, Cowboy and Nginx

So I’ve been banging my head to the wall trying to figure out why my upstream was not being handled by phoenix.

I first checked my nginx configs.
Upstream set to right host and post of 4000 :white_check_mark:
Proxy headers all set :white_check_mark:
Cert via lets encrypt :white_check_mark:
Phoenix app confirmed running via logs and curl on localhost:4000 on server :white_check_mark::white_check_mark::white_check_mark::white_check_mark:
Http2 request coming in on 443 proxy up to phoenix :fire::fire::fire::fire::fire::fire::fire::fire::fire::fire::fire::fire::fire:

So something got me thinking, does this request fall back to http1.1 if htt2 does not work?
That lead me to look into the qa section of nginx where I found this little nugget.

Q&A
Q: Will you support HTTP/2 on the upstream side as well, or only support HTTP/2 on the client side?
A: At the moment, we only support HTTP/2 on the client side. You can’t configure HTTP/2 with proxy_pass. [Editor – In the original version of this post, this sentence was incorrectly transcribed as “You can configure HTTP/2 with proxy_pass.” We apologize for any confusion this may have caused.]

I maybe reading this wrong, but if I’m correct this is saying you can’t using http2 requests with upstreaming your request via proxy.

I than cut out my nginx stack from my app and ran cowboy as my primary http server and all is working fine now. In the end the only indiction my request even hit my server was in my nginx error logs where it simply said

upstream prematurely closed connection while reading response header from upstream

:poop:

well any how I thought I would share my lesson in hopes it saves someone else a few days of their life

20 Likes

It is worth noting that this is the same with Application Load Balancers when using AWS https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html

Application Load Balancers provide native support for HTTP/2 with HTTPS listeners. You can send up to 128 requests in parallel using one HTTP/2 connection. The load balancer converts these to individual HTTP/1.1 requests and distributes them across the healthy targets in the target group. Because HTTP/2 uses front-end connections more efficiently, you might notice fewer connections between clients and the load balancer. You can’t use the server-push feature of HTTP/2.

6 Likes

Those “subtle” things that may just cause a lot of debugging fun is the reason I try to avoid any moving parts that are not absolutely necessary (not using nginx with phoenix for this reason). Thanks for sharing!

5 Likes

I actually ran in to this issue recently without knowing that it wouldn’t work. I decided to drop Nginx completely and let Phoenix handle the certs and http/2 requests. I wrote about the steps I need to do to make that transition.

9 Likes

@alexgaribay how do you handle an umbrella app (with multiple web apps) with multiple domains all running on port 443?

I tried this too, but hit this limitation

Caddy can reverse proxy using HTTP/2.

Caddy v1 http.proxy documentation.

Caddy v2 reverse_proxy documentation (v2 in beta, with reworked config as compared to v1).

I hope to be setting up Caddy in front of Phoenix soon.

2 Likes

What are the main advantages of using it, instead of Cowboy?

Maybe virtual hosting.

That’s the reason I use nginx in front of Phoenix

I wanted to try this but cant for the life of me get basic redirection to force ssl with plug/cowboy to save my life.
Edit: looks like I figured out my roadblock, will give this a try. thanks for sharing.

Putting a webserver/reverse proxy in front of Cowboy might be necessary in order to serve more than one site/app from the same server. Caddy can automatically get SSL certificates via the ACME HTTP or DNS challenges.

Once you get used to a particular server’s/reverse proxy’s configuration it can be appealing to stick with it across different types of sites: Elixir, Python, PHP, etc. I haven’t yet looked into configuring Cowboy for TLS, but I’m interested to know how easy it is to, for example, disable TLS 1.0 and 1.1.

1 Like

If you want some overload protection (like setting maximum concurrent connections) or not have Erlang handle TLS (https://istlsfastyet.com), or do a deployment without downtime, I’d recommend HAproxy, which can do pretty much anything except serve files.

2 Likes

did you get Caddy running in front of Phoenix umbrella?

So something I didn’t understand right away was that I don’t need to have encryption between the reverse proxy an elixir. Even though the upstream proxy is talking with elixir (cowboy) via http1.1 nginx is still returning h2

So in the end it still works to have nginx running in front of your app.