So I’ve been banging my head to the wall trying to figure out why my upstream was not being handled by phoenix.

I first checked my nginx configs.
Upstream set to right host and post of 4000
Proxy headers all set
Cert via lets encrypt
Phoenix app confirmed running via logs and curl on localhost:4000 on server
Http2 request coming in on 443 proxy up to phoenix

So something got me thinking, does this request fall back to http1.1 if htt2 does not work?
That lead me to look into the qa section of nginx where I found this little nugget.

Q&A
Q: Will you support HTTP/2 on the upstream side as well, or only support HTTP/2 on the client side?
A: At the moment, we only support HTTP/2 on the client side. You can’t configure HTTP/2 with proxy_pass. [Editor – In the original version of this post, this sentence was incorrectly transcribed as “You can configure HTTP/2 with proxy_pass.” We apologize for any confusion this may have caused.]

I maybe reading this wrong, but if I’m correct this is saying you can’t using http2 requests with upstreaming your request via proxy.

I than cut out my nginx stack from my app and ran cowboy as my primary http server and all is working fine now. In the end the only indiction my request even hit my server was in my nginx error logs where it simply said

upstream prematurely closed connection while reading response header from upstream

well any how I thought I would share my lesson in hopes it saves someone else a few days of their life