Phoenix WebSocket behind proxy

yurko

Recently we had a bug report from the client that states they are unable to connect to WebSocket with Error during WebSocket Handshake: 'Connection' header value must contain 'Upgrade'.

Obviously we have this header so the only thing I can think of is that ours is downcased (people seem to have problems with that).

For this reason I wanted to capitalize “Upgrade” in the “Connection” header, but it’s trickier then it seems: When I run locally and analyze headers I can see that it is already capitalized, but behind the proxy it becomes downcase.

In nginx I capitalized the proxy_set_header Connection value which does nothing. I then tried both add_header Connection "Upgrade" always; and more_set_headers 'Connection: Upgrade'; from headers-more-nginx module, they work but add an extra “Connection” header instead of modifying the existing one which doesn’t seem solve the problem.

I also tried to clear the existing header before setting the new one, things like more_clear_headers 'Connection'; and such but they seem to “miss” the downcased header as if it didn’t exist.

Did anyone have a similar problem? Any info on how exactly the whole thing works? ideas how to solve it?

As I understand it headers go in, get proxied to the app and get downcased on their way back without much control over the whole thing.

At the moment I’m thinking about dropping nginx and using cap_net_bind_service or iptables to get to port 80 but I hope there is an easier solution.

7 comments

/phoenix #nginx

0 4988 7

2017-05-29 07:21:12 UTC

Showing Posts 1 to 7

Show Best Posts
Show All (oldest first)
Show All (newest first)

rms.mrcs

Here is my nginx proxy that I use in production for a Socket.IO server:

location /socket.io {
        proxy_pass http://localhost:8890;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
}

I have this working for a year, already. Hope it helps.

Post #1

yurko

thanks, ours is similar and it works fine for everything we can get out hands on, though it seems there are edge cases that it can’t handle

Post #2

OvermindDL1

Which are precisely? We might be able to test here.

Post #3

yurko

well it’s not as lean as it used to be There is an extra copy pasted location to experiment with the socket route. I can also almost confirm that it’s an nginx problem - we got remote access to the client and could test it without nginx, at least with websocket.org echo test it works without proxy but it didn’t work with it, didn’t try it with the proper client app yet though.

So here is the config:

upstream phoenix {
  server 127.0.0.1:4000 max_fails=5 fail_timeout=60s;
}

server {
  server_name [ip here];
  listen 80;

  location /socket {
    allow all;

    client_max_body_size 50M;

    # a hack to enforce capitalized response header    
    #more_clear_headers 'Connection';
    #more_clear_headers 'connection';
    #more_set_input_headers 'Connection';
    #more_clear_input_headers 'Connection';
    #more_set_headers 'Connection: Upgrade';
    #add_header Connection "Upgrade" always;
    
    proxy_set_header Proxy "";

    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Cluster-Client-Ip $remote_addr;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";

    # allow remote connections
    proxy_set_header Origin '';

    proxy_pass http://phoenix;
  }

  location / {
    allow all;

    client_max_body_size 50M;

    proxy_set_header Proxy "";

    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Cluster-Client-Ip $remote_addr;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";

    # allow remote connections
    proxy_set_header Origin '';

    proxy_pass http://phoenix;
  }
}

Post #4

yurko

It’s not an nginx issue after all, there seem to be a well hidden firewall that messes up port 80 traffic on the application level. Using another port (incl. wss with 443) solves the issue and 80 wouldn’t work with nginx or directly.

Sorry for the confusion and thanks for all your feedback!

Post #5

OvermindDL1

That would do it. What firewall though?

Post #6

yurko

I’m afraid I can’t tell you much on this topic, it should have been a part of network setup since no one was aware of it and we had to exclude every other option and basically “prove” that something happens on the network level for it to come up.

Anyway I don’t think it’s something to worry about in the long run since it messes up all WS traffic on port 80, so at some point it will be removed or updated.

Post #7

— All posts loaded —