Phoenix Channel Disconnect with 1006 error

rurkss

Hello, community!

We are faced a problem, phoenix sockets are reconnected randomly from time to time.
We try to debug and find out that the socket reconnecting because of some channel error( which we join after socket was connected).
We debug error which comes from channel, and it is “undefined”. By the time those reconnection take place, there are no more connection to that channel, i mean no one except this client has joined to the channel.

Here are screenshots of our debugs:

here we may see how socket joins a room for the first time, and than after 9 minutes get an error of channel and went to reconnect whole socket and join room again:

Part 2.
we dig deeper and found out that phoenix close connection with code 1006, see screenshot below, but what does that means? it works fine for several minutes and than goes to disconnect

Please share your thoughts about how to dig into the problem, how to debug such kind of errors and log them

25 comments

/phoenix #troubleshooting #gigalixir

0 6031 25

2021-02-26 12:50:21 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

idi527

1006 code means “Abnormal Closure”:

Used to indicate that a connection was closed abnormally (that is, with no close frame being sent) when a status code is expected.

Maybe check your backend logs for errors / dropped tcp connections. Can be anything really, a buggy browser can unexpectedly drop tcp connections after a few minutes, VMs on amazon sometimes drop long running tcp connections (heard about it from people who tried to use it with apns), so the cause can be both on the client and on the server …

Maybe https://stackoverflow.com/questions/19304157/getting-the-reason-why-websockets-closed-with-close-code-1006 would help

Post #1

rurkss

We migrate to gigalixir.com not long ago, i wonder if it may cause an issue, as far as i know gigalixir creator here as well, @jesse?

Post #2

idi527

Try checking the logs to know for sure.

Post #3

rurkss

well, yes, i am looking for them,
btw, in parallel we connect to digital ocean, same project, same sockets, same channel join. It go smooth for a 15 minutes already.

UPDATE

i have to admit, there is nothing wrong with code, it is still running on digital ocean with no drops (35 mins), unfortunately it is due to gigalixir server.

Post #4

jesse

I wonder if something is “timing out”. Do you know if the websocket connection is inactive during this time? Either way, it’s something we will investigate today. 10min is a suspiciously round number. Something is probably configured wrong somewhere.

Post #5

dom

It could be happening at the load balancer level, but I’d say it’s still worth making your app robust so it reconnects cleanly. Some browsers are aggressive about putting tabs to sleep when they’re not visible so reconnections can be quite frequent.

Post #6

jesse

My current hypothesis, is that the disconnect is due to Google Cloud’s idle tcp connection disconnect after 10m.

Note that idle TCP connections are disconnected after 10 minutes. If your instance initiates or accepts long-lived connections with an external host, you can adjust TCP keep-alive settings to prevent these timeouts from dropping connections. You can configure the keep-alive settings on the Compute Engine instance, your external client, or both, depending on the host that typically initiates the connection. Set the keep-alives to less than 600 seconds to ensure that connections are refreshed before the timeout occurs. The following examples sets the keep-alives to one minute (60 seconds). Note that applications running on Linux systems don’t enable keep-alive by default. Thus server or client need to explicitly set the SO_KEEPALIVE socket option when opening TCP connections (see also Linux TCP Keepalive HOWTO).

Post #7

rurkss

i am not really sure what i have to do to increase tcp connection, if i try
this command
sudo /sbin/sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=5
gives me file not foud

btw, it is not strictly 10 minutes, disconnect may happen right after join was established, you may see on screenshot, it may happen after 2 minutes as well as after 6mins