bismark

bismark

Phoenix Socket draining

We’re attempting to make our deployment process a little less… disruptive, so we’d like to find a way of manually draining all open websockets.

Background

We’re running on AWS autoscaling groups with an Application Load Balancer. When we deploy, we register the a new instance with the load balancer target group, then deregister the old one. Unfortunately, target groups deregistration is ignorant of websockets, so it waits the 300s for all open connections to complete (which of course the websockets do not do), and then forcefully closes all of the connections. This causes a bit of a stampede (yes, yes, this is partly client issue, let’s just assume the clients are bad actors) and we much prefer to do this more gently.

Question

Is there any “official” way of traversing the open websockets and closing them down? Aside from spelunking through the Supervision tree that is.

Though I’m open to hearing about other options :slight_smile:.

First Post!

rjk

rjk

There is an example on how to close a channel (i.e. websocket) server-side in the docs (Phoenix.Socket — Phoenix v1.8.8)

MyAppWeb.Endpoint.broadcast("users_socket:" <> user.id, "disconnect", %{})

But AFAICS it always needs those ids, which you could gather using presence track which keeps its updates (presence_diff) private.

On a shutdown signal you could iterate these ids and broadcast that above disconnect to your sockets.

Most Liked

bismark

bismark

We’ve come up with a not-so-great workaround for now.

SocketDrano had a couple issues for us:

  • it relied on channel join telemetry events, so it would miss sockets that do not join a channel
  • it was oriented around draining upon sigterm, but we’re relying on CodeDeploy lifecycle events for triggering the draining

So, we ended up “extending” Phoenix.Socket by creating our own Socket module that has overrideable callbacks:

Then we create a custom init callback:

@impl Phoenix.Socket.Transport
   def init(state) do
     super(state)
     |> tap(fn
       {:ok, {_state, %Phoenix.Socket{transport_pid: pid}}} when is_pid(pid) ->
         track_socket(pid)

       _ ->
         :ok
     end)
   end

track_socket then does essentially what SocketDrano does (monitor the socket pids, disconnects them when told to).

I’ve opened a thread on the Phoenix Core mailing list asking about better solutions being added Phoenix itself: https://groups.google.com/g/phoenix-core/c/1umhh2X1oAM/m/_sfxU8xIBgAJ

mweidner

mweidner

For folks coming across this recently: Phoenix sockets now have built-in connection draining (since 1.7.2). You can configure it with the drainer option to socket/3: Phoenix.Endpoint — Phoenix v1.8.8

rjk

rjk

Presence.track can store key/values so you could store hostname and userid? So on a quit signal do a lookup on hostname and you have a list of connected sockets for that node? And you could use Phoenix.PubSub — Phoenix.PubSub v2.2.0 to keep the broadcast local to the node.

Where Next?

Popular in Questions Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130579 1222
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54996 245
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement