Phoenix.PubSubServer messages overloaded

if you are using Phoenix Live Dashboard you can go there and see the list of processes, memory usage and message queues of processes. This could help you pin point the issue.

My hunch is that the message queue that is piling up is on your GenServer, that it’s single process doing the work and not distributing it to a pool of workers?

Unfortunately we do not use Phoenix Dashboard at the moment.
Well, the gen server is implemented by Phoenix, Phoenix.PubSub.PG2Server to be precise. There’s only one instance of this gen server running on each Elixir node.

Alright, but what is listening to these events? There’s some sort of process that receives and processes them that you control, right?

We also need to know if other parts of the system are causing contention with the schedulers. Do you have any unbounded message queues in the app (such as those subscribed to the incoming pubsub messages)? Any other large message queues that appear alongside the pg2 server’s inbox staring to grow? Once you start get unbounded msgque buildup, other processes can start falling behind or calls start timing out even tho they are not the real cause as the scheduler tries to keep up. I’m not completely ruling out pg2server being your bottleneck, but we need to know a lot more to say what’s going on or to place the blame there. Our synthetic benchmarks of pubsub have handled 500k msg/sec on my macbook. What kind of broadcast rate are you pushing thru the cluster when you see this?

Are you filtering messages, thus not consuming all the incoming ones?

If so, then the mailbox will start to grow and for each received message the mailbox will be parsed from the oldest to the newest, until it matches a message you want to consume, thus creating unnecessary work that will affect the throughput.

To solve this issue you need to have a catch all in order to consume the message, otherwise the runtime will not have it removed from the mailbox, and at some point in time you can even run out of memory and bring the entire system down.

Thanks for your replies. Regarding your question:

Do you have any unbounded message queues in the app (such as those subscribed to the incoming pubsub messages)?

AFAIK, there are no other unbounded message queues in our system, at least for the part which we written. Not sure if that’s the case for our external libraries.

Any other large message queues that appear alongside the pg2 server’s inbox staring to grow?

I will look into this next time it happens again.

What kind of broadcast rate are you pushing thru the cluster when you see this?

Just broadcast to a given topic with Phoenix.Endpoint.broadcast

Yep the processes which subscribe to those event are in our control but they are doing nothing beyond pushing message down to clients through web sockets. And since they only receive messages from PG2 server without any blocking (the server just send and forget), I highly doubt that they are the problem.

I’m asking specifically any idea of the rate of broadcasts? Even an estimate would help get an idea of something beyond pubsub is more likely the cause.

Currently i’m measuring the message in rate of the Phoenix.PubSub server with :sys.statistics. The rate is around 1400-1600 messages/s even in times when the message queue starts to piles up, it barely increase. Most of them (i can say 95%) are broadcast messages, the rest are messages from Phoenix.Presence.

Please noted that i’m pooling with 3 seconds interval and the :sys module only measure time in second so the result might be a bit off.

Do you have access to the Erlang observer? Maybe that can help you getting some better measurements