Phoenix.PubSubServer messages overloaded

Goose97

I’m having a hard time debugging this issue related to Phoenix.PubSub on our production system. We have serveral Elixir instances but one is experiencing messages overloaded in the PubSub server. Sometimes the gen server cannot handle incoming messages fast enough so they got piled up in the messages queue. I’m checking the message_queue_len in 3 seconds interval.

Here’s what i know so far:

We are using Phoenix.PubSub with PG2 as adapter
The aforementioned instance is working under normal workload. Connected sockets got spread pretty evenly between all the instances.
During the time when message queue got piled up, we do have a slight increase in incoming messages rate.
AFAIK, the Phoenix.PubSub server is doing nothing but receiving messages from remote process, query from ETS and delegate to local socket processes. There are no heavy tasks involved.

So my question is:

Is there any flaw in my measurement method? I feel like since message_queue_len does not directly reflect message latency, my assumption could be incorrect.
What is the production safe way to debug this issue?

Thanks

19 comments

/phoenix #debugging #troubleshooting

1 3926 19

2021-03-02 11:31:43 UTC

First Post!

Show Best Posts
Show All (oldest first)
Show All (newest first)

hubertlepicki

if you are using Phoenix Live Dashboard you can go there and see the list of processes, memory usage and message queues of processes. This could help you pin point the issue.

My hunch is that the message queue that is piling up is on your GenServer, that it’s single process doing the work and not distributing it to a pool of workers?

Post #1

Most Liked

chrismccord

Creator of Phoenix

We also need to know if other parts of the system are causing contention with the schedulers. Do you have any unbounded message queues in the app (such as those subscribed to the incoming pubsub messages)? Any other large message queues that appear alongside the pg2 server’s inbox staring to grow? Once you start get unbounded msgque buildup, other processes can start falling behind or calls start timing out even tho they are not the real cause as the scheduler tries to keep up. I’m not completely ruling out pg2server being your bottleneck, but we need to know a lot more to say what’s going on or to place the blame there. Our synthetic benchmarks of pubsub have handled 500k msg/sec on my macbook. What kind of broadcast rate are you pushing thru the cluster when you see this?

Post #4

outlog

would it be possible to use pubsub 2.0 - which uses pg instead of pg2?

The pg2 module is deprecated as of OTP 23 and scheduled for removal in OTP 24. You are advised to replace the usage of pg2 with pg. pg has a similar API, but with an implementation that is more scalable. See the documentation of pg for more information about differences. http://erlang.org/doc/man/pg2.html

if nothing just to rule it out..

Post #14

Goose97

We are using OTP 22 so pg is not available yet. But I don’t think we hit the limit of the pg2 module in term of scaling.

Post #16

Last Post!

Goose97

Update on the issue at hand. The Erlang VM schedulers are kinda busy when messages start to piled up. I checked through :erlang.statistics(:run_queue). The exact reason why this happens is still unclear though since the CPU’s utilization (and load average) stays consistently low.

Post #19