Broadcasting thousands of messages causes excessive memory usage

Phoenix Forum Questions / Help
, ,
1

I’m a contributor to Realtime, which listens for Postgres changes via logical replication and broadcasts changes to subscribed clients.

As an example of how Realtime works, if I were to insert 1 million rows in a transaction, then Realtime would first save all 1 million inserts to process state, then loop through each insert (struct with data about the insert) and broadcast eight times, once for each topic.

I’ve been trying to maximize the number of inserts in a transaction the Realtime server can handle on an AWS instance with about ~600MB of available memory. The test I’m running is inserting incrementing integers from 1 to 1 million for two columns in a single transaction. See here for the test data.

As it currently stands, Realtime can handle ~75,000 inserts in a transaction on the instance. The main issue I’m seeing is that the Phoenix Channel transport process is not being garbage collected as it had massive memory usage, and this issue is only exacerbated with each additional connected client.

Here’s a couple of things I tried to make Realtime more performant:

  • I tried memsup and a custom alarm handler that when a process exceeded a memory threshold, then garbage collect that process. However, the smallest interval for checking memory was 1 minute. It wasn’t checking fast enough so this didn’t work.

  • I tried a process to check memory of transport processes every couple of milliseconds and then garbage collect all transport processes when some threshold was reached but this didn’t work either.

  • I finally settled on sort of a hacky/inelegant solution where I checked the memory usage of all the transport processes prior to every insert being broadcast out, and if they exceeded a threshold, then :timer.sleep(2) and Phoenix Channel :hibernate_after 1. I was able to process ~750,000 inserts in a transaction this way.

Let me know if you have any thoughts on the different strategies I tried and/or recommendations for dealing with memory bloat when broadcasting many, many messages.

Thanks!

5 Likes
2

Welcome @wenbo!

The issue here is how large binaries are refcounted by the VM. And I also think that, because you have 8 topics, you are likely encoding the data 8 times, which just generates more binaries.

Here are a couple things you could do:

  1. First of all, if you are broadcasting from the GenServer, I don’t think the channel matters unless you are using intercept, which is generally not recommended as it makes everything slower. The Phoenix docs have some disclaimer on this.

  2. Instead of checking the socket processes before sending the insert, instead make it so you send a :garbage_collect message whenever you send it a large transaction (it is up to you to detect what large is). But sending the garbage_collect message is expected in some cases.

  3. To avoid sending and encoding the same message multiple times, see if you can do broadcast_from!(..., ..., {:binary, encoded_json}) and if that helps

Btw, Real-Time Phoenix may also have some tips. I haven’t read it but the author is on the forum (/cc @sb8244) so he might have some insights too!

9 Likes