I’m a contributor to Realtime, which listens for Postgres changes via logical replication and broadcasts changes to subscribed clients.
As an example of how Realtime works, if I were to insert 1 million rows in a transaction, then Realtime would first save all 1 million inserts to process state, then loop through each insert (struct with data about the insert) and broadcast eight times, once for each topic.
I’ve been trying to maximize the number of inserts in a transaction the Realtime server can handle on an AWS instance with about ~600MB of available memory. The test I’m running is inserting incrementing integers from 1 to 1 million for two columns in a single transaction. See here for the test data.
As it currently stands, Realtime can handle ~75,000 inserts in a transaction on the instance. The main issue I’m seeing is that the Phoenix Channel transport process is not being garbage collected as it had massive memory usage, and this issue is only exacerbated with each additional connected client.
Here’s a couple of things I tried to make Realtime more performant:
I tried memsup and a custom alarm handler that when a process exceeded a memory threshold, then garbage collect that process. However, the smallest interval for checking memory was 1 minute. It wasn’t checking fast enough so this didn’t work.
I tried a process to check memory of transport processes every couple of milliseconds and then garbage collect all transport processes when some threshold was reached but this didn’t work either.
I finally settled on sort of a hacky/inelegant solution where I checked the memory usage of all the transport processes prior to every insert being broadcast out, and if they exceeded a threshold, then :timer.sleep(2) and Phoenix Channel :hibernate_after 1. I was able to process ~750,000 inserts in a transaction this way.
Let me know if you have any thoughts on the different strategies I tried and/or recommendations for dealing with memory bloat when broadcasting many, many messages.