I’ve tried to setup observer on remote node in the past but failed for many reasons. Do you have any idea what to look into specifically?
Goose97
tcoopman
It can give you more real time information and also tell you something about the other processes as well. So more of an overall view on the system to see what the cause could be
0
chrismccord
Creator of Phoenix
Also how does CPU look up to and during the backup? Any estimate of size of the total # presences? You can go CPU bound as you get large number of users being tracked thru Presence, which could be related.
0
outlog
would it be possible to use pubsub 2.0 - which uses pg instead of pg2?
The pg2 module is deprecated as of OTP 23 and scheduled for removal in OTP 24. You are advised to replace the usage of pg2 with pg. pg has a similar API, but with an implementation that is more scalable. See the documentation of pg for more information about differences. http://erlang.org/doc/man/pg2.html
if nothing just to rule it out..
1
Goose97
We are using OTP 22 so pg is not available yet. But I don’t think we hit the limit of the pg2 module in term of scaling.
1
hubertlepicki
I am not saying this is happening but you shouldn’t rule out the possibility of I/O being a bottleneck (in fact, it’s quite often a bottleneck). If these messages are being pushed to clients, there has to be an open socket connection, those connections will eventually buffer up if you are sending data to clients faster than they can handle receiving it.
I would need to check what’s the behavior and safeguards channels have if any for this situation when clients are slow and transmission rate is high, but in principle that can be your bottleneck. I am sure @chrismccord could shed some light what is happening in such situation.
1
chrismccord
Creator of Phoenix
It’s possible you are running up against the limits of your instance hardware for that number of concurrent presences. How partitioned is that 70k presences? IE do you have a few topics with 10’s of thousands of presences, or many topics with much fewer presences per topic?
0
Goose97
Update on the issue at hand. The Erlang VM schedulers are kinda busy when messages start to piled up. I checked through :erlang.statistics(:run_queue). The exact reason why this happens is still unclear though since the CPU’s utilization (and load average) stays consistently low.
0