That dashboard is super neat!
Nice to know that I’m not the only one who is running into these sorts of issues. I’m doing adjustments to config this week but I’m also wondering if maybe using Partisan with libcluster would be the way to go? It doesn’t seem like a lot of people have done this though and I don’t think I have the know-how to do this in elixir yet. Though libcluster does mention that you can use Partisan with it.
Partisan won’t solve your problem. Your problem is caused by disconnects or latency high enough to be considered a disconnect. This problem can’t be solved by a library or any tweak in erts. That means that this problem is architectural. So the real question is what you use this distribution for, what consistency model you want to have.
For example, global splits cluster when full-mesh topology is broken. But global is used in CP scenario, where we just leave cluster unavailable when partition is broken. Do you need global, actually? What do you use it for?
Only considering Partisan because of the following on the readme:
Also, Erlang conflates control plane messages with application messages on the same TCP/IP connection and uses a single TCP/IP connection between two nodes, making it liable to the Head-of-line blocking issue. This also leads to congestion and contention that further affects latency.
This model might scale well for datacenter deployments, where low latency can be assumed, but not for geo-distributed deployments, where latency is non-uniform and can show wide variance.
It was just a thought, I’m not considering that as an experiment until I’ve run dry on config changes.
I need global because its a Phoenix channel application using Phoenix.PubSub so it’s uses distributed Elixir, users need to be able to see events from any other user no matter what server they are connected to
Neither phoenix, nor phoenix_pubsub is using global. If you’re using global, what are you using it for? can you provide a snippet please?
I don’t use it in my application.
I can only assume it’s being used by libcluster.
Edit: sorry for the terseness, I was in a rush to reply. I’m pretty sure this is part of libcluster.
It is not used in libcluster. I’ve grepped libcluster, phoenix and phoenix_pubsub and it is not used in any of them. Perhaps it might worth to grep your deps folder to find calls to global
We’ve considered Partisan and I think it would help, but if one TCP connection is bad I’m assuming they all are usually.
We don’t use global. We do a lot of RPC calls for things and have to expect them to timeout not because the function call is heavy but because the network stalls out.
The PG2 PubSub adapter just sends stuff and I’ve thought about using my network pings to basically route around bad connections with a send_via which would proxy messages to a destination node through another node where I have a good connection.
Not sure if this is a good idea.
Or use some other server as a PubSub adapter which is built inherently for global distribution. I’ve naively thought about Nats. But also Bondy which was built by the guy who maintains Partisan.
He gracefully had a long discussion with me and is actively working on it.
Maybe he’s here @aramallo!
I am @chasers ! 
I think the way Partisan could help in this situation is exactly what @chasers suggests. However, this can only be accomplished when using the HyParView topology which maintains a partial mesh (eventually and probabilistically) forming a fully connected graph and uses transitive message transmission.
Using HyParView rules out the possibility of using partisan’s OTP behaviours at the moment (working on trying to solve this next, the key issue is how to implement monitoring).
Also notice that using Partisan rules out using Phoenix as it relies on disterl and OTP. For Phoenix to work we would need to fork it and teach it how to use Partisan and Partisan’s OTP behaviours.
I have plans to test Partisan itself and Bondy (which uses Partisan) in Fly.io for this very same reason.
If you’re talking about pubsub then forking is not needed. Pubsub already supports adapters to use different backends. E.g. there’s a redis based backend for heroku users.