you are i can yse GenServer.cast to send directly but when you have to sync big table then it become costly, to reduce this, we will store all genserver node pid in each ets table and then it become easy to sync group table data in another node by using Kernel.send.
in simple language, Kernel.send is cheap than GenServer.
It is fine to connect 1000 nodes to each other. We benchmarked Phoenix with 1 million websocket connections and those are much heavier than just plain TCP connections.
IMO, focusing solely on the number of connections is the wrong way to approach the problem. The big question is which data will be sent over the connection and how often. So depending on your application and its communication patterns, getting up to 1000 nodes may be very easy or it may be very hard. For example: the trouble with large Erlang clusters is not really the number of connections but using modules like global, which requires all 1000 nodes to agree. I believe this is one of the reasons why the WhatsApp folks rewrote pg to not rely on global and it has much better scalability properties now.
Honestly, I personally find it hard to predict how a system with 1000 nodes will behave, without any detailed information or any real usage. Sometimes there are obvious bottlenecks, but most often they come from unexpected places that you only learn once you get to a certain scale. I’d prefer to first scale the system to 20 nodes and benchmark. Then 100 nodes and benchmark. The 500 nodes and then 1000. After all, you need real data, real use, and real traffic to make sure a system with 1000 nodes is behaving as expected.
The only thing missing from this thread is someone saying “But Erlang was invented 100 years ago when it ran inside the same rack.”
I say give it a shot, see how it goes.
For mnesia there is GitHub - emqx/mria: Asynchronously replicated Mnesia-like database for Erlang/Elixir · GitHub, maybe that could scale better.
Also maybe of interest - GitHub - ostinelli/syn: A scalable global Process Registry and Process Group manager for Erlang and Elixir. · GitHub
Discord is using Elixir, they have some great blog articles on how they are scaling things.