Manage state in Elixir

I would say that this isn’t a question of state so much as of process registration. The workers themselves sound stateless in your description. Any one of them is capable of fulfilling a request, and none of the persist or share anything between subsequent requests.

Read up on how GenServer’s allow various forms of name registration. Next, take a look at :global from Erlang and Registry from the Elixir stdlib. If you just need a constant-sized worker pool where any of them are used to handle a given single invocation, look at the :poolboy Erlang library. One of these should fit your use-case.

https://github.com/devinus/poolboy

If I understand correctly you need a process pool. You can use something like Poolboy. With Poolboy you can setup a supervisor that supervises your processes.

Why not use Registry?

I know of poolboy but I choose not to use it for this specific case, reason being, that having a pool of processes was an analogy to make the problem easier to understand.

What actually ends up happening is that I have around 10_000 gun connections open and I need to save them somewhere. Upon opening a connection (which is in reality a stream) it remains open forever, waiting for me to send requests for it to send to some domain.

This is why it makes no sense to use poolboy here. In any other case, I would agree.
Should I updated the problem and get rid of the analogy?

Do you need to broadcast the same message to all of them?

You can use a PubSub system. Take a look at :pg2.
Registry is great when each single process needs to have a name/id. In your case you just need to have access the processes all-together. :pg2 also monitors your processes so if anyone crashes, it’s automatically removed from the group.

P.S. It’s also possible to use pg2 on multiple distributed node, but be careful about this because if I remember correctly the replication over multiple nodes is made locking the pg2 processes.

I don’t need to broacast a message. I need each process to accept a given command and make a request to a domain.

My main question now is: What happens if a connection in Registry dies?

Thanks for the pubsub idea though!

If the connection process dies it’s automatically removed from the registry. Registry monitors the processes like pg2.

If the processes are from the same module, I would use :pg2, or some sort of pubsub mechanism, to group them together without reinventing the wheel. You could do it with a DynamicSupervisor where you just add the new processes to the supervisor and asks to the supervisor the processes list..

How would the process get a command if not by receiving a message? Notably if they’re inside a registry you don’t also need pg2, you can use Registry to broadcast, there are examples in the registry docs.

The same thing that happens when any other BEAM process dies: it depends on what supervisor it has. The Registry also monitors the process and will de-list a dead process. But the supervisor can restart the process and re-register the new instance with the Registry.

You need to differentiate between between Singleton and Just Create One.

• If you pass in a handle or name when the process spawns it is more like “Just Create One” because you don’t depend on any globally fixed information.

That being said having 10000 processes pounding on one, single process can still yield all the disadvantages of “being shared” (singleness) in the form of a bottleneck - so a single process isn’t necessarily a good replacement for a single ETS table (depending on the circumstances of course).

I still think that you have to reveal the complete interaction pattern before it can become clear what the best solution is.