Scalable architecture for the pool communicating with external processes

Hello forum,

This question is following the footsteps of the “Back on your feet” talk by @cloud8421 https://www.youtube.com/watch?v=kWYgrA2YshE talk.

Context:

  • there is an application that can report a status of the transaction and one application can report status for one particular account, i.e. it’s required to spawn 1k applications to monitor 1k accounts. Accounts are created and destroyed dynamically, on scale > 1 million a week.

Problem:

  • what is the most scalable and reliable OTP architecture to monitor > 1 million transactions simultaneously

The first and the simplest option is to run a process per account, simple and easy.

Second is to create a pool of processes each of which will be in control of N accounts/processes and circularly poll them (RR).

       +---->ScannerWorker +---> N account RPC processes
       |
Scanner+---->ScannerWorker +---> N account RPC processes
       |
       +---->ScannerWorker +---> N account RPC processes

Now there a couple problems with restoring failed d ScannerWorker

  • If I will use phash2 to distribute accounts between workers I can’t use ScannerWorker PID because it will be impossible to address requests to its accounts because of the new PID after restart. I need to have a stack of IDs and accounts attached to each ID and have newly created ScannerWorker pickup that first ID in the stack and spawn processes for the associated accounts.

Does that makes sense? Where and how that can possibly go wrong? What could I miss?

Hi,

One million processes active at the same time may not be an issue if they’re mostly idle. You would need to monitor and profile your application to make sure you have enough memory and can handle any I/O generated by those processes.

In case one node is not enough to handle 1M processes, you can, with the same pattern, distribute them on more nodes and use consistent hashing (e.g. phash as you already mentioned) to know on which node they are.

In terms of resolving a name to a process, you can use a registry for it. The built-in Registry shipped with Elixir can hold 1M entries as long as you partition it (docs have examples for that). It’s a local registry, so it’s valid only for one node.

Note that on a multi-node scenario, you can combine consistent hashing and registry: given a name, you get to a node, where you can use the local registry.

The key thing is measuring, you can’t get to a satisfactory answer without trying an approach and seeing how that affected the system.

Hope this helps!

1 Like

Thanks @cloud8421,

the answer backed by experience is most appreciated. Lately I figured out that I really have to measure an account application memory footprint to get my limitations right - some of those can be running potentially forever and thus I may need reconsider the approach and target amount of used memory by each shell process instead of amount of processes in Erlang VM.

I will google registry partitioning :slight_smile: