I am facing a problem where I need to spawn multiple processes, and I have to keep a counter about every process. The counter needs to be indepent and accesing it should not interfiere with the process performance.
Saying this, I am exploring the options of Agent vs Registry.
If I use the Agent module, I would register an Agent for each process :via a Registry.
But how should this be different from only using a Registry, with the register function and the counter as a value.
Isn’t a registry the same as an Agent in that case?
Later today I’ll benchmark it, but wanted to know what do you think. I’ll post my results below.
I think the answer to your problem is another process which is only responsible for the counter.
The process doing the work notifies its counter process when the count is updated
Other processes call the counter process to obtain the current state.
Alternately it may make more sense for the counter process to be subscription based - otherwise you are building an “ask architecture” rather than a “tell architecture” (Tell, Don’t Ask).
Personally Agents make me queasy because anybody can mess with their internal state which has the smell of “global mutability”.
Registry pubsub may be an interesting option - though I haven’t looked at it in detail (if I’m reading the source correctly, dispatch/4 will run in the calling process rather than queuing the function to be executed later within the registry’s process, so I would classify this approach as “interfering with process performance”).
I’m not exactly sure what you mean here. Do you mean that you need to keep a total count of your workers? Or that each worker needs to keep an independent counter of something?
If you simply need to keep track of your worker totals, you might consider using a custom Supervisor or DynamicSupervisor. You’d use the Supervisor to spawn and watch your workers for failures. Supervisors also keep a count of their children, so you’d get a global worker count for free, without impacting the work being done within the workers.
Supervisors - spawn and monitor children for failures, restarting them if necessary
Agents - store and mutate local state
Registries - keep track of a mapping from a name → pid
Both Registries and Supervisors support a count. Supervisors will help you keep your workers alive. Registries will help you find and communicate with multiple workers. However, if a worker dies, it’ll be removed from the registry without any attempts at restarting.
I’m not exactly sure what you mean here. Do you mean that you need to keep a total count of your workers? Or that each worker needs to keep an independent counter of something?
No, I mean the spawned process will do something, but only accepts more work if it is doing less than a number a things. Like a queue, if the queue of the process work exceeds certain amount, it should reject the request. And I need to keep count of the length of the queue.
The mailbox queue length is my counter.
So my options are:
Use an agent with a registry
Use only the registry
And now that I’m writing this, I could use Process.info(pid, :message_queue_len) before sending a message with work to the process. But then I’m not really sure how to test the genserver to see if it accepts more message after a limit. If I send unexpected messages, the test logs:
received unexpected message in handle_info/2: which is not nice
I saw dispatch/4 but as your edits says, it may interfere with the process performance. Thank you for the link on Tell, Don’t Ask, was a great reading. With that I may refactor a little bit of what I’ve done.
The actual work is being done by a short-lived process (perhaps Task processes) that send the result to the parent process when they are done.
The parent process will aways accept a new request but may queue a request if the maximum number of child processes is currently running. The parent process can block the requesting process while this is going on.
As child processes complete requests are pulled off the parent’s internal queue to launch fresh child processes.
GenStage.ConsumerSupervisor does something similar but you may need a solution adapted to your particular requirements. Also have a look at Task.Supervisor just so you know what is available.
I would also recommend looking into GenStage for this. You don’t have a tell/ask problem. You want to limit the rate at which you get told something to the amount of capacity you have to process those tells.
This is what GenStage does. You advertise that you’re able to handle some number of events, and then you eventually get at most that number. Once you decide you can handle more events, you advertise that you’re able to handle more. This is all done more or less transparently for you by GenStage.
As you’ve already discovered, architecting a solution to your problem with Process.info doesn’t work well.
Consider using GenStage to solve your rate problem. Then within GenStage, delegate to a module that is focused on data processing. This way you’ve separated your business problem from your rate problem. And you can test your business module easily.
Agents and Registries aren’t required here. Although you’re free to register your GenStage processes with a Registry and/or hold state within an Agent. Up to you, they are different tools for different problems.
Edit: You may also want to look at Flow. Flow helps to provide a higher level data flow/processing abstraction, so you don’t need to worry about implementing a GenStage. If your workers are mapping or reducing something, then Flow could be a great fit.
As I undertand it. The producer would hold events until the consumer asks for them. But I would need the producer to reject any more events and return an error to the client. This is why I don’t see how GenStage can help me.
You are already making a big assumption that you will need processes. Could you please tell us about the problem you want to solve in general and why did you arrive to the conclusion that you need processes?
Why are you keeping count of how much work a process has done? Why not have a single process that does a single work and then dies?
I am currently applying for a job, so I believe it would not be fair to start a discussion about the whole problem. but it’s great that you gave a step back to see the bigger picture.
I can link the system after the interview process is done to see if the discussion is of interest for you guys.
I ended up choosing ets, gproc didn’t fit my use case, but was really interesting going further in the documentation. Thanks.
The handle_demand callback will let you track available processing capacity within the system. You can store a running total in your state.
Implement a publish(task) client api on your publisher that sends a call to your GenStage.
Implement a handle_call GenStage callback. If you have demand, send the task on for processing and reply to your caller. if not, return an error back to your caller.
