Implementing a worker for an infinite job processing

Questions / Help
1

Suppose there is some infinite job, like queue processing or computation.
I want to run a worker (or multiple workers) that will do the job but also respond to some control commands.
gen_server doesn’t seem to be a good choice here because a worker would have it’s own loop for job processing that will interfere with gen_server’s loop.

I thought about spawning a process that would do something like this:

def work(state) do
   # do some work here
  case process_messages(state) do
    {:ok, state} -> work(state)
    :stop -> :stop
  end
end
   
def process_messages(state) do
  receive do
    {:stop, _pid} ->
      :stop
   ...
  end
end

But it seems like I’m reinventing some kind of a gen_server and it feels wrong.
What is a good way to do this?

Thanks!

2

Instead of requeueing by calling work/1 in a genserver you can send yourself a message to trigger further work. Messages are handled in order, so if a stop message came in while working it’ll be handled before any new message you send yourself.

2 Likes
3

yes, indeed, thanks!

4

You want the process doing the actual work to be short lived.

  1. It will be responsible only for that one task. Easier to code, reason and test.
  2. When it is completed and terminates resources (e.g. memory) are automatically reclaimed. This is especially important for binary resources.
  3. This worker process can be a plain old process (POP), special process (see proc_lib) or GenServer.

There would be a manager which holds work-to-be-started.

  1. The manager can spawn workers unbounded or limit the number of workers in the system. Backpressure or rate limiting.
  2. The manager can reorder work-to-be-started if needed. Priority tasks.
  3. The manager can restart a work-to-be-started if a worker dies unexpectedly. Retries and resillency.
  4. The manager can respond to control commands and act upon it. Command and control.
5

Thanks for the suggestion!
The manager process would have to loop in the spawn workers routine but still be responsive somehow, and that’s the same problem the question is about :slight_smile:

6

Once you have this separation you can itemise the Command and Control and determine what the workers actually need to respond to. Most of the time it is just a terminate signal (i.e. use exit/1), or some state query (i.e. use sys) which doesn’t require looping into the spawn workers routine.