Worker with state vs Cache + Worker

I would say when to use gen_statem:

1. you’re modeling an external “real thing” that you have limited control over, or stateful communications protocol.
2. there are recurrent events that must be asynchronous (eg you have to check on your external thing and you don’t know when the response will come back)
3. your model is not transient (if your process crashes you want to recover, and you can’t just throw it away and start from scratch)

If all do not apply, do not use gen_statem.

Thank you very much @gregvaughn .
Nice way to do it

But I my use case, I don’t known if async_stream is the right thing to do, because I need to retrieve the result asynchronously later not after the job completion but when I will receive to check and compare the jobs results.

I’m sorry I don’t have some example to share, but I view it as some basics of functional programming. Here’s some pseudo code (caution: I have not executed it) to think about

Enum.reduce(data_list, (:initial_state, nil), &transition/2)

def transition(data_element, {:initial_state, accumulated_data}) do
  #do something and return {new_state, new_accumulated_data}
end

def transition(data_element, {:state2, accumulated_data}) do
  #do something else and return {new_state, new_accumulated_data}
end

# ... plus as many more transition/2 function clauses you need

Now the result of the Enum.reduce is {final_state, final_accumulated_data} and if you use the async_stream approach I suggested, it will be sent to the calling process automatically. There is no need to store this (in process state or ETS) and later retrieve with some :get_job_result message. That is the whole advantage of the async_stream approach.

Thanks @gregvaughn.

Do you have any example about your idea using state machine with simple Enum.reduce ?
because at the end , I need also to keep state of the data, even if the only received message will be “get_job_result” . So wondering if a simple tasks + ETS will be sufficient.

No, please, a thousand times no. Do not use :sys.get_state in production code. It is intended only for debugging purposes. You speak a lot about managing state carefully, but then you want to do this brutal approach.

My suggestion wasn’t so much about Task.Supervisor as it was about async_stream in which case the results come to you when they complete. No need to reach into another process’ state (which would be akin to me to grabbing money out of your wallet because you owe me).

And gen_statem seems to be overkill for your purposes too. The point of a “gen” style server is to be able to receive messages from other processes. You can use a basic state machine approach with Enum.reduce and an appropriate accumulator map/struct.

Thanks @dimitarvp, I was thinking the same, just I didn’t know if it’s impact more the memory to keep multiple process with state vs only process with more big state.

Good point @gregvaughn, I didn’t think about Task.Supervisor because I didn’t want to use Task because I need to keep state somewhere and I do not want to retrieve the data when they are finished but when I will receive another request to check this state.

And for the :gen_statem approach, jobs will not received really messages only a way simplify the state identification and transitions. Only at the end I will request process state and date for example using :sys.get_state. But maybe there is a better way to do.

IMO you should have many processes with smaller data. Copying from ETS will be easier on the GC (after the process dies) and will also be faster.

The calculation results that are hard to recompute should go into a database. Everything else is fine in ETS.

This sounds like a job for TaskSupervisor.async_stream to me. I use that a lot to “fan out” jobs into concurrent pieces, but still collect their responses. Do review all the options for that function though. I use ordered: false quite often.

However, if you really need :gen_statem to manage incoming messages to each of your jobs, then this approach is too simple.

Thanks @shanesveller, for explaining why you wouldn’t use Elixir Agents in this situation.