Worker with state vs Cache + Worker

No, please, a thousand times no. Do not use :sys.get_state in production code. It is intended only for debugging purposes. You speak a lot about managing state carefully, but then you want to do this brutal approach.

My suggestion wasn’t so much about Task.Supervisor as it was about async_stream in which case the results come to you when they complete. No need to reach into another process’ state (which would be akin to me to grabbing money out of your wallet because you owe me).

And gen_statem seems to be overkill for your purposes too. The point of a “gen” style server is to be able to receive messages from other processes. You can use a basic state machine approach with Enum.reduce and an appropriate accumulator map/struct.

I would not depend on Agents for a production use-case, particularly compared against a purpose-built GenServer or gen_statem module. The latter can have fully custom/arbitrary lifecycle behavior, error handling, proper supervision, etc.

For OP’s actual question, can you say more about the nature and use of the data? What’s its volatility, its source of truth, how expensive is it to rebuild from nothing, are concurrent writes necessary, what’s the appetite for eventual consistency, how dangerous is it for multiple BEAM nodes to have disjoint views of the information, etc.

For what I can say, so a BEAM node will receive a request and must do concurrent jobs to build a context for a further computation. The node need to keep the result of each job.
Some job can be really fast to rebuild and some are really expensive (such as multiple network calls).
The concurrent writting does not really matter, because job will produce independant data.

The other question which interest me also, it is better to keep multiple process with small or medium data or only one process with a lot of data ? (evenif ETS can be used to leverage heap allocation)

My current approach is:

1. Top Supervisor including: Registry and DynamicSupervisor.
2. The dynamic supervisor will spawn for each specific request its own supervisor.
3. This latter will spawn jobs (currently :gen_statem to simply state identification and processing).
4. Each jobs will be registered into the Registry and keep the job result.

When I want to retrieve the all the job data, I’m using Registry.dispatch to broadcast the retrieval of the state and data from the jobs.

Another approach specially regarding 3. and 4. will be to insert a cache inside the latest supervisor and each job as Task where each will fill the cache and die after. (freeing maybe some memory)
The retrieval of the data will be directly from this cache. (but cache memory will increase)