Say I have a message queue implemented as a GenServer, something like this
#... def start_link ...
def init(state) do
{:ok, schedule(state)}
end
# schedule processing every 500 ms
defp schedule(state) do
Process.send_after(self(), :process, 500)
state
end
def handle_info(:process, state) do
{:noreply, process_queue(state)}
end
# catch all handle_info
def process_queue(state) do
# concurrently processes each entry and deletes it from the state
# schedule(state)
end
def put(data) do
GenServer.call(__MODULE__, {:put, entry})
end
def handle_call({:put, entry}, _from, state) do
{:reply, :ok, [entry | state]}
end
The question is about the relationship between handle_call and handle_info that both use the same state. From what I understand there is no way the state breaks, meaning while process_queue does its job (which may take a while) any put calling handle_call will have to wait, is that right?
If I want put to return immediately and add entries to queue after handle_info is done, Iād have to switch from call to cast, and Iām good?
Is there a nicer / simpler approach to achieve that or is this one good? Shouldnāt I handle such state using ETS (thatās what Iām switching from at the moment)?
Not necessarily. Using ETS makes sense if you have a lot of data that you might want to search through. If you have a queue of ājobsā (of whatever kind), then simply using a single process whose state is a list, map or queue should do the job just fine, without adding complexity of using a system ETS for the developer.
Depends on what you want to archieve exactly: If you want the items to be processed and they can be processed completely separately, then you could use Task.async or Task.async_stream to do the job, which means that process_queue will not block for long.
On the other hand, if it was possible to process items completely separately from the start, one could create Tasks right away, and not need to bother to use a queue at all.
Some background: I am using parts of a working app in another, bigger app so some solutions have to be refactored or reimplemented.
I have lots of data, but itās pretty uniform and I have to handle each entry all the time, no searching. Complexity of ETS wouldnāt be a problem though, because the current implementation already uses it, still makes sense to stick with GenServerās state?
I will use Task.async_stream here, there is no question about that The jobs are separate and each one might have to work with an external API so concurrent processing is a must, but still even asynchronously it will take some number of milliseconds to finish.
It was a āno queueā implementation originally I donāt want to bore you with too much details but here is one case that justified the use of queue: since I have to use an external API it might have temporal problems or even one time errors, so at times I need to fallback to a simpler handler for a specific task and at other times - just keep an item in queue and schedule it for the next run since the error must be temporary but I donāt want to block for long.
Anyway thanks for the feedback, it seems there are no obvious problems with the above solution and I might use it.
I might also stick with mutable ETS state, the only problem with it is that itās somewhat verbose to iterate over. Which one would you go with, @Qqwy ?
Just remember that sending messages is already async, so if your 2 streams are going to different processes then you can just call them anyway, saves spawning more processes, which will slow it down slightly.
For the map version earlier on, if the actions you perform are already communicating to another process then you are already async, so you definitely should not use Task then as that would just slow it all down.