It may well indicate that your design is not well formed.

Do you really need to make this a genserver call? Why not just call a private function within the genserver directly? It may be that the design confusion is simply that you’re already in an async operation and there is no need to make an async call from within it.

But assuming that this second call is time consuming and you need this genserver to be done and ready to handle something else, that wouldn’t work. So, do you actually need the result of the call? For what purpose–just to return it or to do further processing? If it’s just to return it, then pass the from arg through to another genserver and return :noreply, letting the second genserver “return” the value via reply

I have had this kind of problem a few times. In most cases, upon reflecting more, it was a symptom of some logic that needed to be extracted. Sometimes I ended up extracting the shared logic into a private function used by several handle_call clauses, and some other times I even ended up extracting it in its own module. The second option can also result in easier testing, and can be called from a Task (or other construct) if necessary, separating functional logic from runtime concerns.

The difficulty that I’m facing here is that the functionality that I’m writing can not just be a simple private function because it needs to be async (explained further below). Since it has to be async, that means that I would need to keep track of which async operation is currently waiting for output from the erlexec program, which would make the logic difficult to follow and probably error prone.

At the GenServer level all the “commands” sent to the external erlexec process have to be async because the GenServer needs to wait for the handle_info({:stdout, _, _}, state) clause to be invoked by erlexec.

I agree that extracting this logic would be useful, I’m just not sure how to go about it without creating a new process that is built on top of my current GenServer (which seems like it would unnecessarily complicate the design).

Here’s two scenarios:

Screenshot 2020-07-01 11.25.15675×771 30.2 KB

Screenshot 2020-07-01 11.19.37667×681 56.4 KB

The “Easier Flow” is relatively easy since it allows building on top of the “run async command” logic of the GenServer, however the downside is that the GenServer state is not available when running the “sub-command”.

What I would really like to do is implement the “Ideal Flow”. The hard part to me is that inside a GenServer handle_call callback I want to run the “run this command” logic synchronously even though the logic actually needs to be async. Also I should make a quick note that the existing “run this command” handle_call callback stores the from in the GenServer state and returns with {:noreply, state} and then later uses the from with GenServer.reply once all the results have been received. When calling the “run this command” logic from the Caller I get the syncrhonicity for “free” because the call is wrapped in a GenServer.call, but within a GenServer I cannot use that same mechanism to make the logic appear synchronous.

Another way of stating this is that within the GenServer I want to have a run_erlexec_command function that will synchronously call erlexec and wait for the result (potentially waiting for multiple results by using a short debounced timeout) and then return. I could introduce another GenServer to get this, but as mentioned in the Original Post that feels like it would make the system more complex which I am hoping to avoid.

is there a reason why the middle thing isn’t just Task?

OK, you’re using :noreply and return already to suspend the caller until you have results. That’s probably key.

Why does the call to erlexec need to be synchronous? Why not in handle_info determine when data is complete and should be returned? (Including, possibly, multiple passes of sending to erlexec again? Essentially, a state machine which issues subcommands, receives results asynchronously until done.

The middle process needs to be long-lived because the erlexec process is long-lived and sends output asynchronously to the middle process. And generally you wouldn’t want to have a long-lived Task process.

hm. Are you recycling the erlexec “conn”, and trying to use the genserver as a “lock” to prevent multiprocess contention on the program? Or does your erlexec’d program support multiplexing?

The call to erlexec doesn’t need to be syncrhonous. I mainly want it to appear synchronous to make the logic flow easier to follow. The handle_info is already determining when the data is complete and then calling GenServer.reply if from (in the GenServer state) is non-nil. So the handle_info could definitely be made smarter to determine if the process that is waiting for the response was an external Caller or if it was the GenServer itself. But if there’s multiple places that the GenServer is running “sub-commands” then it might be necessary to have code for each of those potential “sub-commands” inside the handle_info and I’m worried that the code to manage that will start to become even more complex. A state machine might be able to help manage the complexity but I’m not quite sure how I’d apply a state machine to this problem.

I’m not sure what you mean by that. What do you mean be “conn” in this case?

I am using the GenServer as a lock/synchronization point (although there is only one process that currently talks to it directly). Although there are actually multiple independent instances of the erlexec’d program (and the associated GenServer) running concurrently, but they don’t communicate or interfere with each other in any way.

I know this isn’t exactly erlexec, but for example, I have long-running outbound SSH connections to machines, and I can pass the ssh connection between processes that need to use it (actually in my case two gen_statems hold onto the ssh connection pid, and in my case both of these statems pass the connection to Tasks to achieve their goals).

So I guess, my suggested model is, you have a caller, you could presumably have the caller “check out” an erlexec conn, and pass that conn to Tasks that perform the command/sub-command jobs as necessary. I would say that Task + FSM data structure is appropriate if the job you are doing has fixed scope and transient lifetime; gen_statem/gen_server is appropriate if the lifetime of the FSM is undefined, or if the FSM needs to update its state by being preemptively “pushed” from the outside-of-beam system.