So I have a scenario where a given request will require about 5-6 different stages to happen, which are returned to the user as they are completed. But there’s a catch that the two tasks that constrain the total time of the request, are sequential. Here’s an example.
- REQ: Image comes in.
-
- Initiate ML model that does conversion of image
-
- Various smaller tasks that provide some analysis of the initial photo that are sent back to the user and displayed as they wait
-
- Initial conversion done. Start reverse conversion task(input is the result of conversion 1)
-
- Send initial converted image to user.
-
- Again various tasks of analysis of conversion 1
-
- Second conversion is done. Send to user
-
- Analysis of conversion 2, send to user
Currently the process that orchestrates all this starts an async task for the ML, but there are scenarios where due to the stuff in step 2, the start of the second conversion happens some amount of time after the completion of the first conversion.
I have tried changing it to a setup where the task that does the first conversion, also starts a task for the second conversion, and returns the results from the first, and the task for the second. That way conversion 2 starts at the earliest opportunity, and is not constrained by the main orchestrating task to always have completed those subtasks already.
While this seemed like a good idea in theory, the issue is the main process can’t await the task that was started by initial conversion Task.async call.
** (ArgumentError)
task %Task{mfa: {:erlang, :apply, 2}, owner: #PID<0.1410.0>, pid: #PID<0.1423.0>, ref: #Reference<0.497699279.469565443.66976>}
must be queried from the owner but was queried from #PID<0.1407.0>
I could setup a GenServer that is spun up to manage starting and awaiting the completion of both conversions, but that the main process could still query when it’s ready for the first conversion’s completion, but this sounds like it increases the complexity quite a bit from the current Task.async implementation.
I’m curious if there’s a way for my first idea to work, maybe sort of like the first Task.async that is starting the second task as if on the behalf of the main task. I’ve looked that the “$ancestors” and “$callers” and can see the difference there, but get the feeling that any manipulation of those won’t actually solve my issue, as that task parent PID is probably set at creation.
Any input? Is this possible with Task, or is a custom GenServer implementation the only route forward?
FYI I’m currently using Task.Supervisor.async
, rather than Task.async
if that makes a difference.