Running `Task.start/1` in an elixir script (.exs)

Questions / Help
1

I’ve got a module that handles a long-running web request. I need to make a handful of them, so I’m using Task.start/1 to invoke the function in question in parallel, allowing the requests to resolve each in their own time.

some_module.ex

defmodule Some.Module do
  
  def do_something do
    # logic from some args
    Task.start fn -> long_running_req() end
  end

  defp long_running_req do
    # Some request logic that takes a while that I want to do in parallel
  end

This module is invoked via a .exs script. Unfortunately, when the script finishes running, the code within the Task no longer runs. I think this makes sense to me based on my understanding of Supevisors and processes and the like, but what’s the way around this in the case of my .exs runner?

a_module_runner.exs

[some, args, I, need] = System.argv
Some.Module.do_something(some, args)

Does the .exs file need to spin up a supervisor of its own to wait for all the child processes? Is .exs just not a viable solution because of mix run?

2

Can you elaborate on the decision to use a task? You could do:

  def do_something do
    # logic from some args
    Task.async fn -> long_running_req() end
  end

[some, args, I, need] = System.argv
task = Some.Module.do_something(some, args)
Task.await(task, :infinity)

But then in that case you might as well forget the task entirely.

1 Like
3

There’s also Task.async_stream/3 and Task.async_stream/5 - they could be handed an Enumerable holding functions if needed.

However given that a stream is lazy you’ll need something like Enum.to_list/1 to “pull the values through”.

2 Likes
4

I realized I misrepresented how this runs, my apologies: there will be some n of do_somethings running at any given time.

There is a separate enumerator that’s actually rolling through an array of responses and applying do_something to each response. A better representation is

[some, args, I, need] = System.argv
Other.Module.enumerates([array of data], &Some.Module.do_something/2(some, args))

It looks like @peerreynders suggestion of Task.async_stream may be the thing to look at with that in mind.

1 Like