ValtteriL

ValtteriL

How to make Task.async_stream return results as they are ready?

How to make Task.async_stream return results as they are ready?

I have the following code:

defmodule Demo do
  def run do
    child =
      spawn(fn ->
        Stream.resource(
          fn -> :ok end,
          fn _ ->
            receive do
              {:msg, msg} ->
                IO.puts("# received #{inspect(msg)}")
                {[msg], nil}

              _ ->
                {[], nil}
            end
          end,
          fn _ -> :ok end
        )
        |> Stream.each(fn x ->
          IO.puts("## before #{inspect(x)}")
          x
        end)
        |> Task.async_stream(
          fn x ->
            IO.puts("### during #{inspect(x)}")
            x
          end,
          ordered: false,
          max_concurrency: 2
        )
        |> Stream.each(fn x ->
          IO.puts("#### after #{inspect(x)}")
          x
        end)
        |> Stream.run()
      end)

    1..3
    |> Enum.each(fn x -> send(child, {:msg, x}) end)
  end
end

When run, I get the following output:

iex(6)> Demo.run
# received 1
## before 1
# received 2
### during 1
## before 2
#### after {:ok, 1}
### during 2
# received 3
## before 3
### during 3
#### after {:ok, 2}

The Stream.each after the Task.async_stream is never called with the last message. Async_stream seems to withhold output until it has max_concurrency results ready.

Is there any way to make to avoid the buffering?

Background: I’m building a data processing pipeline that processes endless stream of data and cleanup after each task has to happen immediately after its done. I could do it in the task itself, but that would break encapsulation and force me into complicating things with process messaging. I previously used GenStage, which worked well at small scale, but could use performance improvement as no backpressure is needed.

Most Liked

al2o3cr

al2o3cr

Try passing ordered: false to async_stream; the docs specifically mention “This is also useful when you’re using the tasks only for the side effects”.

Last Post!

dimitarvp

dimitarvp

Replace the Stream.each + Task.async_stream with Enum.each + Task.async, collecting the handles to all tasks and then just Task.await_many them (note that this succeeds and does not block even if the tasks are already completed).

That, or @al2o3cr’s approach.

I don’t get that. Sounds like you are complicating things by not doing it in the task itself from where I am standing.

Where Next?

Popular in Questions Top

vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New

Other popular topics Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36654 110
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 44532 311
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement