How to spawn many, many processes really fast?

You want just to spawn them or get their PIDs as well? Should these processes be inside supervision tree?

I need their I PIDs to send messages to them. A supervisor is not needed. I want to spawn them inside a LiveView process and have them linked to that LV process.

1..40_000 |> Task.async_stream(fn _ → YOUR_CODE end) |> Enum.to_list

Not sure what evil thing you are trying to do… I won’t ask

Thank you! That indeed decreased the execution time from ~12s to ~1.2s, but unfortunately, I believe the processes weren’t linked to the original process correctly. Maybe because the GenServer.start_link/2 was called inside the Task.async_stream? Will those processes then correctly be linked to the “parent process” in which I used to call the GenServer.start_link/2 function originally?

PS: I’m currently playing around with simulating the Game of Life with 50.000 erlang processes. Because, why not?

Screenshot 2021-09-04 at 14.41.47480×481 136 KB

Hi, That’s pretty cool.

To link the process to your main process, you can do this:
1..40_000 |> Task.async_stream(fn _ -> YOUR_CODE end) |> Enum.map(fn {:ok, pid} -> Process.link(pid) end)

Unfortunately, that took even longer than the sequential loop. Sequential was 12s and this version was something between 60 and 85s I decided on using the sequential version now, but thank you anyway

how about starting the processes in some DynamicSupervisors?

That’s weird. On my machine they hardly differ.

Spawning processes, even GenServer processes, is fast in Elixir. I just wrote this script based on your code to test:

defmodule MyGenServerModule do
  use GenServer

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_call(:ping, _from, state) do
    {:reply, {:pong, state[:id]}, state}
  end
end

{t, pids} =
  :timer.tc(fn ->
    for id <- 1..40_000 do
      {:ok, pid} = GenServer.start_link(MyGenServerModule, id: id)
      pid
    end
  end)

IO.inspect("started #{length(pids)} processes in #{div(t, 1000)} ms")

Enum.random(pids) |> GenServer.call(:ping) |> IO.inspect()

The output I got:

"started 40000 processes in 521 ms"
{:pong, 2861}

So maybe you’re doing too much work in the init/1 callback? If true, you can move the hard work into a handle_info/2 callback, by sending a message to self in the init callback. For example:

def init(state) do
  send(self(), :start)
  {:ok, state}
end

def handle_info(:start, state) do
  # start living
end

Edit: :conitune tuple mentioned by @lud is better.

I played with that problem and had 4 processes spawn 10 processes spawn 10 processes spawn 10 processes, etc, to spawn 40K processes.

It is fast, but not that much faster than spawning them sequentially. And if you do that then I don’t know how you could supervise those processes.

For better synchronization I would pick a timestamp 3 seconds in the future, spawn all processes sequentially and have them wait that precise timestamp to start the logic in sync.

Here is my test code:

defmodule Serv do
  use GenServer

  def start_nolink(id, parent) do
    GenServer.start(__MODULE__, {id, parent})
  end

  def init({id, parent}) do
    Process.link(parent)
    send(parent, {:started, id})
    {:ok, id}
  end
end

defmodule Spawner do
  def rec_spawn(parent, scheme) do
    rec_spawn(parent, scheme, 0)
  end

  defp rec_spawn(parent, [], sum) do
    Serv.start_nolink(sum, parent)
  end

  defp rec_spawn(parent, [range | ranges], sum) do
    sum = sum * 10
    starter = fn -> Enum.map(range, &rec_spawn(parent, ranges, sum + &1)) end
    spawn(starter)
  end
end

defmodule Control do
  def check_started(max) do
    _check_started(max + 1)
  end

  defp _check_started(max) do
    case Process.info(self(), :message_queue_len) do
      {:message_queue_len, ^max} ->
        IO.puts("all started OK")

        flush_all()

      {:message_queue_len, n} when n < max ->
        IO.puts("started #{n}/#{max}")
        Process.sleep(100)
        _check_started(max)
    end
  end

  defp flush_all() do
    receive do
      {:started, _} ->
        flush_all()
    after
      0 -> :ok
    end
  end

  def sum_ranges(ranges) do
    {sum, _} =
      List.foldr(ranges, {0, 1}, fn range, {sum, size} ->
        {sum + Enum.max(range) * size, size * 10}
      end)

    sum
  end
end

ranges = [0..3, 0..9, 0..9, 0..9, 0..9]

parent = self()
Spawner.rec_spawn(parent, ranges)

ranges
|> Control.sum_ranges()
|> Control.check_started()