PJUllrich

PJUllrich

Author of Building Table Views with Phoenix LiveView

How to spawn many, many processes really fast?

I’m currently playing around with GenServers and for a totally unrealistic, but still interesting problem, I wondered whether there is a faster way of spawing 40.000+ processes than this:

pids = for id <- 1..40_000 do
  {:ok, pid} = GenServer.start_link(MyGenServerModule, id: id)
  pid
end

Since this for-loop runs synchronously, it takes around 10 to 12 seconds on my macBook to spawn all 40k processes. I wondered, whether this could somehow be achieved asynchronously?

Marked As Solved

gregvaughn

gregvaughn

Well, there’s your bottleneck. The LiveView has to handle 50K messages.

Also Liked

PJUllrich

PJUllrich

Author of Building Table Views with Phoenix LiveView

Thank you! That indeed decreased the execution time from ~12s to ~1.2s, but unfortunately, I believe the processes weren’t linked to the original process correctly. Maybe because the GenServer.start_link/2 was called inside the Task.async_stream? Will those processes then correctly be linked to the “parent process” in which I used to call the GenServer.start_link/2 function originally?

PS: I’m currently playing around with simulating the Game of Life with 50.000 erlang processes. Because, why not? :smiley:

10
Post #4
rvirding

rvirding

Creator of Erlang

Just to be a bit picky here, starting a GenServer does more than just spawn a process. So the GenServer.start_link will spawn the process and then sit and wait until the new process does the necessary initialisation to become an OTP process and then calls the behaviour’s init callback. When that has completed the GenServer.start_link will return {:ok,pid}. This is a synchronous operation which is well defined in how a GenServer behaviour works. Can you get around it? NO, all you can do is try and make the init callback as fast as possible. Check the documentation for tips for doing that.

If you want to check how fast you can spawn processes just do a spawn instead, of course then you won’t get a GenServer. :smile:

lud

lud

There is a feature to do just that:

  @impl true
  def init(state) do
    {:ok, state, {:continue, :after_start}}
  end

  @impl true
  def handle_continue(:after_start, state) do
    # start living
  end

Last Post!

Sebb

Sebb

I was just wondering. I have something similar and tried to find a better solution than trial and error - but didn’t.

Where Next?

Popular in Questions Top

New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

Other popular topics Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36820 110
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement