ImNotAVirus

Decrease CPU consumption for many processes (50k+)

Hi everyone,

I am working on a project where I have to spawn many processes (50k+).
Each process represents an entity that must move on a map.

The behavior of an entity is simple:

Each entity choose a random coordinate in a radius around it.
Then it calculates the path to go there
Once arrived at destination, it pauses for a random time between 850 and 1150 ms
It starts again to choose a random coordinate
etc..

The pathfinding algorithm is written in Rust (NIF) for performance reasons

So I started to do my first tests with only 10k processes.
Unfortunately, I found that just having 10,000 processes spawn looking for a path and then waiting was already permanently using between 45-55% CPU on my Windows.
Curiously, when I did my tests on WSL, the same algorithm takes only 5 to 10% of CPU on WSL2.

This huge difference in CPU consumption is the first thing I can’t understand/explain.

I then tried to benchmark my pathfinding function using Benchee to see if that was the source of my problems.

I got theses results:

Name            ips        average  deviation         median         99th %
astar      355.56 K        2.81 μs   ±863.72%        2.20 μs       18.60 μs

According to these results, if I take the average execution time of a function, calling 10,000 times the pathfinding function should take only 28.1ms.

So normally, the CPU should not even reach 1% (except maybe when launching the application).

This is the second thing I don’t understand: why the CPU is permanently busy.

Having never worked with so many processes, I don’t know where to start in order to debug such problems.

The code used for my tests and benchmark is available here: GitHub - ImNotAVirus/elixir_nif_example · GitHub

This code has been simplified to include only the spawn of the 10k processes, the call to the pathfinding function and the pause of workers.

Thanks in advance

5 comments

#processes #performance #rustler

1 1301 5

2021-10-31 07:45:34 UTC

Most Liked

kip

ex_cldr Core Team

The BEAM implements busy wait to deliver a smoother and more predictable responsiveness. There are tuning parameters +sbwt, +sbwtdcpu and +sbwtdio that can change the default behaviour. This gist shows some example usage.

In general, a high reported CPU utilisation does not necessarily mean your system is under stress. Typically you don’t need to apply the tuning parameters. And all other things being equal, 50k processes isn’t at all unreasonable.

Post #2

NduatiK

Whoa! This is amazing. I am working on something similar for a Stochastic Processes class. Simulating vehicles for traffic light optimization.

I was planning to use libgraph, but this NIF based approach is getting me thinking about performance at scale. Will things work when I jump to more processes? With NIFs, probably.

Thanks and good luck!

PS. Liveview + Surface + SVGs are amazing! Probably would have gone with elm without it, but so far so good.

Post #4

Last Post!

ImNotAVirus

That was one of the first things I thought of. But the problem is that the entities will have to be able to interact with each other later on (for example if one of them enters the field of action of another one).

That’s why I preferred to make one GenServer per entity.

Post #6