Hello @nxy7

Yes, you are right the testing ground is not well explained, let me complete the informations below:

The test scenario:

• 10,000 NPC entities (not players) in a single zone

• Each entity runs behaviors every tick (wander, chase, spatial awareness)

• Each tick: query neighbors, update position, serialize state

• 30 ticks per second

Player model:

• Tested with 100 simulated players

• Each player has AOI (Area of Interest) ~500 visible entities

• Per-player delta snapshot: ~2ms

• Full 10K broadcast (worst case): 22ms

What entities DO per tick:

1. Spatial query: "Who's near me?" (ETS grid lookup)
2. Behavior: Decide action (chase/flee/wander)
3. Update: New position/velocity
4. Encode: Serialize to binary (18 bytes)

Scaling to 10K × 10K:

Split zones. Each zone handles 10K. Shadows (PubSub) connect boundaries. Topology scales horizontally—add nodes, add zones.

Zig/Rust sidecar:

Architecture supports it via Ports. Heavy math (pathfinding, UMAP, physics) can offload. BEAM handles state + networking, sidecar handles computation.

Best,

Nice! I’ve been (slowly) working on a World of Warcraft server, Thistle Tea, that has some similarities.

In Thistle Tea, players and NPCs are processes, but I’ve abstracted the interface to use GUIDs. The idea is that the boundary process layer will be straightforward enough to swap for something that groups entities by zone instead if it ever becomes necessary, without needing to change (much) game logic code.

IO lists are a good idea, they’ve been on my list to benchmark sometime but haven’t got around to it. Right now building packets is just binary concatenation. Bandwidth measurements are a smart idea, I have latency metrics but bandwidth would helpful to understand the entire system performance.

I’m also using ETS for a spatial grid and it’s been working really well. Recently also started using ETS for per entity metadata that other processes need access to in hot loops, that seems to work decent to avoid the message passing overhead where necessary.

I’ve mostly been focusing on building out functionality, recently got a rudimentary NPC AI wired up using behavior trees to get the basics of combat working. Stuff’s tricky because the client really likes crashing if packets are even slightly malformed, and some of the expected packet layouts aren’t very clear, lol.

It uses a Rust NIF for pathfinding, which would’ve been a pain to implement from scratch. Still a bit of wonkiness there for me to debug sometime, though. Mobs like to stutter around a bit when repathing, which makes me think the server isn’t perfectly simulating movement the same way the client is (or some other bug).

Here are a few blog posts I wrote about the process if you’re interested.

2° Round - lessons from the metal: optimizing 10,000 entities on a DL380 (Xeon vs i7)

The Setup
After my last post about 10k entities on a modern i7, I moved the project to a refurbished HP DL380 Gen9 2014 (Dual Xeon E5-2650 v3 @ 2.30GHz). The goal was to see how the BEAM handles high-frequency simulation on older, high-core-count server hardware using Docker.

The “Xeon Wall”
The first run in Docker was a failure. Code that ran in 11ms on my 5GHz desktop workstation hit 240ms+ on the server. The lower single-core clock speed (2.3GHz) exposed serial bottlenecks that were previously hidden:

1. The QPI/NUMA Gap: Docker was straddling both physical CPU sockets. Moving data between Socket 0 and Socket 1 (Distance 21) added massive jitter to the memory-heavy AI passes.

2. The Socket Tax: Moving 320KB of entity data through Linux kernel UDS sockets every 33ms to our Rust physics sidecar was eating ~90ms in context-switching and marshalling.

The Fixes
We had to move from standard “Web” patterns to “Systems” patterns to get under the 33ms budget:

• Shared Memory Slab: We replaced Unix Domain Sockets with a shared memory segment in /dev/shm. Elixir and Rust now communicate via a 64-byte aligned “Foundry Slab.” Marshalling time dropped from 88ms to < 1ms.

• Docker NUMA Pinning: We used --cpuset-cpus and --cpuset-mems to lock the container to Node 0 (Cores 0-9 and their hyperthreads). This kept the BEAM schedulers and the memory local to the same physical silicon.

• The N-1 Pipeline: We deferred the physics merge to the start of the next tick. The workers now “gulp” the previous physics result while they are already reading from ETS for the AI pass, reducing ETS write-pressure.

The Results

• DL380 (Xeon E5-2650 v3): 22-24ms P99 (Solid 30Hz in Docker )

• i7-13700 Workstation: 7-10ms P99 (Running the same “Foundry Slab” architecture)

Takeaway
Running high-performance Elixir in Docker is totally viable, but hardware is not a transparent layer. On older server metal, the “Postman” (I/O) is often the bottleneck, not the “Engineer” (The Logic). By moving to Zero-Copy SHM and respecting NUMA topology, we reclaimed over 200ms of tick time.