Wait? You are creating and destroying network namespaces dynamically in a large scale?

We had some similar setup up for some simulated network testing a few years ago.

We created and destroyed many hundred namespaces per hour, and the system call that should create the next one simply just stalled when it should create the nth namespace.

The OS thread responsible for this call just wouldn’t be scheduled by the OS anymore and wasn’t even kill -9able. Even worse, when the parent was killed, the stale child “survived”.

It was not possible to create any further namespaces when this occured. n was constant per machine across reboots, but different on a couple of hosts we tested.

Back then my personal funtoo machine survived the most namespaces, somewhere in the 10s of millions, where most other systems staled already in single digit millions.

We were not able to hunt down the root cause, as the affected client decided to just use a VM which gets thrown away after a couple of those iterations when we are still in a safe area of already created namespaces.

So if you really do work with a huge number of dynamically created and deleted namespaces, check if any of the described symptoms happen to you as well.

PS: We did not use erlang for that project, though it seemed to be a limitation of the OS, not of erlang.

Zombie process!

It happens because programs have a few states they can be in:

• Active_Scheduleable
• Sleep_Scheduleable
• Wait_Scheduleable
• Wait

Scheduleable means the process is in a coherent enough state so it can receive signals. If it’s not scheduleable then the process is waiting on something deep in the kernel.

The ‘Wait’ one generally only happen on very few specific kernel primitives, accessing low level resources like allocating new network data (not accessing it) is one. If the kernel can’t create it because it’s out of space for it then the program will wait until there are available resources. If the program is the one that caused the kernel to run out of the resources then it is a Zombie, forever dead and inaccessible as there isn’t a single signal in the entire system that can access it, not even kill -9.

These lines are actually Port states, not Process states.

The PORT_LOCK refers to the fact that the Port is locked using a port specific lock instead of a driver lock. This is normal and nothing to worry about.