I don’t have any intimate knowledge of Puma (or Ruby for that matter), but a quick search shows there are alternative servers for ruby which do asynchronous IO and/or Fibers. They might well exist because Puma is doing it wrong. A lot of older frameworks do work that way, and it is often hard to change existing code to work in a fundamentally different way.
However, most webframeworks will run multiple threads. It’s what happens inside those threads that matters. Are those threads unavailable until the request is fully served, or do the get put back into the thread pool when they are waiting for something external? That is the game changer, In the first case your CPU may be sitting there idle because all threads are ‘busy’ waiting for something from the disk of the database. The latter case allows the CPU to work on other things (e.g. new incoming request) in the mean time.
This is also basically the way the Beam VM works, Beam also starts a OS thread per core and then runs multiple Erlang processes on these threads in turn.
I’ll dig a bit deeper into (my view of) the basic principles behind this.
Firstly it’s important to note that this is only relevant when there is IO being done on which the CPU has to wait before it can finish the work. Imagine a service which just calculates the square root of an input number, nothing else. In that case the running a thread per core and just processing the incoming requests in sequence is the most efficient way of doing it. There are two possible outcomes, either you keep op with the requests, or you max the CPU and still can’t keep up. In that last case you simply lack the computing power and there’s is nothing you could have done differently which would have made it work. Worse still, anything you change will mean more overhead, reducing the number of requests you can process.
But most software isn’t like that, any webserver needs to deal with reading from and writing to the network at the least. But it’s likely doing logging to disk, reading files, talking to databases or other services, etc. Because it’s a waste of CPU cycles to do nothing while waiting for those external things you want to do multiple things concurrently, if one piece of code is waiting for something you should run something else in that time. On the conceptual level that is all, it isn’t more complicated then that. At least, when just looking at resource usage, more on that later.
On the technical level the OS solves this for you, just run a process for each piece of work you’ve got and the OS will schedule those processes on the CPU and you’re done. Conceptually there is nothing wrong with that. But a process has all sorts of stuff attached to it (users, permissions, open files, etc.), which means that each process needs quite a bit of memory just to exist and on top of that it means that switching between processes is quite a bit of work. It works fine, it’s just inefficient.
To improve on that we have threads, which are just ‘smaller’ processes which belong to the same process. It’s still the OS that is scheduling them, but there is less information attached to a thread which means they use less memory and switch faster. But it still has stuff attached to it, and we still move in and out of the kernel when switching. This also works fine, but there is still a fair bit of overhead.
Now if we want to be more efficient then threads are, there’s only one option left. We need to start scheduling the work by ourselves. And because the VM running the program has more knowledge of what is going on in the program it’s in a better position to keep track of pieces of work and do the scheduling with minimal overhead. Not zero overhead, just less overhead. This is what Beam has been doing from the start. But it is also basically what the ‘green threads’, ‘fibers’, ‘async’, ‘non-blocking’ things boil down to.
Now there is a bit more to it than just getting work done a fast a possible. When there is one chunk of work which needs an hour of CPU time to complete you likely don’t want everybody else to wait for that. So you start switching between different pieces of work in a way you deem fair, even though that creates extra overhead. How to do this is a whole science in itself with loads of trade-offs. The OS does schedule the threads in a way it deems sensible, you have some influence on that but it’s limited. Therefore, doing your own scheduling also allows you to make certain choices yourself, instead of relying on the OS. Like how you deal with high loads, whether you favor consistent response times over better average performance, how you deal with failing ‘threads’, etc. I’m guessing that for Beam those things where actually more important then the efficiency.
There are also downsides to doing scheduling yourself. Firstly it’s not easy, secondly you might need some knowledge of the hardware to get the most out of it. For example, when multiple cpu’s/core’s became a thing the Beam VM needed to be changed to support it, while those using threads got it for free as soon as the OS supported it. But taking things like hyperthreading, CPU cache size, etc into account can allow you to improve scheduling. The OS is a far better position to take those things into account, and with increasingly complex CPU’s (core-complexes, Big-Little architectures) this is more relevant then before.
While user-level scheduling is all the rage right now (for good reasons), there is already movement towards improving the OS to support more efficient threading. My guess it that this will cause a reverse movement in the future, because the effort of implementing scheduling yourself may at some point not be worth it anymore. The Beam VM might be the outlier there, as it has some design goals which differ from what mainstream software is doing.
Note that this is all about how the software eventually runs on a machine, how a programming language supports this (and hopefully makes it easy) is a whole different topic which is often muddles the discussions about it. Even more so because languages are often tied to a specific VM and thus tied in to a specific way of scheduling work. Not to mention the fact the terminology is often confusing as well.
