Absolutely! No disrespect taken =)

Overall, I think this is generally useful if people are already using software such as Daytona, or even EC2. The whole problem Hyper solves is “I have 4 bare metal computers, how do I create N >> 4 VMs on this cluster”. In a sense, it’s kind of like k8s, except k8s is powered by containers, and Hyper is powered by VMs.

There’s a couple neat features you get with VMs – you can fork the state of the VM, ie. you can take the existing VM and create a new copy at the same state as the previous VM, and my understanding is you can’t really do this with k8s. Right now, Hyper only supports forking the FS, but RAM forking is on the roadmap too.

I’ve read the code and it made me laugh.

But first, some interesting facts. It is 57k LoC changes in 21 day, that makes it around 2k lines or code change a day (if you work the weekend). My estimate is that just reading that amount of code top to bottom like a text would take around 70 minutes daily with average human reading speed of ~240 wpm. I still don’t know if that’s a lot or not.

The funniest thing is Unit.Operators module. There are three structures, each one of them with one field and each one of them holds just an integer, indicating bytes, bytes/s and time in nanoseconds respectively. So, comparison must work out of the box, but this library actually introduces a macro which overloads most of binary operators and injects a case which deconstructs the structure, performs the operation and packs it all back. Fun thing is that I don’t know why. %Something{bytes: a} >= %Something{bytes: b} will always return the same result as a >= b. But this library explicitly deconstructs it, while slowing down all the other, unrelated to these structures, uses of the binary operators.

Architecture is super weird. It uses Horde CRDT to maintain distributed table of vm_id / vm_key -> pid in cluster which is fine, I guess (but I’d still resort to CP solution for that matter). I can still imagine the situation where you get a network split, and two VMs are started with the same key in different parts of split. Then split heals and you have a conflict. Horde resolves it with LWW, which is essentially random, so one of your VMs will just die with all the state it has. See Eventual Consistency — Horde v0.10.0

And what is actually strange is that every node must have it’s own Postgres database, which stores the VM images and layers, but the schema is super simple, it just could be three nested directories in file system. I bet that production deployment the author uses, has a single postgres database storing images. Why don’t you want to use this database to have the centralized and consistent vm_id -> pid storage? Or if you want it to be local, why do you need a database in the first place? You store layer IDs and their relations to images and blobls. You won’t have more than 10^6 of them ever and your queries are just “get/put blob by ID”, why don’t you want to use a filesystem?

And whole system relies on many subsequent :erpc calls. If one of them fails, or something resets the connecting between two nodes, your request won’t even fail, it will be partially executed. Client will receive an error, but VM may be already running, it may be not yet running (but it is about to run) or it may not be running ever. And the fun thing is that code tries to execute the erpc call to start the VM on different nodes. So in case something goes wrong with erpc call to node A, it will try to start the VM on node B, and now you will have two VMs in the cluster.

I don’t understand why you want to install dependencies manually in runtime. Imagine being unable to start your k8s because Github is unavailable and you have to download umoci and manifests.

However, it has separate mix tasks to install firecracker and suidhelper. Which is strange, because mix is a build tool and it’s tasks are available only during build and you don’t need firecracker and suidhelper during build, you need them at runtime.

And hyper doesn’t use mix releases and I suspect your production deployment is using docker. It is super strange given that you’re providing a program which manages docker. Kind of chicken and egg problem

Plus some smaller bugs and comments

This whole idea with mutable rootfs genserver process. I don’t understand it. This process manages locks and it self-deletes when idle for 30s. However, when you start it, you subscribe to it after it is started, then you start VM supervision tree, and then you acquire the mutable rootfs genserver by the VM tree process. First of all, why do you need a process at all, it just holds the map of a few fields, you can just pass around this map. Second, you have an idle timeout. If some part of system executes too slow, you risk to never acquire the lock. Why do you need an idle timeout in the first place and all these lock. I just don’t see what problem it all solves.
Same argument is true for image servers and such, but I may be wrong, I haven’t read them in the detail.

Internal VM ID generation is plain random, what means that you have a very low chance to get the same ID generated twice and there is no uniqueness check after ID generation, so you will create a VM with the ID which is already assigned to another VM and something bad will happen, like a request going to the old one will hit the new one instead. I’ve concluded that the chance to have one such collision on a cluster of a single node is around 1% in a year, but still, why not to use the database?

Then, its a system-level program, but it handles VM budgets for CPU with System.schedulers_online() instead of querying system for amount of cores. What means that if you start the Hyper with less schedulers, your whole budget system will be incorrect.

If you have no free users to build the VM, your request will fail, but you can just wait for the user to be freed. The Users genserver is the perfect place to add this logic. And your uid range must be free of any users, however the genserver code is perfectly capable of handling partially free ranges with freed list in its state.

Overall, I’ve read only the happy path of vm creating and it solves the problem generally, but details are very strange, buggy and weird. Users should not expect hyper to have anything close to the level of k8s resilience.

So cool! I had written a low-level Firecracker library with the intention of doing something like this, but never got around to it. I’m glad to see a project like this because I think the BEAM is perfect for it