Just lurking because minimal to 0 relevant experience but love an ambitious project. But I couldn’t stand by to let unit tests be slandered like this. inferior?? come on, they’re a trade-off. you give: moderate extra time specifying input and outputs for a small piece of your API. you receive: assistance designing said api and a fairly strong guarantee future implementation changes don’t break the contract. refactorer’s best friend. This is especially true with web apps where true e2e tests are a huge pain.
tfwright
sodapopcan
I think this may be the classic case of people having different ideas of what the “unit” is. The type of granular tests I’m considering unit tests are the enemy of refactoring.
2
garrison
I had a bug not long ago where one of the fuzzers tripped an assert. Not often, maybe 1-5% of the sim runs failed.
The distributor writes special meta keys into the sharded WAL to inform the storage servers they need to change their shard map and/or fetch keyspace from another server. These special mutations are logged at a particular version (it’s a multiversion database) because they have to happen at exactly the right time. They are pushed from the Distributor, to a CommitBuffer to be batched, and then they are fragmented and shipped to the sharded WAL (TLog servers). The Storage server pulls batches from the TLog servers with a peek call. All of this is distributed and nondeterministic, including how things end up being batched on the CommitBuffer and how those batches are pulled by the Storage server.
When the Storage server applies the batches, it does so one batch at a time. Within the batch it applies one mutation at a time. It can pull a few batches at a time from the TLogs, so this is a nested loop, essentially.
To observe the special mutations, the Storage server has to diff the new value with the previous value, so it has to read the previous value and then run them both through a case statement so that it can discover what to do. For example, if the previous value was nil and the new value was a shard key, it would be a command to fetch that shard. And the inverse, and so on. Some combinations would be impossible because they have no semantic meaning (like nil → nil). I am speedrunning this explanation, obviously.
There was a bug in this code. In the second level of the nested loop, I accidentally read the old key at the current version of the Storage server rather than the version being applied currently; i.e. at the wrong level of the nested loop.
This led to a very rare edge case: if the meta mutations were batched just right, there would be two for the same key within the set of batches that happened to be pulled by the Storage server from the TLogs. This is impossibly unlikely in practice because the same meta key would almost never be written multiple times this way. But if the stars aligned, the diff might “miss” a value in between the batches.
The fuzzer that caught this aggressively spams shard moves against the cluster. Even then, only a few runs tripped the assert, because the stars had to align to make it happen. This would be very nondeterministic, but thankfully I spent months reimplementing half of OTP deterministically so that I don’t have to deal with flaky simulator runs.
Can you write a unit test to prevent this bug?
Actually, do you think you could even write a regression test for this bug?
2
garrison
Have you considered that perhaps the fact that they’re a pain is the problem we should be looking into?
1
tfwright
That’s all super interesting. Genuinely! That’s the kind of stuff I’m in this thread for. But unit tests have their place. Maybe not in your project.
Me? uh, no. i’ve got….other stuff. But sure, I have no doubt they can be improved and will be. LiveView test is a huge improvement over other JS based frameworks (though that is….not saying much). Just there’s no way around the fact it’s more complex, and unless we want to imagine a scenario where the complexity differential is approaching 0, which is certainly not present day, unit tests provide huge value for the cost.
3
Asd
Hi, it’s a good DB, I’ve read almost all of the storage and transactions code (except the constructor) and I find it much easier to read than other databases! Here are my questions:
From README
Hobbes will be closed to public contribution for the foreseeable future.
Why? Can I participate 
with some PRs please?
Hobbes is currently proprietary as it is not intended for public use.
You’re missing a LICENSE file though. Copyright must declare an owner at least
From architecture/overview.md
Hobbes was designed and built by Corporate to meet our unique needs.
What Corporate? Is it a name of the place you work at? https://corporate.fm ?
“our”? How many people are working on the project? Just curious.
As far as I can see, the storage path works like this:
- Mutation gets into MutationLog
- HybridKV pop from mutation log on flush
- This data is inserted into storage kv during flush
- Storage kv is dumped on disk during commit during flush
And I see these problems:
- Mutation log is in-memory, if it dies, data is lost
- Storage kv always keeps everything in memory, never reduces size
- Full kv dump is async
- Full kv dump is slow
- In case something fails, transaction is lost and user is not notified about it
I know that this is work in progress, so I am interested in the approach you’re planning to take for storage part. Is it going to be a B-tree variation or maybe some LSM or some new approach?
Looks like it’s not going to support arbitrary terms as keys. Why?
Don’t tell me about term_to_binary ordering, I wrote a nanolsm just to prove that storage can have arbitrary terms as keys with range reads and everything
I wrote a thing similar to Construct, but it’s a WIP thing, so I haven’t released it yes, and it uses a similar API, but slightly different approach. Instead of fuzzing, it traverses all possible paths of how things can happen in distributed system (with or without failures).
Idea behind it is this: we form a tree, where each node is a set of actions (like “send this message to some process” or “receive this message in this process” or “unblock this process after send”, etc.) which can happen at some moment, and the edge is the action which is executed which leads us to the new set of actions. Root node is formed when all tracked processes are stuck in checkpoints (like after send or right after spawn, etc.). When we execute some action, we move down this tree. And each new run is trying to traverse the new path of the tree (starting from root).
The main problem with this library is the action tree increasing in size, since even simple GenServer.call can lead to many outcomes.
I can upload the code if you want to see it.
I’ll ask more questions once I finish reading and testing it
Congrats on the initial release, I can see in git history that it took one and half years and it’s a lot of progress in this amount of time. Frankly, I am almost jealous (in a good way) of your productivity 
. I can’t wait to see newer iterations of the project.
3
dimitarvp
garrison
An honor!
Many different reasons, but for now this is a “just me” thing. Maybe down the road when I have self-hosted Git infra (really hoping ATProto works out) I would consider accepting contributions from a small number of trusted people, if those people are willing to follow some rules. For the time being I don’t want to worry about it.
This is not true (copyright is automatic), but it won’t matter once we move to alpha because then there won’t be any copyright 
It’s just me for now! “Our” is being used in a professional sense to reflect that I consider this to be the work of an organization, even if that organization is currently just me. There will be commercial products in the future; Hobbes is part of the tooling I’m building out to support those products. I am very interested in sovereignty.
And yes that is my (“Our”) website, which is currently just a blog. Which I will be posting to more regularly now, so grab the RSS if you’re interested in that!
2
garrison
This sounds cool and of course I’d love to see it
Traversing all reachable paths in a breadth-first way like this would probably prevent you from reaching bugs that only occur after some time has elapsed. The fuzzers are essentially traversing depth-first with a lot of noise injected, if you think about it. Good for finding bugs.
1
garrison
Your understanding looks correct. This design comes right from FoundationDB, where the in-memory state is multiversion and the on-disk state is not (hence the MVCC window of roughly 5 seconds).
However, I am going to obsolete this entire thing by pushing the MVCC all the way down into the storage engine on disk, at which point this entire path is going to be removed! So enjoy it while you can, I guess.
The WAL data is stored durably on the TLogs and would simply be pulled again after the Storage server reboots. The WAL is not popped from TLogs until after it is persisted by the Storage server.
Because there is no on-disk storage engine; it’s essentially mocked for sim testing. The flat storage would be on-disk. The in-memory storage is pruned by the flush and would not grow unbounded.
I’m not sure what you’re referring to exactly, but if you mean FlatStorageKV this is a mock to test recovery after restarts. It’s not a real storage engine. I was going to write another KV using SQLite (this would be trivial) but I changed my mind and started designing one from scratch.
This could mean a lot of different things. Can you clarify?
A committed transaction would never be lost. Unless fault-tolerance is violated, at which point the database should be considered lost and restored from backup.
1