Leveled LSM with the versions from the cluster pushed directly into the storage engine in place of the sequence numbers. Deep integration is the goal.
The other big advantage of a custom storage engine is that it will enable more advanced data movement. FDB has in the past performed shard moves “logically” (by reading individual keys) rather than by directly copying the binary pages on disk. They have been moving away from this using RocksDB. I think I can do better because I will control the on-disk format.
I am working on an RFD (request for discussion) for the storage engine. I plan to post the RFDs as forum threads and solicit design feedback, so look out for that.
As we have previously discussed I just think binary/binary is a nice underlying abstraction. There will be binary encodings for both keys and values, but I will design those encodings. The purpose of the value encodings, by the way, is to enable predicate pushdown in the future.
Using lexicographic order and using a custom comparator are essentially equivalent. I know you prefer the latter, but I don’t and I think we have pretty much exhausted the discussion already tbh.
