If the state is small and performance can be manged by a single GenServer, do you still need a database? I agree true state syncing is much more interesting and useful than message passing.
derek-zhou
garrison
This very much depends on context. There are tons of things for which a GenServer is exactly what I would recommend, e.g. for the “realtime cursors” example.
But if you have transactions over application data things start to get tricky. It’s very important that the “log” of events you’re consuming is designed properly. If you can observe transactions out of order, or partially, things go off the rails quickly. The more stuff you do outside of the database, the more room there is for weird consistency bugs.
OTP is very powerful, but it’s also pretty old and lacks good primitives for building distributed systems with strong consistency. Hobbes basically only uses message passing, local name registration, and ETS. Tools like :mnesia, :global, and even monitors are useless in practice.
Also, for a lot of stuff you actually want persistence, and even just safely writing data to disk is so much harder than you probably think. That’s why I’m trying to build a primitive that can take care of that stuff. I think I can lower the barrier to entry substantially.
0
derek-zhou
I don’t want a log of events. As you hinted, thing go off rails quickly when network is misbehaving. I want state deltas:
- producer advertise new state version
- consumer poll for state updates
- producer compute state delta and reply to the consumer poll
- consumer apply received state delta locally
This should be more robust in real world network.
1
garrison
You can implement state sync over message passing this way, and it’s fine for many things. But the polling increases the latency, so eventually you will start to get ideas about how you could push the state updates to the client directly and you will reinvent the log from first principles as well 
Producing a delta is also not trivial in practice. You would need to diff the current version against a past version, which means your state is now a persistent data structure. Now you need to invent garbage collection. And how are these objects versioned? Are you guaranteeing linearizability? Etc.
1
derek-zhou
jstimps
It’s true that EctoFDB itself doesn’t attempt to incrementalize FDB state, but I’m not convinced we can say that it cannot be done. FDB can take consistent backups, both snapshot (backup agent) and real-time (DR agent). Someone may be able to tap into that system in a novel way. I tried, but ran out of time and focus, and abandoned that effort. It does require reverse engineering the data stream, which is not documented, and could change in any new version. Maybe that is enough to call it impossible.
On the other hand, according to my favorite ghost in a jar (Gemini), Postgres does officially support low level WAL parsing, so kudos to them for that.
Will Hobbes incrementalization be based off of a read version? e.g. the client is notified of the content of a change at a particular version stamp, and the client is responsible for understanding the full state at that precise version stamp so that it can consistently process the delta? Or will the communicated delta provide a from-versionstamp and a to-versionstamp, allowing the client to evolve their local state to the new versionstamp?
Am I on the right track here?
1
garrison
You can definitely modify FDB and add a new role to parse WAL segments (this has been done before for streaming backups). Parsing the backup output from the DB itself is an interesting idea but I would expect there to be a lot of latency. I could be wrong.
There’s also the fact that mutations are spread semi-randomly onto TLogs due to FDB’s frankly unhinged replication policy. If you check the Hobbes repo I actually just wrote a new RFD about that because I’m finally moving to copysets. Gonna be some messy diffs today lol.
My idea is to allow a subscription to a key or range on a given Storage server. The Storage server already applies its mutations in perfect version order, so it can simply check the list of subscriptions as it goes.
It’s important that you can perform a read+subscribe without missing anything in between, and we can support that by performing the subscribe at the same read_version as the reads. At the time the subscription is opened the Storage server can read the multiversion store for that range and send logs for any keys that were modified after read_version immediately. I assume FDB watches must do something similar.
I don’t see any reason to send “from” versions. The primitive is “give me all changes for this range and do not miss any”. As long as you actually fulfill that promise, you can always reconstruct a valid view of the range.
There is one more edge case: you have to send nacks to the client when there are no mutations at a version. Otherwise you can’t maintain a view across shards because the frontier won’t advance.
0