I wrote this somewhere before, but the funny thing about quick is itâs not interesting at all. Apple just implemented the dumbest possible queue on top of a really, really good database.
The hard part of implementing something like a quick queue is keeping the multi-level index in sync. Except itâs not hard at all, because you can literally just write the two completely disparate index rows along with the user data in a single atomic transaction with perfect consistency, with zero effort, because FDB. That is the power of actually taking the time to design a system with useful guarantees.
I mean I donât speak for anyone else, but Iâll probably generalize versionstamps by just implementing them exactly like FDB because theyâre pretty good. For a literally-serialized system like SQLite any counter will do.
Oh, so thats a replicated state machine then. And this one is backed strictly by Foundation DB. There were some other projects which did similar replicated state machines on other databases, however I canât remember any names
Thats better, but still, I donât understand why input must come from messages. If you had input provided into the state machine by a function call (not GenServer.call or cast), you could atomically add batches of input, you could perform dirty actions without this returning-the-closure pattern, you wouldnât have a problem of some unexpected casts arriving, these messages having temporary data, etc.. And even if user wants to have input coming from messages, they could write their own wrapper GenServer which would have very explicit control on what gets added to the replicated state machine input queue and what gets ignored.
Iâd suggest something like FDBReplicatedStateMachine or fdb_rsm_server or FoundationReplicatedServer, because it describes what your program does. Key words here are âReplicatedâ, âState machineâ and âFoundationDBâ
I am still reading the library code. So far, there is a lot of room for optimization. For example, there is unnecessary double await in call. There is case dgen_queue:length(...) of 0 -> code, which can be optimized to not compute length and just check if empty
There are also some strange design decisions. For example, if dgen_config:init is not called, config will work, it will just ignore user provided values. And it makes it impossible to have two DGenServers with different backends.
Next thing is that current dgen_backend behaviour simply matches erlfdb interface. I think that you should limit a behaviour and make it more generic, cause not all distributed databases support futures, directory and keyspace operations. Otherwise you wonât see any other backends in the future. For example, this sophisticated state encoding/decoding approach youâre using is only a subject to erlfdb implementation, because If I were to implement a postgres backend, I would not need this encoding approach
Also, I think @Asd is probably right about the name making no sense, but there is a strong counterpoint to be made that calling the library âdegen serverâ is extremely funny.
w.r.t. the double await, I think youâre talking about dgen:call/4. The first wait is to a BEAM process. This puts the message onto the durable kv-queue and returns a sentinel key from which the caller can receive the final result. Given the current design, this is necessary because the caller doesnât know the details about the queueâs identity. Via regular BEAM message passing, DGen allows anyone to push a message, as long as they have a pid, or can look one up. I could have instead chosen to represent the queue details in a struct that the caller must have in order to push. This choice would violate the premise, which was to mimic the GenServer interface, because I like it and find it useful for composing programs. You may disagree with the premise, which is fine, but this is not an unnecessary action.
The second wait in that dgen:call/4 is receiving that final result, which comes from the server-pushed resolution of the watch future. In this case the message comes from the storage backend rather than the DGenServer process. The result of the operation is then retrieved.
This is not correct. The length of :dgen_queue is computed by the difference of two values in the kv store = (number of pushes - number of pops). It does not have a key that represents âemptinessâ of the queue. Adding such a key would force us to add more key conflicts to the push and pop functions. This would likely slow them down. As it stands now, we retrieve 2 values concurrently.
Youâre right. Itâs awkward and wrong.
I tried to be clear about this in the post: I donât know how to put another backend in here, but I desperately want to, and the interface of :dgen_backend will definitely have to change a lot.
So why do this at all? I find FDB Layers useful. They can be composed into higher level abstractions, and it results in the most ergonomic state management Iâve ever worked in. However, they necessarily tie you to FDB. While I happily run FDB, I donât want to forever. Since there are other great projects developing that are inspired by FDB, I hope DGen becomes a real Layer that can be compatible with those projects. A Postgres backend is not interesting - the community already has Oban.
At risk of being overly pedantic, Iâm going to challenge this, but only slightly. The original idea of supervisor is to do this, but gen_server itself is not opinionated about how, when, or why itâs restarted, or if it is at all.
Of course, the design of gen_server is amenable to being used by the supervisor in a powerful and useful way, just like you describe. Iâm a direct beneficiary of the genius design of this simple idea.
DGenServer breaks the rules a little bit. It can still be stopped and restarted by the supervisor, but if a poison message is the result of the crash, it may very well require an operator to intervene - either by correcting database state, fixing a bug in the code, or changing some upstream service. I agree this is a weakness in the design.
This is dismissing FoundationDB as a proper database. Why?
This is a prime example of why asking a one-line âwhyâ question in a forum is such a bad idea. I am sure both of you have good intentions, but because of the lack of common context, trading a bunch of short âwhyâ questions will only steer the discussion further away from truth seeking.
