TIL: Erlang’s :dets is written entirely in Erlang. For whatever reason, I thought it was going to be written in native code. Nope. 
jallum
2
garrison
I spent more time on it and I have changed my mind.
The underlying cause of the double {:ok, {:error, "foo"}} situation is the coercion of throw/catch-style control flow back into case-style control flow. The reason for this is that RDBMS generally trigger a rollback on constraint violations by default so Ecto has to exist within that abstraction. This is the cause of all sorts of weird edge cases I had never thought deeply about. E.g. what if you catch a constraint with a Changeset? The transaction is still aborted but can continue by ignoring further operations. If you think about it this behavior is actually insane.
Where it gets interesting is with nested transactions. It turns out that throw/catch-style control flow is very useful here because if a constraint is violated you want to pop all the way out of the stack. Retrying only the nested part of a nested transaction makes no semantic sense because transactions are run on a snapshot which will simply violate the constraint again. So you need to retry from the beginning (at a new snapshot).
So why not “use the platform”? I don’t need a rollback() function if I can just raise. Then I can write my own retry loop around transaction() with a try/rescue. In order to write a unified retry loop this implies that the KV layer needs to raise on commit failures as well.
Putting it all together, I am now favoring a stripped-down transaction!() API which spits out its return value verbatim and raises on failure.
Repo.transaction!(fn ->
Repo.put("foo", "bar")
"foo"
end)
|> case do
"foo" -> :ok
end
There could then be a transaction() which has the old (tuple wrapping) behavior but the raising version would be strongly favored for retry loops. I also considered whether the retry loop behavior belongs in the KV layer but, at least in my case, I don’t think so. How to retry is going to be intimately married to how constraints are implemented which is behavior that belongs in higher layers. In my case I intend to work with layers myself so I can always move the abstractions around if I turn out to be wrong about this. But I think of the KV layer as a “pro tool”.
For Bedrock maybe you are interested in exposing a more friendly API to users, in which case perhaps you would want to provide a retry loop out of the box.
Either way, I retract my argument in favor of transact(). My instinct was that lessons from Ecto apply here but I no longer believe that’s the case. The concept of “rollbacks” can be thrown away entirely.
2
jstimps
I agree that the retry should be client side, at the layer if you will. But for what it’s worth, in the rare occasion I’ve needed a change in retry semantics at a higher level layer, I’ve made sure my invariant is represented by a set of keys in the DB and relied upon key conflict errors to drive my retries, adding explicit conflicts if necessary, but usually not.
In other words I’ve never personally found myself wanting a retry aside from the behavior that’s recommended for FDB clients by FDB API itself. Maybe there are other cases though. Just adding my experiences.
Regarding rollback, thank you for sharing your thoughts. I’ve never been able to apply rollback to my mental model of FDB style transactions, and appreciate you putting words to my intuition.
2
garrison
The salient point here is that retries are something which are inherently tied to higher-level behavior. Some constraints might be retryable, like taking a lock, and some might not, like reserving a username, which is likely to remain reserved even if you retry. You might think of that latter case as having its retry loop in the human layer 
I think by using exceptions I can then funnel all of the different failure cases (network failure, key conflict, constraint failure, arbitrary application failure) through the same code path. The loop can decide which errors should be retried and which should not. I think this may end up being semantic at the application level. Therefore I don’t think the KV layer is the best place to worry about such things. It can just raise. Like I said, “pro tool”.
I’m not sure if I understand what you mean here (an example might help).
But have you considered that you are allowing your application code to be shoved into an FDB-shaped (key conflict) hole when really the control should run the other way?
I don’t even think it’s the write buffering that’s the problem. Rolling back a transaction on constraint failure just doesn’t even make sense. What if I don’t want to roll back? Am I supposed to read the row first and emulate the constraint myself? Constraints literally exist so I don’t have to do that! Because it doesn’t even work under READ COMMITTED! I suppose this is just how 50+ years of accumulated mistakes shape up.
BTW, buffering writes as FDB does is an old technique. In Andy Pavlo’s courses I recall the term “private workspace” being used (though that could also be on a server). I’m pretty sure Spanner used this approach as well (with a thick client). And many more throughout history I assume.
IIRC one of the Spanner papers specifically mentions they don’t have RYW. As if it’s a good thing, too, which I find quite funny.
0
jallum
This is essentially the conclusion I’d come to and represents the semantics I’d presented originally.
Such a retry-loop exists, currently. There are a class of errors from the storage servers (temporarily unavailable, aborts, etc.) that are automatically retried (with an optional limit) with backoff + jitter.
I can see a case for transact / rollback, but it’s a convenience. In ecto, rollback(reason) does two things: 1. it uses an exception under the hood to bubble up from deep nesting, turning reason into {:error, reason} and 2. it wraps up some try-catch boilerplate that you’d need to have in place otherwise, everywhere. The way they’ve done it – it’s entirely opt-in, so if a user doesn’t like the idea of rollback/1, they don’t have to use it.
2
garrison
There is no failure handling in the livebook (that I saw) so I think when I read the code I simply assumed it worked like Ecto.transaction() and wrapped the result outside of the call. I did not read carefully enough to track the return types.
But yeah, you were right IMO, particularly here:
If the app really wants a “rollback” they can just raise themselves. To be clear, I still think most transactions should avoid this exception-style error handling. The prevalence of rollbacks within Ecto seems to come straight from mistakes made by RDBMS, which I did not fully appreciate before.
The rollback() is intimately linked with the {:ok, {:ok, value}} error coercion. You could have the former without the latter, but then what is the point? It’s just a useless wrapper around raise.
I’m not sure I’d be comfortable forcing application devs to wrap all of their transactions in a try block, but I do feel comfortable forcing layer developers to do it (sorry @jstimps) because they have a better idea of what retry abstraction to use.
0
garrison
Arguably off-topic but I randomly saw this post on Bluesky and I just had to post it here
Also, it sounds like Bluesky is going to be using FDB! Great for us as they’re very developer-facing and it will help raise awareness about the architecture.
1