I’ve been idly thinking of a name for these types of systems since database feels the too much and storage engine feels like too little. Lately I’ve been fond of “State Engine”. The combination of MVCC storage server and serializable client-side-compute behaves conceptually as one giant GenServer and many small GenServers at the same time.
jstimps
garrison
garrison
The native tenancy feature has met its tragic end in an excruciating 100-commit PR. Shoutout to the one guy at Apple who seems to be single-handedly cleaning up the FDB codebase. Honestly, I think it would be easier to rewrite the thing in a real programming language, but realistically what do I know.
I had long thought Apple actually used the tenant features, but I guess it was actually a Snowflake thing. Presumably they prefer to manage tenancy in the record layer. I do think the idea of having a sanity check for commits that cross tenant boundaries has merit, though the FDB implementation seemed excessive. They kept a tenant id => keyspace map on the CommitProxies and Storage servers and used it to validate the id directly. Perhaps just validating a shared subspace prefix would be enough.
The native tenancy was the closest thing FDB has ever had to actual access control, though, which is an interesting premise. RIP.
1
jstimps
Man I knew it was on the chopping block but I’m sad to see it go. I suppose I have some code of my own to delete too.
Luckily a while ago I switched the ecto_foundationdb tenants to simply be implemented by the Directory Layer instead of the doomed managed tenants. So any ecto_fdb databases should be future proof unless they explicitly chose the experimental ManagedTenant.
I would have loved to see Tenant-aware sharding and encryption. It would make for a powerful security/compliance story.
OTOH, I share your appreciation for the project continuing to move forward in the open and clean out some of the cobwebs.
1
garrison
Good call! I would probably have done the wrong thing here as I really did figure it was an Apple feature. In hindsight I’m sure if you tracked down the commits it was all Snowflake engineers (who are now long gone). Prime example of FDB doing exactly zero communication with their community lol. (Obvi I am not blaming the individuals; they are quite literally just doing their jobs.)
It’s not clear to me from your phrasing if you’re aware of this or not, but both of these things actually did exist half-baked in the implementation.
Per-tenant encryption is kinda weird. For one it was at the page level and of course the database is multitenant so there’s no guarantee that pages are cleaved on tenant lines. I got the impression that pages including multiple tenants just… weren’t encrypted that way? It was weird.
Cleaving shards on tenant lines is more interesting because if you have a lot of tenants that are small (like Apple does) then it can probably be done without screwing up the shard size variance too much. The advantage would be to avoid “unlucky” tenants that get split across multiple shards, which I would imagine in practice would increase p99 for random users (persistently, until they get re-sharded) which is probably not great.
Overall the whole access control thing is an interesting angle. FDB’s security model is just like Erlang’s (that is: there isn’t one) and for essentially the same reasons. I have gone back and forth on whether this is a good idea or not in both cases, and honestly I’m not really sure. I’d be curious to hear what Erlang experts think as they’ve probably been debating this for 40 years lol.
0
Schultzer
If people have deep enough access to your server then you’re f. anyway, also security is about minimizing risk, not completely avoid it, which is impossible. How are you going to stop the NSA from tapping cables.
0
jstimps
Hi everyone,
EctoFoundationDB v0.6.0 is published. The focus of this release is a new module called EctoFoundationDB.Sync that represents a milestone in the read-path Sync Engine that I’ve been documenting in Livebooks over the past 9 months or so.
Inspired by what Phoenix.Sync has done for Postgres users, EctoFoundationDB.Sync offers a similar batteries-included syncing experience for those of us that wish to use FoundationDB for their apps ( => me!). The guiding principle is to declare the queries upfront and have the assigns automatically update, without PubSub.
Here’s an example LiveView showing how we can manage various syncing operations on the database as the user navigates on the page (handle_params). The “magic” auto-updating is done via careful FDB watches and LiveView’s attach_hook.
defmodule DemoLive do
use Phoenix.LiveView
alias EctoFoundationDB.Sync
import Ecto.Query
@query_catalog from(p in Product, order_by: p.name)
@query_reviews from(r in Review, order_by: {:desc, r.inserted_at})
def mount(_params, _session, socket) do
tenant = Tenant.open!(Repo, "sync-sample")
# :catalog drives our navigation bar, allowing the user to select a Product
{:ok, socket
|> put_private(:tenant, tenant)
|> Sync.sync_all(Repo, :catalog, @query_catalog)}
end
def handle_params(%{"id" => id}, _uri, socket) do
# When the user selects a Product, it's loaded in :product and its Reviews in :reviews
{:noreply,
socket
|> Sync.sync_one(Repo, :product, Product, id)
|> Sync.sync_all_by(Repo, :reviews, @query_reviews, product_id: id)}
end
def render(assigns) do
# ...
end
# That's it! Really -- nothing more
end
There is yet another Livebook that demonstrates the capabilities end-to-end: Sync Engine III - Batteries Included. (Livebook is awesome btw!)
The Livebook includes the LiveView shown above as well as one that is more sophisticated using LiveComponents.
I’ve really enjoyed the database / data management discussion on the forum lately. And especially the new projects that you all continue to contribute to the community. I do a lot of lurking, and not a lot of responding, so I wanted to thank all of you for sharing your thoughts and your code.
4
garrison
Great work as always (and great docs as always)!
Help me understand the guarantees here. It seems like to sync a collection you add a watch on a special (per-tenant) metadata key that is updated along with any row in the “table” (using atomics presumably). So you are guaranteed to observe that an update to a collection has occurred.
FDB does not pass along any information about what has been updated, though, so you cannot incrementalize this. Therefore you must re-run the query as a whole? When that happens you get strict serializability guarantees so the collection will reflect the update and will be a consistent snapshot.
If you have multiple syncs, you have multiple watches which fire independently. However, if you re-run the query for each particular watch you will observe changes out of order (tearing). If you re-run all queries when any watch is triggered you can maintain a consistent snapshot, but this comes at a further performance cost because there is no incrementalization. But I think you’re taking the former approach based on the docs?
I will warn you that dealing with this tearing behavior from the application side quickly becomes maddening. I did this for a while with LiveView and PubSub and that’s how I ended up with such unreasonable opinions about consistency. It’s actually much worse than I had thought at the time, though.
I don’t think FDB can do better than re-running all queries every time anything changes, but that’s fine for many applications (especially since everything is tenant-scoped). I guess you could write all changes in duplicate to a versionstamped log (event sourcing!), but then you have to clean it up and that’s a whole thing.
1
jstimps
Thanks for taking a look, and the kind words!
You nailed it exactly right, but I will add one point about the new watch that is created after the read.
When a key changes (e.g. one of the metadata keys as you mentioned for a collection), all futures corresponding to the watches on that key are resolved. So if we have 5 LiveViews active on a particular collection, we have 5 watches on a single key, and each future is resolved independently. Then it’s up to the client to decide how to retrieve the new data. In the case of EctoFDB.Sync, I chose to have each listening entity do their own new read of the data. There are no guarantees on the read version for this specific transaction.
However, a new watch is created in the same transaction as the read. So that if the key were to change again after that read, the same entity would be notified again to perform yet another read, and yet another watch in the same transaction. Presumably, the key will eventually stop changing and each entity can cease querying the database. I believe this approach to creating a watch in the transaction does provide a guarantee[^1] that each listener will eventually (scary word!) be consistent. Each page will not get the same GRV, but they should have the same bytes from the key-values once it settles, assuming I implemented this all correctly.
I’m evoking eventual consistency here because in this discussion we’re including the LiveView and the browser itself as a participant in the database. I’m not sure what it would look like to have multiple pages having a truly consistent view of the data and with point-in-time guarantees on their agreement. Perhaps some coordination of a GRV across pages. Have you come up with some other approach here?
On the question of performance, EctoFDB.Sync is indeed relying on the database to be able to perform well having many concurrent transactions performing identical key reads. So far I’ve seen acceptable performance from FDB without having to do some sort of caching layer. FDB is supposed to live very near your application anyway, so caching would be a design mistake, IMO. But my EctoFDB use cases are very light workloads so far. Will this approach scale? Time will tell.
[^1] Note that if a key changes from A → B → A quickly enough, then the watch will not fire, and stays unresolved. In such a case, any listener still has the correct data anyway.
1
garrison
Oh I think I left some ambiguity on the table by mistake. I don’t mean multiple pages, I mean multiple queries on the same page. An example would be better.
Say we have a page with authors and books. We sync all authors and books to the client, with two sync_all() calls, and render them like this:
<div :for={{_id, book} <- @books}>
<div>Title: {book.title}</div>
<div>Author: {List.keyfind!(@authors, book.author_id, 0).name}</div>
</div>
(There are some who will suggest that you join in the database here. Ignore them, ngmi)
You have probably already spotted the problem: we have no ordering/frontier guarantees here, so @books can update before @authors and tear. If a new book is added with a new author, we will crash because the author is not known when the book appears.
If you are smart you can imagine some simple hacks to get around this, but if you are wise then you will recognize that this is the road to hell. And I have been there.
What you want is for the changes to arrive in versioned batches so that you can always stitch together a consistent snapshot of a page. This property is not called eventual consistency but internal consistency, and I first learned its name from that article I linked. (Before I knew what it was called I had also made up a name, “externally-consistent snapshot”, which is amusingly an oxymoron.)
However you have neither changes nor batches and you are not incrementalizing anything because FDB won’t let you, so all you have to do is re-run the queries when something changes. But, and this is what I was getting at, you need to re-run all queries on the page within a transaction to get a consistent snapshot.
This has performance implications but it’s not the end of the world, particularly if your app naturally shards into tenants (which are conveniently disjoint in their queries).
1