garrison

garrison

That first one doesn’t seem like a big deal to me (who does that), but performance could be an issue yeah. Worst case I can think of is someone inserting one versionstamped row and then selecting 1000 rows of something else and returning them. The question is whether that happens often enough to be a problem or if you can just throw an escape hatch in the API for the edge cases. I’ll have to think about it more.

This remains an open question. I haven’t started work on any high-level APIs yet as I am still busy reinventing the universe, but I am at a stage where I need to be thinking about these things. I don’t like dependencies so I’ll probably lean towards reimplementing them from scratch.

Phoenix integration could be an issue but I think the form handling is probably extensible enough to make something work. Honestly I don’t really like forms anyway, especially complex forms. It’s usually better to mutate the DB state directly rather than try to shove a complex interaction through the bottleneck of a single POST. The changeset relation APIs terrify me, I never touch them.

Not to spoil my own (eventual) announcement, but I have been slowly cloning FDB from scratch (this is probably somewhat obvious from my posts lol). The goal is to have an (ahem) foundation on which to build higher-level databases like a relational DB, taking inspiration from record layer and so on. I was very sad to see record layer has become an SQL DB, there were some good ideas in there that they are now discouraging (polymorphic relations). Protobufs are a terrible idea, though.

Honestly I just want a database I can trust. A while back my plan was to use Postgres and migrate to CockroachDB in the future; but they rugpulled it. When I started I never thought I would get as far as I have, but I could not have chosen a better database to learn distsys from. FDB is a masterpiece. Its only sin was choosing C++.

1
Post #41
jam

jam

Nice! This looks really good. Any idea of what kind of scale can you get with this sync approach? It reminds me a bit of mongo change streams which last time I checked had some limitations to the point where I think they’d need to be paired with a pubsub system.

Hmm, what do you think of something like this as a way to deviate less? Not sure if it’s possible, but if it is then maybe the async_* functions wouldn’t be needed.

Repo.transact(fn ->
  futures = Repo.async(fn -> [
    Repo.get_by!(User, name: "Alice"), 
    Repo.get_by!(User, name: "Bob")
  ] end)

  [alice, bob] = Repo.await(futures)

  Repo.insert!(%Team{members: [alice.id, bob.id]})
end,
prefix: tenant)
2
Post #42
jstimps

jstimps OP

Short answer is I don’t know yet, but here’s some info from the FDB docs that might steer a future benchmark / load test.

The first limitation we’d hit at scale is the client-side default limit of 10,000 pending watches. This can be increased with a config change.

By default, each database connection can have no more than 10,000 watches that have not yet reported a change. When this number is exceeded, an attempt to create a watch will raise a too_many_watches exception. This limit can be changed using Database.options.set_max_watches(). Because a watch outlives the transaction that creates it, any watch that is no longer needed should be cancelled by calling Future.cancel() on its returned future.

Also,

Storage servers have their own limits, and when they are exceeded they result is that the client falls back to automatic polling. This all happens transparently.

I’m less familiar with Mongo, but I have used RethinkDB’s changefeeds, with frustrating results. I believe the key difference with FDB is that the watch itself does not distribute data to the listener. It’s only a signal that something has changed. With this approach, the watch is conceptually merely an optimization to polling.

Interesting idea! I think that would be a fairly straightforward change. I’ll put it in the GitHub as a potential enhancement. I would definitely welcome a PR, but also it’s something that I could get around to fairly quickly. Thanks for the idea!

2
Post #43
garrison

garrison

Honestly FDB watches are a tragedy, such wasted potential. If you want to implement live queries properly it is extremely important that you don’t miss any changes. You always want to observe a consistent prefix of the database state (I have seen this referred to as “internal consistency”) so that you don’t get torn updates or just straight up wrong data (missing a row in a collection).

In order to properly stream changes to a keyspace you need to read at a given version and subscribe at that version atomically so that you don’t miss any changes. You also need to guarantee that all mutations in the range are received. Watches make neither of these guarantees!

Also, they don’t even let you watch a range at all, only individual keys.

What hurts is FDB is architecturally so amenable to doing this properly. Every mutation is versioned and the MVCC architecture ensures you can read+subscribe at a given version and immediately receive all mutations since that version since they are already in memory. All you need to do is scan the in-memory MVCC store for that range, return those mutations immediately, and then add an entry in an interval tree or similar to check future mutations against when they are pulled from tlogs.

You can even do this across storage servers because FDB versions are global, so this approach can scale out. In most databases this would be a nightmare to do correctly but FDB’s underlying architecture is so strong the implementation practically writes itself.

But they never built it.

0
Post #44
garrison

garrison

I have been thinking about the async APIs a lot and I’m unable to come up with anything better. There is something about passing around futures in Elixir that just bothers me, but I think it would have to be corrected with coding style rather than a different API, i.e. trying to inline them as often as possible, like:

[alice, bob] = Repo.await([get("alice"), get("bob")])

Rather than:

alice_future = get("alice")
bob_future = get("bob")
[alice, bob] = Repo.await([alice_future, bob_future])

Obviously real queries will be messier than this so it will be interesting to see if that style holds up. I did have one neat idea, though: one of my favorite features of Ecto is how select() returns data matching the shape you ask for. Similarly, you could do something like this:

{alice, bob} = Repo.await({get("alice"), get("bob")})
%{alice: alice, bob: bob} = Repo.await(%{alice: get("alice"), bob: get("bob")})
%{users: %{alice: alice, bob: bob}} = Repo.await(%{users: %{...})

…and so on. I wonder if it would be more composable that way.

1
Post #45
jstimps

jstimps OP

Honestly, me too, even though I’m the one that is responsible for it in this case. A BEAM-friendly way is to write a GenServer for each query, and call them refs instead of futures :wink: . Maybe there’s something here, but it’s a lot more typing.

defmodule CreateTeam do
  # the TxGenServer would handle create, commit, retry, on the tx
  use EctoFoundationDB.TxGenServer

  # ...

  def init(_tx) do
    alice_ref = Repo.get_by(User, name: "Alice")
    bob_ref = Repo.get_by(User, name: "Bob")
    {:ok, %{alice: alice_ref, bob: bob_ref}}
  end

  def handle_ready(alice_ref, state=%{alice: alice_ref}) do
    check_finished(%{ state | alice: Repo.await(alice_ref) })
  end

  # .. same for bob ..

  def check_finished(state=%{alice: alice, bob: bob})
  when not is_reference(alice) and not is_reference(bob) do
    Repo.insert!(Team, %{members: [alice.id, bob.id]})
    {:stop, state}
  end

  def check_finished(state), do: {:noreply, state}
end
1
Post #46
garrison

garrison

I don’t think process shenanigans save us here, unfortunately. Stepping back to the KV layer for a minute, the problem I’m worried about is that you really want to batch multiple requests into a single message to the storage server. If I’m querying 10 keys at once there’s a pretty good chance they’re all going to the same server (or a couple) so batching is a really good idea. But in order to do that there’s really no way to avoid an API like this:

%{"key1" => v1, ...} = get(["key1", "key2", "key3"])

I don’t actually know what the FDB client does here. I would think they want to batch too, but they use individual futures for the keys. Are the reads internally delayed a short time? That just sounds messy to me.

Anyway, a high-level query API (like yours) will still have to compile down to that multi-key get interface under the hood, and so I really don’t see any way out here. I guess I can cope by telling myself that Task.await_many() exists :slight_smile:

What’s funny is when using Ecto/SQL we generally get around this by just eating tons of round-trips and not caring, but that doesn’t seem like something to aspire to!

2
Post #47
jstimps

jstimps OP

You’re surely already aware, but for the sake of others: unfortunately the fdbclient network protocol is not documented publicly, and all client implementations are reliant on libfdb_c. The FDB developers view the client as part of the cluster with some important functions that could be missed with a faulty implementation. This makes it a bit challenging to find answers to behavioral questions.

However, this quote implies that the client does batch Get requests, but doesn’t go into detail.

Layers can easily make multiple concurrent get calls and wait on the returned futures. Internally, client library is free to combine multiple get calls destined to same storage server, into a single network request.

Related: get-read-versions (GRVs) are definitely batched. This is widely documented and is a key part of achieving high read throughput.

Of course, the best throughput always comes from GetRange requests, so those are always preferred if possible.

1
Post #48
garrison

garrison

Probably what happens is the get()s are queued up on the client thread and then once you trigger an await on any of them they are all sent out as a batch. For some reason in my head I thought they would be sent on get() rather than await() but obviously that makes no sense.

The implicit awaiting going on in some of the language bindings seems confusing to me, though. I prefer the explicit Repo.await() in your API as it mirrors what is intuitively a multi-get operation even if the FDB client is still trying to be clever under the hood :slight_smile:

It’s funny, if you look at an Ecto query the APIs are not actually as divergent as they first appear:

bob_future = from u in Users, where: u.name == "bob", limit: 1, select: u
bob = Repo.one!(bob_future)

bob_future = Repo.get_async("/users/bob")
bob = Repo.await(bob_future)

It’s actually the same API! The only difference is that Ecto does not support executing multiple queries in one round-trip. AFAIK it actually is valid to send multiple SQL statements in one request, but it seems like nobody ever does that. I wonder why that is.

0
Post #49
jstimps

jstimps OP

No, the await is simply a receive block, nothing more:

https://github.com/foundationdb-beam/erlfdb/blob/31067d075014fd71546f2185c2a9cfb2ef7cf3ae/src/erlfdb.erl#L814

The get() batching, however it exists, would be a dynamic decision by libfdb_c. Non-batching behavior is to send the get() on the wire immediately. The receive block in the wait is satisfied by the data returning on the network thread, which dispatches the message to the calling process.

https://github.com/foundationdb-beam/erlfdb/blob/31067d075014fd71546f2185c2a9cfb2ef7cf3ae/c_src/main.c#L61C1-L92C2

1
Post #50