<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="368993" data-post-id="368993">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jstimps" data-post="40" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>The caller may choose to store their output in a data structure that cannot be inspected (such as a NIF resource of some kind), or it may be so deeply nested that traversing again it is an unacceptable use of time post-commit.</p>
</blockquote>
</aside>
<p>That first one doesn’t seem like a big deal to me (who does that), but performance could be an issue yeah. Worst case I can think of is someone inserting one versionstamped row and then selecting 1000 rows of something else and returning them. The question is whether that happens often enough to be a problem or if you can just throw an escape hatch in the API for the edge cases. I’ll have to think about it more.</p>
<aside class="quote no-group" data-username="jstimps" data-post="40" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>I wonder if your work can continue to use Ecto Changesets? The changeset integration with Phoenix is strong.</p>
</blockquote>
</aside>
<p>This remains an open question. I haven’t started work on any high-level APIs yet as I am still busy reinventing the universe, but I am at a stage where I need to be thinking about these things. I don’t like dependencies so I’ll probably lean towards reimplementing them from scratch.</p>
<p>Phoenix integration could be an issue but I think the form handling is probably extensible enough to make something work. Honestly I don’t really like forms anyway, especially complex forms. It’s usually better to mutate the DB state directly rather than try to shove a complex interaction through the bottleneck of a single POST. The changeset relation APIs terrify me, I never touch them.</p>
<aside class="quote no-group" data-username="jstimps" data-post="40" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>I’m interested in hearing more about it!</p>
</blockquote>
</aside>
<p>Not to spoil my own (eventual) announcement, but I have been slowly cloning FDB from scratch (this is probably somewhat obvious from my posts lol). The goal is to have an (ahem) foundation on which to build higher-level databases like a relational DB, taking inspiration from record layer and so on. I was very sad to see record layer has become an SQL DB, there were some good ideas in there that they are now discouraging (polymorphic relations). Protobufs are a terrible idea, though.</p>
<p>Honestly I just want a database I can trust. A while back my plan was to use Postgres and migrate to CockroachDB in the future; but they rugpulled it. When I started I never thought I would get as far as I have, but I could not have chosen a better database to learn distsys from. FDB is a masterpiece. Its only sin was choosing C++.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368993" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/42">Post #41</a>
	                </div>
	            </div>
              <div id="likers-container-368993" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368993"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #41"></div>
  </section>
</div>
    <div class="postbit" id="369105" data-post-id="369105">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jam" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/120/36266_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jam
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Nice! This looks really good. Any idea of what kind of scale can you get with this sync approach? It reminds me a bit of mongo change streams which last time I checked had some limitations to the point where I think they’d need to be paired with a pubsub system.</p>
<aside class="quote no-group" data-username="jstimps" data-post="37" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>I’m finding myself having to deviate from the standard Ecto API more often as the implementation increases in scope. (all the async/await functions</p>
</blockquote>
</aside>
<p>Hmm, what do you think of something like this as a way to deviate less? Not sure if it’s possible, but if it is then maybe the <code>async_*</code> functions wouldn’t be needed.</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">Repo.transact(fn -&gt;
  futures = Repo.async(fn -&gt; [
    Repo.get_by!(User, name: "Alice"), 
    Repo.get_by!(User, name: "Bob")
  ] end)

  [alice, bob] = Repo.await(futures)

  Repo.insert!(%Team{members: [alice.id, bob.id]})
end,
prefix: tenant)
</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369105" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/43">Post #42</a>
	                </div>
	            </div>
              <div id="likers-container-369105" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369105"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #42"></div>
  </section>
</div>
    <div class="postbit" id="369108" data-post-id="369108">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jam" data-post="43" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/48/36266_2.png" class="avatar"> jam:</div>
<blockquote>
<p>what kind of scale can you get with this sync approach</p>
</blockquote>
</aside>
<p>Short answer is I don’t know yet, but here’s some info from the FDB docs that might steer a future benchmark / load test.</p>
<p>The first limitation we’d hit at scale is the client-side default limit of 10,000 pending watches. This can be increased with a config change.</p>
<blockquote>
<p>By default, each database connection can have no more than 10,000 watches that have not yet reported a change. When this number is exceeded, an attempt to create a watch will raise a <a href="https://apple.github.io/foundationdb/api-error-codes.html#developer-guide-error-codes" rel="noopener nofollow ugc">too_many_watches</a> exception. This limit can be changed using <a href="https://apple.github.io/foundationdb/api-python.html#fdb.Database.options.set_max_watches" rel="noopener nofollow ugc"><code>Database.options.set_max_watches()</code></a>. Because a watch outlives the transaction that creates it, any watch that is no longer needed should be cancelled by calling <a href="https://apple.github.io/foundationdb/api-python.html#fdb.Future.cancel" rel="noopener nofollow ugc"><code>Future.cancel()</code></a> on its returned future.</p>
</blockquote>
<ul>
<li><a href="https://apple.github.io/foundationdb/api-python.html#watches" rel="noopener nofollow ugc">FDB Python Docs | Watches</a></li>
</ul>
<p>Also,</p>
<blockquote>
<p>Storage servers have their own limits, and when they are exceeded they result is that the client falls back to automatic polling. This all happens transparently.</p>
</blockquote>
<ul>
<li><a href="https://forums.foundationdb.org/t/question-about-watches-limit/2158" rel="noopener nofollow ugc">FDB Forums | Limits on Watches</a></li>
</ul>
<aside class="quote no-group" data-username="jam" data-post="43" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/48/36266_2.png" class="avatar"> jam:</div>
<blockquote>
<p>mongo change streams</p>
</blockquote>
</aside>
<p>I’m less familiar with Mongo, but I have used RethinkDB’s changefeeds, with frustrating results. I believe the key difference with FDB is that the watch itself does not distribute data to the listener. It’s only a signal that something has changed. With this approach, the watch is conceptually merely an optimization to polling.</p>
<ul>
<li><a href="https://github.com/apple/foundationdb/wiki/An-Overview-how-Watches-Work" rel="noopener nofollow ugc">FDB Docs | How Watches Work</a></li>
</ul>
<hr>
<aside class="quote no-group quote-modified" data-username="jam" data-post="43" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/48/36266_2.png" class="avatar"> jam:</div>
<blockquote>
<p>something like this as a way to deviate less</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">Repo.async(fn -&gt; [
# ...
] end)
</code></pre>
</blockquote>
</aside>
<p>Interesting idea! I think that would be a fairly straightforward change. I’ll put it in the GitHub as a potential enhancement. I would definitely welcome a PR, but also it’s something that I could get around to fairly quickly. Thanks for the idea!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369108" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/44">Post #43</a>
	                </div>
	            </div>
              <div id="likers-container-369108" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369108"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #43"></div>
  </section>
</div>
    <div class="postbit" id="369113" data-post-id="369113">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Honestly FDB watches are a tragedy, such wasted potential. If you want to implement live queries properly it is <em>extremely</em> important that you don’t miss any changes. You always want to observe a consistent prefix of the database state (I have seen this referred to as “internal consistency”) so that you don’t get torn updates or just straight up wrong data (missing a row in a collection).</p>
<p>In order to properly stream changes to a keyspace you need to read at a given version <em>and subscribe at that version</em> atomically so that you don’t miss any changes. You also need to guarantee that <em>all</em> mutations in the range are received. Watches make neither of these guarantees!</p>
<p>Also, they don’t even let you watch a range <em>at all</em>, only individual keys.</p>
<p>What hurts is FDB is architecturally <em>so amenable</em> to doing this properly. Every mutation is versioned and the MVCC architecture ensures you can read+subscribe at a given version and immediately receive all mutations since that version since they are already in memory. All you need to do is scan the in-memory MVCC store for that range, return those mutations immediately, and then add an entry in an interval tree or similar to check future mutations against when they are pulled from tlogs.</p>
<p>You can even do this <em>across storage servers</em> because FDB versions are global, so this approach can scale out. In most databases this would be a nightmare to do correctly but FDB’s underlying architecture is so strong the implementation practically writes itself.</p>
<p>But they never built it.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369113" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/45">Post #44</a>
	                </div>
	            </div>
              <div id="likers-container-369113" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369113"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #44"></div>
  </section>
</div>
    <div class="postbit" id="369115" data-post-id="369115">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I have been thinking about the async APIs a lot and I’m unable to come up with anything better. There is something about passing around futures in Elixir that just bothers me, but I think it would have to be corrected with coding style rather than a different API, i.e. trying to inline them as often as possible, like:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">[alice, bob] = Repo.await([get("alice"), get("bob")])
</code></pre>
<p>Rather than:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">alice_future = get("alice")
bob_future = get("bob")
[alice, bob] = Repo.await([alice_future, bob_future])
</code></pre>
<p>Obviously real queries will be messier than this so it will be interesting to see if that style holds up. I did have one neat idea, though: one of my favorite features of Ecto is how <code>select()</code> returns data matching the shape you ask for. Similarly, you could do something like this:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">{alice, bob} = Repo.await({get("alice"), get("bob")})
%{alice: alice, bob: bob} = Repo.await(%{alice: get("alice"), bob: get("bob")})
%{users: %{alice: alice, bob: bob}} = Repo.await(%{users: %{...})
</code></pre>
<p>…and so on. I wonder if it would be more composable that way.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369115" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/46">Post #45</a>
	                </div>
	            </div>
              <div id="likers-container-369115" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369115"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #45"></div>
  </section>
</div>
    <div class="postbit" id="369116" data-post-id="369116">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="46" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>something about passing around futures in Elixir that just bothers me</p>
</blockquote>
</aside>
<p>Honestly, me too, even though I’m the one that is responsible for it in this case. A BEAM-friendly way is to write a GenServer for each query, and call them refs instead of futures <img src="https://forum.elixirforum.com/images/emoji/apple/wink.png?v=15" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> . Maybe there’s something here, but it’s a lot more typing.</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">defmodule CreateTeam do
  # the TxGenServer would handle create, commit, retry, on the tx
  use EctoFoundationDB.TxGenServer

  # ...

  def init(_tx) do
    alice_ref = Repo.get_by(User, name: "Alice")
    bob_ref = Repo.get_by(User, name: "Bob")
    {:ok, %{alice: alice_ref, bob: bob_ref}}
  end

  def handle_ready(alice_ref, state=%{alice: alice_ref}) do
    check_finished(%{ state | alice: Repo.await(alice_ref) })
  end

  # .. same for bob ..

  def check_finished(state=%{alice: alice, bob: bob})
  when not is_reference(alice) and not is_reference(bob) do
    Repo.insert!(Team, %{members: [alice.id, bob.id]})
    {:stop, state}
  end

  def check_finished(state), do: {:noreply, state}
end</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369116" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/47">Post #46</a>
	                </div>
	            </div>
              <div id="likers-container-369116" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369116"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #46"></div>
  </section>
</div>
    <div class="postbit" id="369117" data-post-id="369117">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jstimps" data-post="47" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>A BEAM-friendly way is to write a GenServer for each query, and call them refs instead of futures</p>
</blockquote>
</aside>
<p>I don’t think process shenanigans save us here, unfortunately. Stepping back to the KV layer for a minute, the problem I’m worried about is that you really want to batch multiple requests into a single message to the storage server. If I’m querying 10 keys at once there’s a pretty good chance they’re all going to the same server (or a couple) so batching is a really good idea. But in order to do that there’s really no way to avoid an API like this:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">%{"key1" =&gt; v1, ...} = get(["key1", "key2", "key3"])
</code></pre>
<p>I don’t actually know what the FDB client does here. I would think they want to batch too, but they use individual futures for the keys. Are the reads internally delayed a short time? That just sounds messy to me.</p>
<p>Anyway, a high-level query API (like yours) will still have to compile down to that multi-key get interface under the hood, and so I really don’t see any way out here. I guess I can cope by telling myself that <code>Task.await_many()</code> exists <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>What’s funny is when using Ecto/SQL we generally get around this by just eating tons of round-trips and not caring, but that doesn’t seem like something to aspire to!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369117" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/48">Post #47</a>
	                </div>
	            </div>
              <div id="likers-container-369117" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369117"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #47"></div>
  </section>
</div>
    <div class="postbit" id="369129" data-post-id="369129">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>You’re surely already aware, but for the sake of others: unfortunately the fdbclient network protocol is not documented publicly, and all client implementations are reliant on libfdb_c. The FDB developers view the client as part of the cluster with some important functions that could be missed with a faulty implementation. This makes it a bit challenging to find answers to behavioral questions.</p>
<p>However, this quote implies that the client does batch Get requests, but doesn’t go into detail.</p>
<blockquote>
<p>Layers can easily make multiple concurrent <code>get</code> calls and wait on the returned futures. Internally, client library is free to combine multiple <code>get</code> calls destined to same storage server, into a single network request.</p>
</blockquote>
<ul>
<li><a href="https://github.com/apple/foundationdb/wiki/Multi-Gets-and-Sets" rel="noopener nofollow ugc">FDB Docs | Multi Gets and Sets</a></li>
</ul>
<p>Related: get-read-versions (GRVs) are definitely batched. This is widely documented and is a key part of achieving high read throughput.</p>
<p>Of course, the best throughput always comes from GetRange requests, so those are always preferred if possible.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369129" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/49">Post #48</a>
	                </div>
	            </div>
              <div id="likers-container-369129" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369129"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #48"></div>
  </section>
</div>
    <div class="postbit" id="369170" data-post-id="369170">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Probably what happens is the <code>get()</code>s are queued up on the client thread and then once you trigger an await on any of them they are all sent out as a batch. For some reason in my head I thought they would be sent on <code>get()</code> rather than <code>await()</code> but obviously that makes no sense.</p>
<p>The implicit awaiting going on in some of the language bindings seems confusing to me, though. I prefer the explicit <code>Repo.await()</code> in your API as it mirrors what is <em>intuitively</em> a multi-get operation even if the FDB client is still trying to be clever under the hood <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>It’s funny, if you look at an Ecto query the APIs are not actually as divergent as they first appear:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">bob_future = from u in Users, where: u.name == "bob", limit: 1, select: u
bob = Repo.one!(bob_future)

bob_future = Repo.get_async("/users/bob")
bob = Repo.await(bob_future)
</code></pre>
<p>It’s actually the same API! The only difference is that Ecto does not support executing multiple queries in one round-trip. AFAIK it actually is valid to send multiple SQL statements in one request, but it seems like nobody ever does that. I wonder why that is.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369170" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/50">Post #49</a>
	                </div>
	            </div>
              <div id="likers-container-369170" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369170"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #49"></div>
  </section>
</div>
    <div class="postbit" id="369174" data-post-id="369174">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="50" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>Probably what happens is the <code>get()</code>s are queued up on the client thread and then once you trigger an await on any of them they are all sent out as a batch</p>
</blockquote>
</aside>
<p>No, the await is simply a receive block, nothing more:</p>
<p><a href="https://github.com/foundationdb-beam/erlfdb/blob/31067d075014fd71546f2185c2a9cfb2ef7cf3ae/src/erlfdb.erl#L814" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/foundationdb-beam/erlfdb/blob/31067d075014fd71546f2185c2a9cfb2ef7cf3ae/src/erlfdb.erl#L814</a></p>
<p>The <code>get()</code> batching, however it exists, would be a dynamic decision by libfdb_c. Non-batching behavior is to send the <code>get()</code> on the wire immediately. The receive block in the wait is satisfied by the data returning on the network thread, which dispatches the message to the calling process.</p>
<p><a href="https://github.com/foundationdb-beam/erlfdb/blob/31067d075014fd71546f2185c2a9cfb2ef7cf3ae/c_src/main.c#L61C1-L92C2" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/foundationdb-beam/erlfdb/blob/31067d075014fd71546f2185c2a9cfb2ef7cf3ae/c_src/main.c#L61C1-L92C2</a></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="369174" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/51">Post #50</a>
	                </div>
	            </div>
              <div id="likers-container-369174" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="369174"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #50"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/61642/load_more?page=6">Load more posts (26 remaining)</a>
</div></template></turbo-stream>