<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="359286" data-post-id="359286">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="30" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>necessitate 8GB per process for a small DB. I would imagine this requirement is for operating at scale.</p>
</blockquote>
</aside>
<p>The FDB knobs are tuned with the 8GB per process assumption. Changing these knobs is generally reserved for advanced deployments. <a href="https://forums.foundationdb.org/t/constrained-ram-in-an-application-development-environment/347" rel="noopener nofollow ugc">ref</a></p>
<aside class="quote no-group" data-username="garrison" data-post="31" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>rewriting a table, even in the background, is actually physically expensive. FDB is a huge distributed btree, and btrees have a lot of write amplification.</p>
</blockquote>
</aside>
<p>Without key clears or inserts I don’t believe the btrees will need to rebalance in any significant manner. Unless I am missing something?</p>
<aside class="quote no-group quote-modified" data-username="garrison" data-post="31" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>I think this design would work…</p>
</blockquote>
</aside>
<p>Yes, this is very much in line with my thinking, taking inspiration from Erlang’s atom table, but making it a permanent part of the metadata in the DB. However, there is likely a challenge with coordinating changes to Ecto Schemas across a distributed app. As you deploy a new app release, some clients will be on one Ecto.Schema and others on the new Ecto.Schema. It is during these transition periods that I believe something more must be done to prevent wires being crossed. For example, a naive approach would cause a node on the old Ecto.Schema to start getting <code>nil</code>s for a field that has been remapped in the symbol table. This is probably all surmountable, but it’s a significant amount of work.</p>
<aside class="quote no-group" data-username="garrison" data-post="31" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>for correctness you would have to read the “schema version” in every transaction to ensure it matches what the client expects. This would create a bottleneck for a large DB, but you could use the <code>\xFF/metadataVersion</code></p>
</blockquote>
</aside>
<p>EctoFDB already does what you describe in each transaction. I’m not familiar with <code>\xFF/metadataVersion</code>; I will have to look into it, thanks for the tip!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="359286" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/32">Post #31</a>
	                </div>
	            </div>
              <div id="likers-container-359286" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="359286"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #31"></div>
  </section>
</div>
    <div class="postbit" id="359290" data-post-id="359290">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jstimps" data-post="32" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>The FDB knobs are tuned with the 8GB per process assumption. Changing these knobs is generally reserved for advanced deployments.</p>
</blockquote>
</aside>
<p>Do they actually statically allocate all that cache memory at startup? If so that’s a shame if it’s not configurable. There is nothing in FDB that would consume so much RAM for a small DB. Those knobs are <em>probably</em> pretty safe, though, I would think.</p>
<p>Tigerbeetle for example allocates <em>all</em> of their memory at startup, which is really cool. But it has to be <em>configurable</em>.</p>
<aside class="quote no-group" data-username="jstimps" data-post="32" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>Without key clears or inserts I don’t believe the btrees will need to rebalance in any significant manner. Unless I am missing something?</p>
</blockquote>
</aside>
<p>When you rewrite a key you are still writing the page to disk. There would be no splits or joins, but you are still wearing out your SSD with the writes. They can only take so many.</p>
<p>Database b+trees are not balanced like a binary tree, they are very wide and they just split the root instead. There is a lot of write amplification if you write a single key in a page because you have to rewrite the whole page. 100b k/v with 4k page size is 40x write amplification, for example. For a cow tree you then have to rewrite each parent page too, so if it’s 3 levels deep that’s 120x. Redwood uses a really weird versioned page table because they were trying to do MVCC at the storage level (but I get the feeling they gave up), so it’s not so simple. Also obviously batching saves a <em>lot</em> here.</p>
<p>Btrees do not like random inserts because they fragment the tree. If a node splits at &gt;100% and joins at &lt;50% then over time you would expect the average node to be 75% full, so there is 25% wasted space (“slack”). To reduce the slack you can rewrite the btree with sequential insertions. FDB data movement requests keys, in order, and writes them back into the “destination” btree, which conveniently compacts it. Users were abusing this property to keep their btrees in good shape by constantly excluding and re-adding nodes one at a time to compact the btrees. So they added that as a feature (“perpetual storage wiggle”).</p>
<p>Which brings me to an amusing point: since the storage wiggle rebuilds the btrees from scratch every so often anyway, Redwood <a href="https://forums.foundationdb.org/t/continous-growth-of-redwood-disk-usage-with-no-client-connected-possible-bug/3659/13" rel="noopener nofollow ugc">does not merge nodes at all</a>!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="359290" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/33">Post #32</a>
	                </div>
	            </div>
              <div id="likers-container-359290" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="359290"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #32"></div>
  </section>
</div>
    <div class="postbit" id="359293" data-post-id="359293">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jstimps" data-post="32" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>As you deploy a new app release, some clients will be on one Ecto.Schema and others on the new Ecto.Schema. It is during these transition periods that I believe something more must be done to prevent wires being crossed.</p>
</blockquote>
</aside>
<p>The <em>database schema</em> (which comes from migrations) is stored statefully and transactionally in the database. FDB’s consistency guarantees will ensure the data is never corrupted (assuming my design has no mistakes!), so you don’t need to worry about that.</p>
<p>The <em>Ecto schemas</em> of course would, ideally, match up with the database state - but Ecto is specifically designed so that this is <em>the user’s problem</em>. Your guarantees here are the same guarantees that, say, Postgres, provides: there is a schema where “these fields” have “these names”. The user is responsible for ensuring their Ecto schemas are compatible with that. If not you throw an error! (or Ecto does).</p>
<p>There are situations where a user might have to break out those “safe Ecto migration” strategies to handle their own nodes’ inconsistencies. But you don’t have to worry about that - your job is just to ensure migrations execute quickly without rewriting the table!</p>
<p>BTW, my comment had grown long but there are a number of enhancements you can make to the schema:</p>
<p>You can store a type along with each “field” and raise on a mismatch like Postgres would.</p>
<p>You can store a “default” value with each field so that new columns can default to something other than <code>nil</code>. If the field is missing from a row use the default. Note that you have to be careful never to update the default value once the field is created or you would corrupt old rows.</p>
<p>You could even do foreign keys, which would be pretty cool.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="359293" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/34">Post #33</a>
	                </div>
	            </div>
              <div id="likers-container-359293" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="359293"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #33"></div>
  </section>
</div>
    <div class="postbit" id="368882" data-post-id="368882">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<h2><a name="p-368882-v050-1" class="anchor" href="#p-368882-v050-1" aria-label="Heading link" rel="nofollow"></a>v0.5.0</h2>
<h3><a name="p-368882-notable-enhancements-2" class="anchor" href="#p-368882-notable-enhancements-2" aria-label="Heading link" rel="nofollow"></a>Notable Enhancements</h3>
<ul>
<li>Index Metadata [<a href="https://hexdocs.pm/ecto_foundationdb/metadata.html" rel="noopener nofollow ugc">doc</a>] now makes use of FoundationDB’s <code>\xff/metadataVersion</code> key,<br>
which allows the client to cache metadata and maintain transactional isolation<br>
without having to wait on any keys. Thanks <a class="mention" href="/u/garrison" rel="nofollow">@garrison</a> for pointing this out!</li>
<li><code>Versionstamp</code> [<a href="https://hexdocs.pm/ecto_foundationdb/Ecto.Adapters.FoundationDB.html#module-versionstamps-autoincrement" rel="noopener nofollow ugc">doc</a>]: Added the ability to insert objects with a monotonically increasing integer id, via FoundationDB’s versionstamp.</li>
<li><code>SchemaMetadata</code> [<a href="https://hexdocs.pm/ecto_foundationdb/Ecto.Adapters.FoundationDB.html#module-schema-metadata" rel="noopener nofollow ugc">doc</a>]: This is a new built-in Indexer that allows your app to watch and sync a collection of objects in a tenant. (Demo: <a href="https://hexdocs.pm/ecto_foundationdb/collection_syncing.html" rel="noopener nofollow ugc">Sync Engine Part II - Collections</a>)</li>
</ul>
<p><a href="https://hexdocs.pm/ecto_foundationdb/changelog.html" rel="noopener nofollow ugc">Changelog</a></p>
<p><img src="https://forum.elixirforum.com/images/emoji/apple/heart.png?v=15" title=":heart:" class="emoji only-emoji" alt=":heart:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368882" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/35">Post #34</a>
	                </div>
	            </div>
              <div id="likers-container-368882" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368882"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #34"></div>
  </section>
</div>
    <div class="postbit" id="368963" data-post-id="368963">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Very nice work!</p>
<p>Versionstamps for auto increment are something I have actually thought about a lot. I’m not so sure it’s a good idea most of the time, for a number of reasons. The Record Layer IIRC uses UUID keys (probably v4?) for records, though that is not always a good idea either.</p>
<p>For one, they are a bit wasteful: 96 bits because they need to encode the 64 bit version plus a transaction <em>and</em> operation index (2 bytes each). And then if you want to ship tenants around like CloudKit you need another few bytes for a generation index because FDB versions are not correlated across clusters (which is actually a design mistake in FDB IMO, it could be fixed). But most of the bits in all three parts of the versionstamp are wasted (nobody is doing 2^16 * 2^16 * 1,000,000 writes/second).</p>
<p>And then for all of those wasted bits you don’t get global uniqueness guarantees because the ids chosen will be highly correlated (they count from zero). With a 96 bit versionstamp plus, say, 32 bits for the tenant move counter you literally have a 128 bit UUID except without the UU. Obviously in most cases you don’t need globally unique ids across clusters, but if we’re going to burn the bits anyway it sure would be <em>nice</em>…</p>
<p>But the bigger problem is that they harm transaction composition. You <a href="https://forums.foundationdb.org/t/why-is-read-or-wrote-unreadable-key-necessary/3753" rel="noopener nofollow ugc">cannot read versionstamp keys from the RYW cache because it would violate strict serializability</a>. * It is also annoying to work with primary keys which are only known after commit in Elixir because structs are immutable (I see you discovered this as well), so you have to make the API ugly, which I am not a big fan of.</p>
<p>I think most of the time you are better off using UUID keys (v4 or v7 depending on the use case) or a 64 bit autoincrement key via the <a href="https://activesphere.com/blog/2018/08/05/high-contention-allocator" rel="noopener nofollow ugc">old high-contention allocator trick</a> (it always amused me that this blog post uses Elixir syntax for seemingly no reason).</p>
<p>There are a few cases, though, where versionstamp keys are useful. In particular, a high-contention queue where you want atomic appends which are strict-serializable. This was the use-case they were created for: CloudKit does a lot of syncing between sometimes-offline devices (naturally) and so they need a good way to grab “all changes since X version”. I know you’ve seen the talk so you’re aware of this <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>* Incidentally I’m not sure this is actually true, but usually when that happens the FDB devs have simply thought of something I didn’t, so I put the burden of proof on myself for the time being.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368963" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/36">Post #35</a>
	                </div>
	            </div>
              <div id="likers-container-368963" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368963"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #35"></div>
  </section>
</div>
    <div class="postbit" id="368971" data-post-id="368971">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>You’re right about the weaknesses of Versionstamps (really autoincrementing keys in general). I should probably make that more clear in the docs.</p>
<p>Another missing feature of FDB Versionstamps: you can’t set the versionstamp on both the key and value at the same time. This makes maintaining an index key while supporting GetMappedRange awkward, but still doable by constructing a more complex mapper tuple:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">  defp mapper(tenant, idx_len) do
    offset_fun = fn offset -&gt;
      [
        "{V[#{offset}]}",     # prefix
        "{V[#{offset + 1}]}", # source
        "{V[#{offset + 2}]}", # namespace

        # pk: get it from the index_key because the versionstamp lives there, not in the value
        "{K[#{offset + 5 + idx_len}]}",

        "{...}" # rest
      ]
    end
</code></pre>
<aside class="quote no-group" data-username="garrison" data-post="36" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>cannot read versionstamp keys from the RYW cache because it would violate strict serializability</p>
</blockquote>
</aside>
<p>If you allow me to speculate a bit, I suspect this stems from the “set versionstamp” ops being atomic ops. It seems unnatural to allow keys related to atomic ops on the RYW tx to be readable from the cache (for example, an “add” operation on a key does not have a meaningful RYW cached value). In the case of set_versionstamp_key, I suppose it could be feasible and useful, but your transaction logic would still have to understand that portions of that cached key are not in their final form. And if your transaction needs to code around this defensively anyway, I figure you’re better off doing your own in-transaction cache.</p>
<p>Your point still stands about versionstamps making transaction composition harder - they sure do!</p>
<aside class="quote no-group" data-username="garrison" data-post="36" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>primary keys which are only known after commit in Elixir because structs are immutable (I see you discovered this as well), so you have to make the API ugly</p>
</blockquote>
</aside>
<p>Guilty as charged, but the API I landed on is acceptable, IMO. I may yet propose a <code>postprocess</code> callback in Ecto to allow the adapter to participate in the struct creation, if time allows.</p>
<p>Confession: I’m finding myself having to deviate from the standard Ecto API more often as the implementation increases in scope. (all the async/await functions, transaction vs transact vs transactional). FDB is unique in a lot of ways, though, so I don’t presume that a lot of these ideas are generalizable to other DB clients - yet <img src="https://forum.elixirforum.com/images/emoji/apple/grin.png?v=15" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20">.</p>
<aside class="quote no-group" data-username="garrison" data-post="36" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>In particular, a high-contention queue where you want atomic appends which are strict-serializable.</p>
</blockquote>
</aside>
<p>Agreed, this is the <a href="https://github.com/foundationdb-beam/ecto_foundationdb/blob/23202f1dcc7885dd8bc32e72c1d043dde1e6810d/test/ecto/integration/versionstamp_test.exs#L104" rel="noopener nofollow ugc">most</a> / <a href="https://hexdocs.pm/erlfdb/kv_queue.html" rel="noopener nofollow ugc">compelling</a> use case for Versionstamps <img src="https://forum.elixirforum.com/images/emoji/apple/smile.png?v=15" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"> . I’ll update the docs to reflect this, too.</p>
<aside class="quote no-group" data-username="garrison" data-post="36" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>CloudKit does a lot of syncing between sometimes-offline devices</p>
</blockquote>
</aside>
<p>I’d like to explore this area a little bit myself. My undeveloped idea is to use the modules from <a href="https://github.com/OpenRiak/riak_dt" rel="noopener nofollow ugc">riak_dt</a> as Ecto.Type fields and implement the Ecto Adapter to perform merging upon an update. That is, to allow 2 FDB clusters to evolve separately and then have them merge at a later time.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368971" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/37">Post #36</a>
	                </div>
	            </div>
              <div id="likers-container-368971" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368971"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #36"></div>
  </section>
</div>
    <div class="postbit" id="368973" data-post-id="368973">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jstimps" data-post="37" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>I suspect this stems from the “set versionstamp” ops being atomic ops</p>
</blockquote>
</aside>
<p>This is part of it but versionstamp is substantially weirder. With an atomic add you know the key but not the value. With a versionstamp key you know the <em>value but not the key</em>. Which means that you need to block any gets or scans within the range where the key <em>could</em> be (mercifully bounded to the read version plus 5s) because you don’t know where exactly the key is supposed to be <em>in that range</em>, even though you know the value!</p>
<p>If you think about the actual use-case here (primary key), you would want to be able to use it (and compose with it) as a foreign key. So function <span class="hashtag-raw">#1</span> creates a struct with a relation, and function <span class="hashtag-raw">#2</span> can look up the foreign key <em>from the RYW cache</em> without knowing that it actually <em>only</em> exists in the RYW cache. Even though we don’t know the <em>real</em> versionstamp key this entire pattern can be implemented on top of a placeholder key without violating correctness. I mean, I <em>think</em> it can anyway. This stuff is hard.</p>
<p>Weirdly, even a range scan should be correct. There can’t be any versionstamp keys in our snapshot above the read version anyway (obvious invariant), so there are no other keys in that range to read and therefore the <em>only</em> key in a range scan of <code>[read_version, read_version + 5000000)</code> should be our key! Of course if any other keys get inserted then we abort anyway so reading that range is a terrible idea but it’s <em>correct</em>, which is the important part.</p>
<p>So the only case where you would actually run into problems is if a previous transaction <em>lied</em> and inserted a versionstamp key into the range (keys are just bytes after all). Then we don’t know where our key lines up compared to that key, so a read of that range violates correctness.</p>
<p>But, why would you do that?! If we’re storing structured data we know this situation will never arise, so a strict serializability violation shouldn’t happen, ever. So it’s a weird case, I think, where the safety mechanism is only needed because FDB has to support unstructured keys for a use case where the data will only ever be structured in practice. The tyranny of key value stores, <a href="https://buttondown.com/jaffray/archive/its-time-to-stop-building-kv-databases/" rel="noopener nofollow ugc">I guess</a>.</p>
<p>But I still feel like I might be missing something, so I could just be totally wrong.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368973" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/38">Post #37</a>
	                </div>
	            </div>
              <div id="likers-container-368973" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368973"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #37"></div>
  </section>
</div>
    <div class="postbit" id="368978" data-post-id="368978">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jstimps" data-post="37" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>Guilty as charged, but the API I landed on is acceptable, IMO. I may yet propose a <code>postprocess</code> callback in Ecto to allow the adapter to participate in the struct creation, if time allows</p>
</blockquote>
</aside>
<p>For me the underlying problem here isn’t Ecto, it’s immutability. I see you’ve gone with a promise/await style API to batch updates but I’m not a big fan of that style. I want something which looks closer to how Ecto works now, like:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">%User{} = user = transaction(fn -&gt;
  user = Repo.insert! %User{name: "foo"}
  token = Repo.insert! %UserToken{user: user, token: generate_token()}
  {:ok, user}
end)
</code></pre>
<p>With Ecto we do this all the time, and since the User gets its id <em>at insert time</em> with e.g. Postgres we can just return it. But with FDB of course this is not so easy. We need to populate the versionstamp after <em>commit</em>, so then we need to run every struct we want to return through some function again, which is ugly and error-prone.</p>
<p>Actually I was going to continue making this point but I think I’ve just had an epiphany: you can use the transaction() barrier to deep-search the return value and populate any missing versionstamps.</p>
<p>I can’t believe I never thought of this before LOL, that’s actually a beautiful solution. Never mind, I guess!</p>
<aside class="quote no-group" data-username="jstimps" data-post="37" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>I’m finding myself having to deviate from the standard Ecto API more often as the implementation increases in scope</p>
</blockquote>
</aside>
<p>Yeah, I’ve already accepted that my own work is going to necessitate throwing Ecto out with the bathwater. It’s a shame because Ecto is probably my favorite library of all time, but at the end of the day it’s evolved to be an SQL client more than anything else and I don’t think that is something that can or should be changed at this point.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368978" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/39">Post #38</a>
	                </div>
	            </div>
              <div id="likers-container-368978" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368978"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #38"></div>
  </section>
</div>
    <div class="postbit" id="368980" data-post-id="368980">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="39" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>you can use the transaction() barrier to deep-search the return value and populate any missing versionstamps.</p>
</blockquote>
</aside>
<p>I have considered this approach, but I think it can be slow and error-prone. The caller may choose to store their output in a data structure that cannot be inspected (such as a NIF resource of some kind), or it may be so deeply nested that traversing again it is an unacceptable use of time post-commit. These are edge cases, but would be the source for confusion, and for that reason I’ve avoided it.</p>
<aside class="quote no-group" data-username="garrison" data-post="39" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>Ecto</p>
</blockquote>
</aside>
<p>I wonder if your work can continue to use Ecto Changesets? The changeset integration with Phoenix is strong. I’m interested in hearing more about it!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368980" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/40">Post #39</a>
	                </div>
	            </div>
              <div id="likers-container-368980" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368980"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #39"></div>
  </section>
</div>
    <div class="postbit" id="368986" data-post-id="368986">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="39" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>With Ecto we do this all the time, and since the User gets its id <em>at insert time</em> with e.g. Postgres we can just return it. But with FDB of course this is not so easy.</p>
</blockquote>
</aside>
<p>This is still possible with EctoFDB, just not with Versionstamps. Using <code>:binary_id</code> works the same as traditional Ecto.</p>
<p>The async is only there to match libfdb_c’s futures, allowing the caller to only wait on network when they deem necessary.</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">Repo.transaction(fn -&gt;
  f1 = Repo.async_get_by!(User, name: "Alice")
  f2 = Repo.async_get_by!(User, name: "Bob")
  [alice, bob] = Repo.await([f1, f2])

  Repo.insert!(%Team{members: [alice.id, bob.id]})
end,
prefix: tenant)
</code></pre>
<p>Functionally equivalent, but now with 2 network waits:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">Repo.transaction(fn -&gt;
  alice = Repo.get_by!(User, name: "Alice")
  bob = Repo.get_by!(User, name: "Bob")

  Repo.insert!(%Team{members: [alice.id, bob.id]})
end,
prefix: tenant)
</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368986" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/41">Post #40</a>
	                </div>
	            </div>
              <div id="likers-container-368986" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368986"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #40"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/61642/load_more?page=5">Load more posts (36 remaining)</a>
</div></template></turbo-stream>