<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="380815" data-post-id="380815">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Ah, thanks for clarifying, apologies for my misunderstanding.</p>
<p>You’re right that the entire books+authors query would have to be re-run in a single transaction to get internal consistency. This query would be for all intents and purposes equivalent to a select+join, as far as I can tell, with some watches thrown in.</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">def my_query(tenant) do
  Repo.transactional(tenant, fn -&gt;
    [authors, books] = Repo.await([
        Repo.async_all(Author),
        Repo.async_all(Book)
      ])
    aw = SchemaMetadata.watch_changes(Repo, Author)
    bw = SchemaMetadata.watch_changes(Repo, Book)
    {authors, aw, books, bw}
  end)
end
</code></pre>
<p>Then more code to handle resolving the watch and run <code>my_query(tenant)</code> again.</p>
<p>EctoFDB.Sync’s functions don’t currently help in this case. The API hides the transaction behind a nice simple function call. And it promises to create all the correct watches for you, but only for certain query types.</p>
<p>However, I see no reason why the approach itself couldn’t be used with a little extra work from the developer.</p>
<p>Thanks for bringing this up, it’s important for me to recognize and call out the limitations of the sync features.</p>
<p>There is one thought in the back of my mind while I was writing this post: I suppose anytime one tries to create query conventions on these transactions, one risks creating a new query language (please no).</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="380815" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/72">Post #71</a>
	                </div>
	            </div>
              <div id="likers-container-380815" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="380815"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #71"></div>
  </section>
</div>
    <div class="postbit" id="381380" data-post-id="381380">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jam" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/120/36266_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jam
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Sync looks cool. Can you comment on the scalability of it? I don’t know anything about FDB watches <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="jstimps" data-post="68" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>The “magic” auto-updating is done via careful FDB watches and LiveView’s attach_hook.</p>
</blockquote>
</aside>
<p>Does this mean it works only with LiveView?</p>
<aside class="quote no-group" data-username="garrison" data-post="71" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>There are some who will suggest that you join in the database here. Ignore them, ngmi</p>
</blockquote>
</aside>
<p>Go on…</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381380" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/73">Post #72</a>
	                </div>
	            </div>
              <div id="likers-container-381380" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381380"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #72"></div>
  </section>
</div>
    <div class="postbit" id="381382" data-post-id="381382">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Thanks!</p>
<aside class="quote no-group" data-username="jam" data-post="73" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/48/36266_2.png" class="avatar"> jam:</div>
<blockquote>
<p>Can you comment on the scalability of it?</p>
</blockquote>
</aside>
<p>Not really, not yet at least. I don’t have experience with heavy watch usage in a production setting. I am using it in a tiny production project called LiveSecret (<a href="https://github.com/jessestimpson/livesecret" rel="noopener nofollow ugc">GitHub</a> | <a href="https://livesecret.link/" rel="noopener nofollow ugc">Production App</a>).</p>
<p>I can refer you to the FDB docs that discuss where they have set their default limits (weirdly located in the <a href="https://apple.github.io/foundationdb/api-python.html#watches" rel="noopener nofollow ugc">python client docs</a>).</p>
<blockquote>
<p>By default, each database connection can have no more than 10,000 watches that have not yet reported a change. When this number is exceeded, an attempt to create a watch will raise a <a href="https://apple.github.io/foundationdb/api-error-codes.html#developer-guide-error-codes" rel="noopener nofollow ugc">too_many_watches</a> exception. This limit can be changed using <a href="https://apple.github.io/foundationdb/api-python.html#fdb.Database.options.set_max_watches" rel="noopener nofollow ugc"><code>Database.options.set_max_watches()</code></a>. Because a watch outlives the transaction that creates it, any watch that is no longer needed should be cancelled by calling <a href="https://apple.github.io/foundationdb/api-python.html#fdb.Future.cancel" rel="noopener nofollow ugc"><code>Future.cancel()</code></a> on its returned future.</p>
</blockquote>
<aside class="quote no-group" data-username="jam" data-post="73" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/48/36266_2.png" class="avatar"> jam:</div>
<blockquote>
<p>Does this mean it works only with LiveView?</p>
</blockquote>
</aside>
<p>Sync was written with LiveView in mind, but it can be used without LiveView. It does require that your process state that get passed in is:</p>
<ol>
<li>a map</li>
<li>the tenant is stored in a <code>:private</code> map with key <code>:tenant</code></li>
<li>the assigns are stored in an <code>:assigns</code> map</li>
<li>Sync also stores it’s own internal data in a key <code>:ecto_fdb_sync_data</code></li>
</ol>
<p>If you’re ok with these requirements, you would simply have to call <a href="https://hexdocs.pm/ecto_foundationdb/EctoFoundationDB.Sync.html#handle_ready/4" rel="noopener nofollow ugc"><code>Sync.handle_ready/4</code></a> from your process’s <code>handle_info</code>, or equivalent. The documentation is written in a LiveView-focused manner, but there’s nothing LV specific here.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381382" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/74">Post #73</a>
	                </div>
	            </div>
              <div id="likers-container-381382" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381382"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #73"></div>
  </section>
</div>
    <div class="postbit" id="381385" data-post-id="381385">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jam" data-post="73" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/48/36266_2.png" class="avatar"> jam:</div>
<blockquote>
<p>Go on</p>
</blockquote>
</aside>
<p>I was trying very hard <em>not</em> to go on but you know I cannot resist.</p>
<p>Keep in mind we are talking about <em>sync</em> here, so we’re actually talking about incrementalizing the join. To perform an incremental join you have to <a href="https://github.com/jamii/dida/blob/main/docs/why.md#constraint-1---internally-consistent-results" rel="noopener nofollow ugc">maintain the join on the client</a>, so from the “database” perspective you are not going to get out of that. Of course you could abstract this away into a library, but it’s still there.</p>
<p>From the <em>application</em> perspective you also want access to the underlying collections (<code>books</code> and <code>authors</code>) because, in real apps, there is probably another widget on the page somewhere with a list of authors <em>and</em> a list of books <em>and</em> a list of authors with books and so on. This is just how real apps work; they often show multiple views of the data at once.</p>
<p>The “traditional” SQL backend approach here is to query the data multiple times up-front (<code>select * from authors</code>), (<code>select * from books</code>), (<code>select * from authors inner join books...</code>). But how do you incrementalize these queries? A change for a particular book comes out of the WAL <em>one time</em>, and you have to do the work to figure out which queries to update for that row.</p>
<p>If you spend enough time on this topic you will eventually come to the conclusion that what you actually want to do is just <code>select</code> the <em>entire database</em> and then query it locally, and we call that “local-first” (which does <em>not</em> necessarily mean offline, btw). The difference between theory and practice here is where you draw the lines of what “entire database” means.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381385" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/75">Post #74</a>
	                </div>
	            </div>
              <div id="likers-container-381385" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381385"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #74"></div>
  </section>
</div>
    <div class="postbit" id="381386" data-post-id="381386">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jam" data-post="73" data-topic="61642">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jam/48/36266_2.png" class="avatar"> jam:</div>
<blockquote>
<p>Can you comment on the scalability of it? I don’t know anything about FDB watches</p>
</blockquote>
</aside>
<p>I am just going to make an informed guess here because I am too lazy to read the “docs” (as if), but the watch feature is probably just a hash table of <code>key =&gt; watch</code> on each storage server that pushes an update to the client when the key is modified. Changes stream into the storage servers from the WAL so it would be easy to check them against the hash table while applying the WAL to the ptree.</p>
<p>There is an edge case where the read version of the watch is lower than the latest write at the time the server receives it, so you would handle that by checking the ptree and immediately resolving the watch if a write is present with <code>version &gt; read_version</code>. This also implies that watches must be subject to the mvcc window.</p>
<p>Assuming the above (which is the obvious design) they should scale horizontally. Idk how many watches a single storage server can take in practice, though. Sockets are not free.</p>
<p>The unfortunate bit is that when a watch resolves you must re-run the entire query from scratch, but Jesse has done the best he can here. FDB just doesn’t have tools to parse the WAL like e.g. Postgres does, so there is no good way to incrementalize queries. This is one of the reasons I started writing a new database, because even though FDB <em>could</em> easily add incremental support I really do not think Apple cares at all.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381386" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/76">Post #75</a>
	                </div>
	            </div>
              <div id="likers-container-381386" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381386"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #75"></div>
  </section>
</div>
    <div class="postbit" id="383568" data-post-id="383568">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>v0.7.0 is released. This is primarily a large maintenance release and a hopeful step toward v1. There are various changes detailed in the <a href="https://hexdocs.pm/ecto_foundationdb/changelog.html#v0-7-0-2026-02-16" rel="noopener nofollow ugc">changelog</a>.</p>
<p>From a project stability point of view, the most notable change is a internal refactoring of Query and Future to support this innocuous-sounding bugfix:</p>
<ul>
<li>A <code>:limit</code> in <code>Ecto.Query</code> will now work as expected when encountering objects split across multiple keys.</li>
</ul>
<p>The keystone to this refactor was embracing the iterator pattern (helped along by <a class="mention" href="/u/garrison" rel="nofollow">@garrison</a>’s <a href="https://corporate.fm/blog/iterators-on-iterators" rel="noopener nofollow ugc">Iterators on Iterators</a>). Previously, we had been trying to use Stream. However, we needed partial evaluation with later continuation to control the retrieval of data before crossing a transaction boundary. Once we tried with an iterator, everything clicked in place. And it’s trivial to create a Stream from an iterator, so we still get the beautiful Stream API when it’s safe for us to do so.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="383568" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-foundationdb-an-ecto-adapter-for-foundationdb/61642/77">Post #76</a>
	                </div>
	            </div>
              <div id="likers-container-383568" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="383568"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>