<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="355117" data-post-id="355117">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="MarthinL" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  MarthinL
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="41" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>You may get a better reply but briefly, the purpose of that first <code>update/2</code> clause is to receive the updated <code>entry</code> from <code>send_update/2</code>. In the example the main LiveView receives an updated <code>entry</code> over PubSub and dispatches it to the relevant LiveComponent.</p>
</blockquote>
</aside>
<p>First time I asked I read your response to mean that it’s exactly that first clause that’s the key but that makes the difference between only the parent, only the children or both being updated, simply because that what I asked about - what in the example makes that difference and that where you pointed. But it can only make that difference if some of the time it gets called and does something other than the default. I understood the load_children function to be a (PubSub-less) simulation of of a chance that impacts the children without the parent and I was busy trying to figure out how the reverse would be made possible - the parent getting updated without impacting the children what was the primary concern at the time.</p>
<aside class="quote no-group" data-username="garrison" data-post="41" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>m wondering - how do you plan to render the “informational content” which you are storing separately?</p>
</blockquote>
</aside>
<p>I’ve been compiling a chart complete example of my own to answer that question if I’m able to pull it off in the first place.  That’s exactly why I’ve been engaging to intently with the example and how to does its bussiness and how ran into the befuddling fact that the code I was as lead to believe is the key to the solution.</p>
<p>Not yet being sure how the tools actually work means it might yet find what doesn’t to do impossible.</p>
<p>However, principle is this. (Obviously) for the first (mount)? It will be a full normal render. The tree component will build the structure into which the content is rendered, still fundamentally recursive following the structure element’s recursion. At each level of recursion the children are drawn in from the linear list and rendered into place where they end up getting placed as inner content slots within the parent. Nothing about that is unusual or outside the norm. The only fact that clashes with the assumptions LiveView makes is that the resulting dom is recursively nested rather an a repetition of non-overlapping dom elements.</p>
<p>The differences only come into play when updates are getting involved.  I’m still trying to identify and leverage the exact opportunities available to this with, but the general principle is to implement a full set of tree operations at the LiveComponent level. I’m hoping to do most of that in the single whole-tree LiveComponent but it will most likely require that a per-node LiveComponent cooperates in a prescribed manner, much like implementing a behaviour. This would all be towards enabling tree operations to optimise by either preventing or (selectively) absorbing unnecessary changes getting picked up by diffing.</p>
<p>The complete set of tree operations have been largely known theoretically since Knuth’s time, but to make them real enough to program one only needs to look at the things we can do with file systems and their directory structures. It’s a great analogy of a tree and how tree information (directory structure) even though directories are often also special files, are managed differently from the files. The difficulty with the DOM is that in filesystem terms the contents of the files and the directory structure is all in one tree. Our challenge is to create a way to surgically manipulate the DOM remotely anyway.</p>
<p>It was in the context of my attempts to show this working rather than just talking about it that I got caught up in the cognitive dissonance of how Steffen’s example actually works. But that’s now waiting for an answer (sorry, but yours didn’t help me at all) so am forced to describe rather than show.</p>
<p>The overarching concept though is based on the premise that even the most complex, contrived and unpredictable ways a tree and its contents can get manipulated by incoming data or by user interaction can be detected as and reduced to a sequence of primitive changes along the lines of</p>
<ul>
<li>saving a new file in a directory</li>
<li>saving to an existing file</li>
<li>moving a file from one directory to another</li>
<li>moving an entire directory from one parent to another</li>
<li>renaming a file or directory</li>
<li>deleting a file</li>
<li>deleting a directory and everything below it</li>
<li>change attributes which affect access and visibility rights which impacts what is shown in a listing and the order they appear in</li>
<li>Walk through a directory structure from a starting point with a possible depth limit to look for and/or take actions on each file or directory found</li>
<li>Work out how much space is used by what optionally summarised or detailed per item.</li>
</ul>
<p>It is vitally important to note that the tree operations I describe above can be applied on two levels - the big underlying tree or the visual presentation of that tree for one or more users. None of what I am describing here is aimed at manipulating the underlying tree. That part is for you and me to implement however suits our data and our users best. What I am talking about here is to take the things we or our users do with the data and programmatically (largely declaratively) map those actions on what needs to happen to the visually represented trees derived from that data and the changes to it.</p>
<p>In other words, I am <strong>not</strong> suggesting that we merely detect the require changes in any way similar to what LiveView does with diffing and try to retrofit the deltas to the tree operations. The aim would be to take the actions right there where they are implemented for us or our users to change the underlying tree with and propagate those in tree primitive form (via PubSub) as what the various LiveView component instances currently presenting that data somewhere in a tree for a user must make happen to correctly and efficiently reflect the changes to the displayed DOMs.</p>
<p>If the trees were small enough it would been feasible to keep updating a whole tree at a time which is what diffing ends up doing be default. By limiting the loaded window and taking the bulk of the content (file content) out of the “directory structure” leaving only ids (disk node numbers of files) we’re making the tree small enough to let LiveView do the heavy lifting. What remains is to use our knowledge of the structure of the data, what it will look like in the DOM and what we know is meant to be happening to filter and augment what diff picks up and turn that into the minimal updates that needs to go to wire.</p>
<p>Update: I eventually found where the first update clause gets involved. I expected it to play a role in the “Load Children” test case while it actually plays its part with the “Rename bar.txt” test case. The reason my changes broke the “Load children” case is not related to that clause.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="355117" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/42">Post #41</a>
	                </div>
	            </div>
              <div id="likers-container-355117" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="355117"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #41"></div>
  </section>
</div>
    <div class="postbit" id="355120" data-post-id="355120">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="MarthinL" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  MarthinL
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="MarthinL" data-post="42" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/3da27b/48.png" class="avatar"> MarthinL:</div>
<blockquote>
<p>Update: I eventually found where the first update clause gets involved. I expected it to play a role in the “Load Children” test case while it actually plays its part with the “Rename bar.txt” test case. The reason my changes broke the “Load children” case is not related to that clause.</p>
</blockquote>
</aside>
<p>See above.</p>
<p>While I’m adjusting my mental models to accommodate this realisation I’m faced with yet another uncertainty I shall post about soon.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="355120" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/43">Post #42</a>
	                </div>
	            </div>
              <div id="likers-container-355120" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="355120"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #42"></div>
  </section>
</div>
    <div class="postbit" id="355176" data-post-id="355176">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="MarthinL" data-post="42" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/3da27b/48.png" class="avatar"> MarthinL:</div>
<blockquote>
<p>But that’s now waiting for an answer (sorry, but yours didn’t help me at all)</p>
</blockquote>
</aside>
<p>Since you are still waiting I will try to give you some more background.</p>
<p>As mentioned in Steffen’s post, there are two things which trigger the <code>update/2</code> callback. The first is a re-render from the parent, which will call it with whatever assigns are passed to the <code>&lt;.live_component /&gt;</code> invocation in the parent.</p>
<p>The second is the function <code>send_update/3</code>, which can be used to send some values (<code>assigns</code>) <em>directly</em> to a LiveComponent. It’s conceptually similar to message passing, but it’s not <em>actually</em> message passing because, as you might recall, LiveComponents exist within the same process as the parent. <a href="https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#send_update/3" rel="noopener nofollow ugc">Docs here</a>.</p>
<p>When you invoke <code>send_update(MyLiveComponent, id: "some-id", key: "value")</code> then the component <code>MyLiveComponent</code> with id <code>some-id</code> on the page will receive <code>update(%{key: "value"}, socket)</code>.</p>
<p>(I will point out for the record that this really isn’t Phoenix’s most intuitive API.)</p>
<p>In Steffen’s example, this <code>send_update/3</code> functionality is used to rename an entry, as I think you have since discovered.</p>
<p>There is a third way for a LiveComponent to be re-rendered: receiving an event (like <code>phx-click</code>). But unlike the first two, <strong>this does not call <code>update/2</code></strong>. Instead it invokes the <code>handle_event/3</code> callback, which updates the <code>assigns</code> accordingly (and then the component is rendered).</p>
<p>In the example, it is the <em>event</em> functionality which dynamically loads more children, which is why <code>update/2</code> is not called in that case. If for some reason you wanted to trigger a reload of the children via, say, PubSub, you could create an <code>update/2</code> case for that functionality and use <code>send_update/3</code> to trigger it.</p>
<p>So back to the two <code>update/2</code> clauses in the example. The first one, as we discussed earlier, pattern matches on an existing socket <em>with an <code>entry</code> already present</em>. What this means, in effect, is that this first clause will <em>only</em> be invoked if the <code>entry</code> has already been added to the LiveComponent. In other words, it will <em>only</em> be invoked <em>after</em> the first render.</p>
<p>Conversely, the <em>second</em> <code>update/2</code> clause will <em>only</em> be invoked <em>on the first render</em>.</p>
<p>Now it’s very important to understand that there is a trick being used here: streams. Each LiveComponent in the example <em>streams</em> its children, and the child of a stream is <em>only</em> rendered <em>immediately after</em> a <code>stream_*</code> call (like <code>stream/4</code> or <code>stream_insert/4</code>).</p>
<p>And so that is how we arrive at the behavior you have observed: as it turns out, nothing in Steffen’s example actually calls <code>stream/4</code> on an entry’s children again after the initial render. And so nothing in this example <em>ever</em> triggers <code>update/2</code> after the first render, <strong>unless</strong> you use <code>send_update/3</code> (the rename operation).</p>
<p>That is why you have not observed the first clause being invoked.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="355176" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/44">Post #43</a>
	                </div>
	            </div>
              <div id="likers-container-355176" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="355176"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #43"></div>
  </section>
</div>
    <div class="postbit" id="355194" data-post-id="355194">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="MarthinL" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  MarthinL
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="44" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>Conversely, the <em>second</em> <code>update/2</code> clause will <em>only</em> be invoked <em>on the first render</em>.</p>
</blockquote>
</aside>
<p>I’ve figure most of what you’re saying out, but this conflicts with what my debug statements showed. It might be because times it went via clause <span class="hashtag-raw">#2</span> was “technically” first renders of new elements, but when I definitely saw the second clause matching after the initial render.</p>
<p>I’ve been doing some pattern matching of my own to try instrument tree operations as I’ve described, and in the process found a section in the LiveView manual (around send_update I think) that describes conditions whereby the assigns get merged. So I went looking for the additional assigns I make in send_update in the assigns parameter rather than the socket’s assign member where I was looking for it because the :entry assign was there. So there’re definitely some fluidity and non-intuitive conditions going on behind the scenes, but once you find the stuff you’ve been sending across, it looks quite doable to implement the tree primitives I wrote about. I managed to get the first one, add_child (given as a map, adding to a parent given as an id) to work like a charm. In the current implementation I’m essentially using the :children stream to pass the children to be added as “parameters” for the tree operation, but I suspect I’ll eventually change that. However, as it is it became quite useful that the streams get “cleared” after being used on the server. It is making a lot of sense to me now to think of a stream as a virtual construct on which list-like operations are implemented but it only lives in actual (partial) lists for short periods of time.</p>
<p>Anyway, thanks for you help, it does help. I’ve been distracted today generating big enough arbitrary test data of the kind we use in this discussion so I can post that as part of my suggested solution in the hope that it becomes rather obvious when the tree operations are working optimally and when they’re triggering big updates. With small data the difference is too small to notice without deep inspection into hard to reach places.</p>
<aside class="quote no-group" data-username="garrison" data-post="44" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>And so that is how we arrive at the behavior you have observed: as it turns out, nothing in Steffen’s example actually calls <code>stream/4</code> on an entry’s children again after the initial render. And so nothing in this example <em>ever</em> triggers <code>update/2</code> after the first render, <strong>unless</strong> you use <code>send_update/3</code> (the rename operation).</p>
</blockquote>
</aside>
<p>I gathered as much. The main point he was trying to make in response to how he understood my questions was to show how a parent would be updated without updating the children as well. So the rename bar.txt button and its’s send_update was more or less what the example was about, but the Load Children button which relies on the default mechanisms was what my first attempts at doing some changes of my own broke so horribly. That got me off course, but I think I’m back now.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="355194" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/45">Post #44</a>
	                </div>
	            </div>
              <div id="likers-container-355194" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="355194"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #44"></div>
  </section>
</div>
    <div class="postbit" id="355195" data-post-id="355195">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="MarthinL" data-post="45" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/3da27b/48.png" class="avatar"> MarthinL:</div>
<blockquote>
<p>but this conflicts with what my debug statements showed. It might be because times it went via clause <span class="hashtag-raw">#2</span> was “technically” first renders of new elements, but when I definitely saw the second clause matching after the initial render</p>
</blockquote>
</aside>
<p>The first render <em>for each component</em>, yes. When you load the additional children they are rendered for the first time, so they would invoke the second clause (because their <code>socket</code>s do not yet contain an <code>entry</code>).</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="355195" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/46">Post #45</a>
	                </div>
	            </div>
              <div id="likers-container-355195" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="355195"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #45"></div>
  </section>
</div>
    <div class="postbit" id="363685" data-post-id="363685">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="MarthinL" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  MarthinL
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><strong>Update 3:</strong><br>
Since my last post on this thread I’ve invested a great deal of energy to come up with a solution that maps the non-linear domain of indefinitely recursive data onto the (flat and) rectangular domain of browser windows and grids. It’s been quite a journey so far, with really exciting results, but it’s not over yet. The objective is to build something suitable for all three use-cases for this I have in my project while also being of potential use by others with similar use cases. The general idea is to leverage existing LiveView with Streams facilities as much as possible but in a suitably intelligent way which prevents the recursive nature of the data from breaking the bank on memory and/or network bandwidth utilisation.</p>
<p><strong>But then…</strong><br>
Having spent a lot of time building not only the beginnings of this library but also a way to generate non-repetitive test data that is structured in a way that makes it possible to visually confirm that the right data and changes to data is being reflected on the screen while also being able to control the size of the test database for different stages of development and testing, I will soon be at a point where I need to confirm that all the required updates and nothing but the required updates are being sent to the client. Because of the gist <a class="mention" href="/u/steffend" rel="nofollow">@steffend</a> posted I have a rough idea about controlling what detected changes are allowed to go through to the client but I’d need more complete coverage than that. That way only picked up a specific scenario to cull some of the the updates based on a privileged understanding of how the update code moves parameters from the socket to the assigns. I’m going to need a more complete view than that.</p>
<p><strong>Meanwhile, a question:</strong><br>
Where and how can I trace, log or debug what changes LiveView detects and which of those end up going out on the wire?</p>
<p>I’m reasonably comfortable that I’d be able to get a view on how it plays out in the PubSub domain, but I need direction as to how to get visibility of the LiveView changes and traffic. I’d hate to have to resort to WireShark etc for that. <a class="mention" href="/u/garrison" rel="nofollow">@garrison</a>, you mentioned somewhere how you were able to determine that the updates being sent through didn’t meet with your hopes and expectations when using streams and I presume you used the same method to verify that not using streams had the desired effect. Did you use external tools or were you able to do that in the code itself?</p>
<p><strong>I’m after:-</strong><br>
a) a way to see what changes liveview has detected in the assigns when processing an update, and<br>
b) a way to see (probably in the browser’s JS console) what updates had been received at the client.</p>
<p>P.S. I know of the existence of some LiveView dashboard but never explored what it can and can’t do. I’d be deligted if it handsomely covers part b) above.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="363685" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/47">Post #46</a>
	                </div>
	            </div>
              <div id="likers-container-363685" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="363685"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #46"></div>
  </section>
</div>
    <div class="postbit" id="363690" data-post-id="363690">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="MarthinL" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  MarthinL
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="MarthinL" data-post="47" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/3da27b/48.png" class="avatar"> MarthinL:</div>
<blockquote>
<p>Since my last post on this thread I’ve invested a great deal of energy to come up with a solution that maps the non-linear domain of indefinitely recursive data onto the (flat and) rectangular domain of browser windows and grids. It’s been quite a journey so far, with really exciting results, but it’s not over yet.</p>
</blockquote>
</aside>
<p>BTW, by far the most interesting and challenging problem I had to solve in this regard was a functional and efficient algorithm that retrieves exactly the data required for one rectangular viewport of the tree being displayed to use in paginating the tree in two dimensions. LiveView, Streams and most of HTML is built for one-dimensional lists (of records) where all the pagination required can be controlled by the limit and offset parameters which gets carried through from the LiveView contreoller via Ecto all the way to the database that implements those directly. None of which applies to hierarchical or recursive data. I’ve not run formal tests but I believe the solution I ended up with has a constant execution time that depends only on the size of the view window, not the size of the underlying database. That would be an improvement on native offset and limit queries which could slow down drmatically when run against large datasets. It’s a relatively complex algorithm involving multiple small read operations which are all contained within the database environment itself rather than going back and forth over the network between the database and web servers.</p>
<p>I’ve also managed to decouple the rendered content from the hierarchy. Simply put, none of the tree nodes that end up in the assigns or stream has any (pre)loaded child nodes. Each is rendered as their own HTML never overlapping the HTML of another node. Visually and behaviourally the content appears nested recursively as it should be, but there’s no actual nesting involved, massively reducing the complications of diffing content. Since the pagination code can then accurately track exactly which nodes should be added and removed from the client to render the new page as a change from the current and the order of nodes in DOM memory no longers matters (their IDs uniquely dictate where in the window their HTML is positioned) it theoretically opens the door to using streams once again without needing to complications associated with nested streams.</p>
<p>I’m pleased with what I’ve been able to put together so far. Fingers crosses it all works out as planned.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="363690" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/48">Post #47</a>
	                </div>
	            </div>
              <div id="likers-container-363690" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="363690"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #47"></div>
  </section>
</div>
    <div class="postbit" id="363700" data-post-id="363700">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="MarthinL" data-post="47" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/3da27b/48.png" class="avatar"> MarthinL:</div>
<blockquote>
<p>a way to see what changes liveview has detected in the assigns when processing an update, and</p>
</blockquote>
</aside>
<p>When you call <code>assign(socket, :key, value)</code> LiveView diffs the value by comparing it to the previous value (essesntially <code>value == prev_assigns[:key]</code>). If the value has changed LiveView adds the key to the <code>assigns.__changed__</code> map. You can look inside the <code>__changed__</code> map or call <code>Phoenix.Component.changed?/2</code> to see if a key has changed after an assign. You could also probably inspect it on render to see all the changes in that batch - I <em>think</em> that would work.</p>
<p>I don’t know if there are any fancier tools. I have seen some hype about <a href="https://github.com/software-mansion/live-debugger" rel="noopener nofollow ugc">this LiveDebugger tool</a> - I haven’t tried it and I don’t know exactly what it does, but it might be relevant.</p>
<aside class="quote no-group" data-username="MarthinL" data-post="47" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/3da27b/48.png" class="avatar"> MarthinL:</div>
<blockquote>
<p>a way to see (probably in the browser’s JS console) what updates had been received at the client</p>
</blockquote>
</aside>
<p>You can inspect the messages sent over the socket in your browser devtools. It’s going to be different per browser but generally you can click on the socket in the network tab and see the messages and their sizes going over the wire.</p>
<aside class="quote no-group" data-username="MarthinL" data-post="47" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/3da27b/48.png" class="avatar"> MarthinL:</div>
<blockquote>
<p>the updates being sent through didn’t meet with your hopes and expectations when using streams</p>
</blockquote>
</aside>
<p>Just to clarify a bit (it’s been a while), the problem I had with streams was that they made it harder to build the UIs I want - not performance problems. Streams <em>do</em> solve the performance problem caused by large comprehensions in a LiveView, they just make it really hard to build a complex/reactive UI because they constrain updates to a single node and don’t give access to the existing data. I have continued to have better results with LiveComponents.</p>
<p>There are cases where stream diffs will be larger than LiveComponent diffs, though. The stream has to send the whole object back over the wire while the LC can diff each attribute since it holds the previous attributes in memory. Whether this matters depends on how many attributes you have per object and how large they are. And it is of course a tradeoff between server memory and wire size.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="363700" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/49">Post #48</a>
	                </div>
	            </div>
              <div id="likers-container-363700" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="363700"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #48"></div>
  </section>
</div>
    <div class="postbit" id="363703" data-post-id="363703">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="MarthinL" data-post="48" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/3da27b/48.png" class="avatar"> MarthinL:</div>
<blockquote>
<p>LiveView, Streams and most of HTML is built for one-dimensional lists (of records) where all the pagination required can be controlled by the limit and offset parameters which gets carried through from the LiveView contreoller via Ecto all the way to the database that implements those directly.</p>
</blockquote>
</aside>
<p>I want to point out a small correction here because I think it’s relevant.</p>
<p>LiveView, the DOM, and UI in general virtually always take the shape of a tree. I’m not sure exactly why this is the case, but it usually works out that way. The DOM is (of course) a tree, UIs are trees of components, and so on. Even in 3D rendering there is always a render tree.</p>
<p>So your data, the DOM, and the UI are all tree-shaped. The problem you’re dealing with is that <em>pagination is not tree-shaped</em>, at least not in your case. I think you are aware of this since your next paragraph describes the problem quite well.</p>
<p>This problem space is actually very old - things like virtualizing a folder tree view in a file explorer, but also providing drag/drop functionality. Unfortunately I think a lot of the knowledge for how to do this might have been straight up lost as we transitioned to the web and gave up on building good interfaces. The only place I have ever seen this mentioned is <a href="https://acko.net/blog/react-the-missing-parts/" rel="noopener nofollow ugc">in this article</a> (the “Yeet” section). The solution presented there is probably not relevant to your use-case, though, as this is an article about client-side state management and your state is in the database.</p>
<p>I’m sure there are more resources on this topic out there somewhere. I’m going to have to spend some time really digging at some point.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="363703" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/50">Post #49</a>
	                </div>
	            </div>
              <div id="likers-container-363703" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="363703"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #49"></div>
  </section>
</div>
    <div class="postbit" id="363721" data-post-id="363721">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="MarthinL" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  MarthinL
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="50" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>LiveView, the DOM, and UI in general virtually always take the shape of a tree. I’m not sure exactly why this is the case, but it usually works out that way. The DOM is (of course) a tree, UIs are trees of components, and so on. Even in 3D rendering there is always a render tree.</p>
</blockquote>
</aside>
<p>Yes, I’m deeply aware that the HTML and the DOM’s internal structure is inherently hierarchical. Sometimes those hierarchies are essential for the correct structuring and layout of content but when your content is in itself also hierarchical or worse, recursive, such intertwined hierarchies can become overwhelmingly complex and frankly counter-productive.</p>
<aside class="quote no-group" data-username="garrison" data-post="50" data-topic="69023">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>This problem space is actually very old - things like virtualizing a folder tree view in a file explorer, but also providing drag/drop functionality. Unfortunately I think a lot of the knowledge for how to do this might have been straight up lost as we transitioned to the web and gave up on building good interfaces.</p>
</blockquote>
</aside>
<p>I mostly agree with you there but not entirely. One person’s “giving up on building good interfaces” is another’s “drive towards simpler interfaces”, but I do agree that the result of the anemic set of primitives in HTML (compared to e.g. Win32 or OSF/Motif) made it really hard to build truly rich interfaces on the web. Remember Microsoft’s ActiveX controls and Silverlight and even Adobe’s now infamous attempt to replace HTML and Browser technology with Flash/Plex and Acrobat? Misguided and self-serving as all those were, they were in my opinion symptomatic of exactly that issue.</p>
<p>I’m a massive fan of a special kind of simplicity which stand in stark contrast with the crap you get when you try to simplify things by ignoring the parts of the problem that makes it complex, and useful. Simplicity that’s the result of absorbing and encapsulating all of the natural complexities present in any real environment behind a singular, simple abstraction that discards none of the complexities but instead fully deals with it internally while presenting a clear and simple concept or view of it outwardly, that’s the kind of simplicity I like. It’s not often achieved in real life, and in most of the cases where it is achieved it slowly evolves through sheer need. The road HTML had been on could be seen as a journey towards that type of simpliocity driven by the needs of programmers to get more done with less pain, but it is a long way still from where it really encapsulates all the complexities involved in designing and building a good user experience. What is genuienely rare, is for such types of simple abstractions of complex domains to be purposfully designed and built with that as objective. It isn’t often attempted and succeeds even less often. It’s what I am trying to do here, but it’s far too early to tell if my attempt will hit the mark at all. I know it will work for me, but I don’t know yet if it will ever benefit anyone else. Only time will tell.</p>
<p>By the way. You mention folder tree views in file explorer (email clients and word processors also used them for outlining) in the positive light of how good the tools used to be. Funny thing is, when I started this project that was what I had in mind when I thought of a tree on a screen. It was only when I really got into trying to visualise what this algorithm I needed was meant to do that I realised what a poor representation of a tree those were. It’s horses for courses. All those trees were used for secondary content like an outline or folder view shown nxt to the main content window. Vertical space was not an issue because it could scroll, but horizontal space was at a preium because using too much impeded on the space available to the main content. The result was that the branch / folder name was shown above its content with the content indented slightly. The “better looking” trees had connecting lines (which had fallen away in more recent times). The trouble was that if you had more than a few files or sub-folders in a folder the name of the parent folder would scroll out of view before you reach the end of the list of subfolders. Context is everything, and instead of showing the context of what folder(s) are being shown, all that left on the screen is a load of lines and these days, not even that.</p>
<p>I was so used to that for of tree that I never even realised that it was at best a dead tree, i.e. it was shown lying on its side. When I switched to a tree that showed the root at the bottom and the leafs at the top, I could suddenly fit a whole lot more data onto the screen and visualise a lot bettter what I needed to do. But more than that it meant I could use the space that used to be just lines and write the parent node’s detail centered underneath the child nodes and keep it there regardless of how I scrolled (left and right) through the children. That’s when I realised I had no alternative but to adopt that same layout for at least one of my application’s trees where the tree content IS the main content window and not some extra information shown alongside it. One could call it a revolution in tree display technology, or at least 1/4 of a revolution - 90 degrees.  <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> It works especially well for me because in addition to the labels (like folder names) the content I need to display for each node is in itself a multi-line block of yet another instance of a tree display (my second use case for the library). The third use case is stranger still. That one’s derives from the fact that the data I display isn’t truly a tree but in some respects more like an inverted tree or network in that a node can have multiple “parents” or children in a different hierarchy. So if the user has navigated to some place in the tree or the “field of trees” they’ve walked into, they can choose an option to turn around and “look back” from there to see that other “tree”. I know that doesn’t make a great deal of sense at this point to anyone but myself, but it will become apparent in using the application.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="363721" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/using-streams-with-recursive-and-or-deeply-nested-schemas/69023/51">Post #50</a>
	                </div>
	            </div>
              <div id="likers-container-363721" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="363721"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #50"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/69023/load_more?page=6">Load more posts (24 remaining)</a>
</div></template></turbo-stream>