Using streams with recursive and/or deeply nested schemas

You may get a better reply but briefly, the purpose of that first update/2 clause is to receive the updated entry from send_update/2. In the example the main LiveView receives an updated entry over PubSub and dispatches it to the relevant LiveComponent.

If you are not using that send_update/2 functionality then that update clause won’t be called because there is nothing to trigger a re-render of the components at all (they are all rendered as nested streams, once).

I’m wondering - how do you plan to render the “informational content” which you are storing separately?

First time I asked I read your response to mean that it’s exactly that first clause that’s the key but that makes the difference between only the parent, only the children or both being updated, simply because that what I asked about - what in the example makes that difference and that where you pointed. But it can only make that difference if some of the time it gets called and does something other than the default. I understood the load_children function to be a (PubSub-less) simulation of of a chance that impacts the children without the parent and I was busy trying to figure out how the reverse would be made possible - the parent getting updated without impacting the children what was the primary concern at the time.

I’ve been compiling a chart complete example of my own to answer that question if I’m able to pull it off in the first place. That’s exactly why I’ve been engaging to intently with the example and how to does its bussiness and how ran into the befuddling fact that the code I was as lead to believe is the key to the solution.

Not yet being sure how the tools actually work means it might yet find what doesn’t to do impossible.

However, principle is this. (Obviously) for the first (mount)? It will be a full normal render. The tree component will build the structure into which the content is rendered, still fundamentally recursive following the structure element’s recursion. At each level of recursion the children are drawn in from the linear list and rendered into place where they end up getting placed as inner content slots within the parent. Nothing about that is unusual or outside the norm. The only fact that clashes with the assumptions LiveView makes is that the resulting dom is recursively nested rather an a repetition of non-overlapping dom elements.

The differences only come into play when updates are getting involved. I’m still trying to identify and leverage the exact opportunities available to this with, but the general principle is to implement a full set of tree operations at the LiveComponent level. I’m hoping to do most of that in the single whole-tree LiveComponent but it will most likely require that a per-node LiveComponent cooperates in a prescribed manner, much like implementing a behaviour. This would all be towards enabling tree operations to optimise by either preventing or (selectively) absorbing unnecessary changes getting picked up by diffing.

The complete set of tree operations have been largely known theoretically since Knuth’s time, but to make them real enough to program one only needs to look at the things we can do with file systems and their directory structures. It’s a great analogy of a tree and how tree information (directory structure) even though directories are often also special files, are managed differently from the files. The difficulty with the DOM is that in filesystem terms the contents of the files and the directory structure is all in one tree. Our challenge is to create a way to surgically manipulate the DOM remotely anyway.

It was in the context of my attempts to show this working rather than just talking about it that I got caught up in the cognitive dissonance of how Steffen’s example actually works. But that’s now waiting for an answer (sorry, but yours didn’t help me at all) so am forced to describe rather than show.

The overarching concept though is based on the premise that even the most complex, contrived and unpredictable ways a tree and its contents can get manipulated by incoming data or by user interaction can be detected as and reduced to a sequence of primitive changes along the lines of

  • saving a new file in a directory
  • saving to an existing file
  • moving a file from one directory to another
  • moving an entire directory from one parent to another
  • renaming a file or directory
  • deleting a file
  • deleting a directory and everything below it
  • change attributes which affect access and visibility rights which impacts what is shown in a listing and the order they appear in
  • Walk through a directory structure from a starting point with a possible depth limit to look for and/or take actions on each file or directory found
  • Work out how much space is used by what optionally summarised or detailed per item.

It is vitally important to note that the tree operations I describe above can be applied on two levels - the big underlying tree or the visual presentation of that tree for one or more users. None of what I am describing here is aimed at manipulating the underlying tree. That part is for you and me to implement however suits our data and our users best. What I am talking about here is to take the things we or our users do with the data and programmatically (largely declaratively) map those actions on what needs to happen to the visually represented trees derived from that data and the changes to it.

In other words, I am not suggesting that we merely detect the require changes in any way similar to what LiveView does with diffing and try to retrofit the deltas to the tree operations. The aim would be to take the actions right there where they are implemented for us or our users to change the underlying tree with and propagate those in tree primitive form (via PubSub) as what the various LiveView component instances currently presenting that data somewhere in a tree for a user must make happen to correctly and efficiently reflect the changes to the displayed DOMs.

If the trees were small enough it would been feasible to keep updating a whole tree at a time which is what diffing ends up doing be default. By limiting the loaded window and taking the bulk of the content (file content) out of the “directory structure” leaving only ids (disk node numbers of files) we’re making the tree small enough to let LiveView do the heavy lifting. What remains is to use our knowledge of the structure of the data, what it will look like in the DOM and what we know is meant to be happening to filter and augment what diff picks up and turn that into the minimal updates that needs to go to wire.

Update: I eventually found where the first update clause gets involved. I expected it to play a role in the “Load Children” test case while it actually plays its part with the “Rename bar.txt” test case. The reason my changes broke the “Load children” case is not related to that clause.

See above.

While I’m adjusting my mental models to accommodate this realisation I’m faced with yet another uncertainty I shall post about soon.

Since you are still waiting I will try to give you some more background.

As mentioned in Steffen’s post, there are two things which trigger the update/2 callback. The first is a re-render from the parent, which will call it with whatever assigns are passed to the <.live_component /> invocation in the parent.

The second is the function send_update/3, which can be used to send some values (assigns) directly to a LiveComponent. It’s conceptually similar to message passing, but it’s not actually message passing because, as you might recall, LiveComponents exist within the same process as the parent. Docs here.

When you invoke send_update(MyLiveComponent, id: "some-id", key: "value") then the component MyLiveComponent with id some-id on the page will receive update(%{key: "value"}, socket).

(I will point out for the record that this really isn’t Phoenix’s most intuitive API.)

In Steffen’s example, this send_update/3 functionality is used to rename an entry, as I think you have since discovered.

There is a third way for a LiveComponent to be re-rendered: receiving an event (like phx-click). But unlike the first two, this does not call update/2. Instead it invokes the handle_event/3 callback, which updates the assigns accordingly (and then the component is rendered).

In the example, it is the event functionality which dynamically loads more children, which is why update/2 is not called in that case. If for some reason you wanted to trigger a reload of the children via, say, PubSub, you could create an update/2 case for that functionality and use send_update/3 to trigger it.

So back to the two update/2 clauses in the example. The first one, as we discussed earlier, pattern matches on an existing socket with an entry already present. What this means, in effect, is that this first clause will only be invoked if the entry has already been added to the LiveComponent. In other words, it will only be invoked after the first render.

Conversely, the second update/2 clause will only be invoked on the first render.

Now it’s very important to understand that there is a trick being used here: streams. Each LiveComponent in the example streams its children, and the child of a stream is only rendered immediately after a stream_* call (like stream/4 or stream_insert/4).

And so that is how we arrive at the behavior you have observed: as it turns out, nothing in Steffen’s example actually calls stream/4 on an entry’s children again after the initial render. And so nothing in this example ever triggers update/2 after the first render, unless you use send_update/3 (the rename operation).

That is why you have not observed the first clause being invoked.

I’ve figure most of what you’re saying out, but this conflicts with what my debug statements showed. It might be because times it went via clause #2 was “technically” first renders of new elements, but when I definitely saw the second clause matching after the initial render.

I’ve been doing some pattern matching of my own to try instrument tree operations as I’ve described, and in the process found a section in the LiveView manual (around send_update I think) that describes conditions whereby the assigns get merged. So I went looking for the additional assigns I make in send_update in the assigns parameter rather than the socket’s assign member where I was looking for it because the :entry assign was there. So there’re definitely some fluidity and non-intuitive conditions going on behind the scenes, but once you find the stuff you’ve been sending across, it looks quite doable to implement the tree primitives I wrote about. I managed to get the first one, add_child (given as a map, adding to a parent given as an id) to work like a charm. In the current implementation I’m essentially using the :children stream to pass the children to be added as “parameters” for the tree operation, but I suspect I’ll eventually change that. However, as it is it became quite useful that the streams get “cleared” after being used on the server. It is making a lot of sense to me now to think of a stream as a virtual construct on which list-like operations are implemented but it only lives in actual (partial) lists for short periods of time.

Anyway, thanks for you help, it does help. I’ve been distracted today generating big enough arbitrary test data of the kind we use in this discussion so I can post that as part of my suggested solution in the hope that it becomes rather obvious when the tree operations are working optimally and when they’re triggering big updates. With small data the difference is too small to notice without deep inspection into hard to reach places.

I gathered as much. The main point he was trying to make in response to how he understood my questions was to show how a parent would be updated without updating the children as well. So the rename bar.txt button and its’s send_update was more or less what the example was about, but the Load Children button which relies on the default mechanisms was what my first attempts at doing some changes of my own broke so horribly. That got me off course, but I think I’m back now.

The first render for each component, yes. When you load the additional children they are rendered for the first time, so they would invoke the second clause (because their sockets do not yet contain an entry).