How can I get LiveView to tell morphdom to move a whole sub-tree to a different parent in the dom?

Hi, there’s a whole long TL:DR going on about Recursive Trees in LiveView for context, but in the process of formulating a solution there is one thing that can be done to a tree that I’m struggling to figure out how it could map efficiently onto the wire (for component updates as well as via PubSub) and the dom operations it triggers client side.

As the above discussion has started doing let’s stick to using file-system directory structure as commonly understood metaphor. In that vein, the operation I’m having trouble with would be similar to

mv /dir1/dir2/dir3 /dir1/dir15/dir3

meaning to reposition the dir3 folder and all of its contents from being a subdirectory of /dir1/dir2 to become a subdirectory of /dir1/dir15.

I don’t have any morphdom experience or prior exposure, and most certainly don’t understand how LiveView uses it, but from a naive observer’s perspective there does appear to be primitives in morphdom that could achieve this.

In the best case scenario, it would automatically be taken care of if I simply update the dir3 node as a child of dir15, but have no idea if the parent relationship is visible or addressible anywhere.

I also thought of creating a new empty temporary node at the destination and make that look like the source node through updates in the hope that diffing and morphdom will notice its the same content and move it but i run into the same problem as above with added undertainty if it’s even possible to change the id of a LiveView component after it’s rendered.

This may seem trivial, and perhaps it is or could be, but in deeply recursive trees the impact on the user would be comparible to miving a directory through the GUI (especially windows) or from the command line with the mv command. The latter being completed instanteneous since it’s really just a pointer change and no uswer interface needs to update, while the former tracks progress as it traverses through the structure to collect what copies to make, shows progress as it copies each file and once done it completely refreshes the display. I’d love to avoid that approach if possible, and I think the brower’s DOM itself is quite capable of displaying it post-change very quickly, but I need help figuring out a reliable high-levlel approach to “trick” LiveView into giving morpdom the correct instruction.

Even if it means dispatching a custom message to the client which directly instructs morphdom to apply the move directly to the DOM before makeing the equivalent update to the LiveView which will then look to morphdom as already done.

Or perhaps the resulting JS on the client would just brute-force change the id for the root of what consitutes the dir3 node to a temporary value so that when we insert a new child into dir15’s LiveComponent with that same ID morphdom would pick up that content and move it into the new node. Then we might change the id back eiher through the LiveComponent or directly.

Or, it strikes me now (this is an edit), I could do a send_update from the LiveCompoent of either dir2 or directly dir3 to the component for dir15 to insert the content as a child, and hope/trust/know/why/how liveview diffing and morphdom will pick it up.

I don’t know, I’m just throwing out ideas, and would absolutely love hearing from others more familiar with the environment about how this an be acheived. It’s not the end of the world if such a change results in a big rerender and update, as I believe it shouldn’t happen all that often and if it does it might not be a bad thing if it happens slowly so the user can follow what’s happening, but it would be good to have some idea beforeharnd about if and how it could be done.

P.S. Is the “parent reference” in nested Phoenix.Component and/or Phoenix.LiveComponent instances stored explicitly or is it effectlively only kept on the call stack as these components invoke each other?

When you’re using LiveComponents, they are assigned (by you!) a static id which uniquely identifies that component on the page. If you take a look at the LiveComponent docs and scroll down a bit you will see the following:

Two live components with the same module and ID are treated as the same component, regardless of where they are on the page. Therefore, if you change the location […] it won’t be remounted.

What this means is that LiveView is smart enough to send down a minimal diff notifying the client that a LiveComponent has been moved to a different place in the DOM, so the client can move that entire subtree wholesale without the whole thing being sent over the wire again.

The above does not apply to normal components, which would be sent again in full.

I do want to point out that you show a lot of concern for morphdom in your post (in the title, even), but your main concern should be the diffing that happens in LiveView before the diffs are sent over the wire. The morphdom part is really just a “final step” that you shouldn’t concern yourself with too much, as it’s plenty fast unless you are passing in truly huge diffs (in which case you will always run into wire problems first anyway).

1 Like

Now there is something else to note here, and I’m keeping this in a separate comment because it’s pulling in context which I have from our previous discussions on this topic.

I’ve only ever done this in the “declarative” way (without streams) as we’ve discussed. When I render my trees and a subtree has been moved, I reload the entire tree (only the structural parts, as I’ve mentioned) and then render it back out declaratively into the LiveComponents. This means that the entire tree state transition is, for me, handled by the LiveComponent diffs. Because each node has a static (database) id, the node by its very nature disappears from its old parent and appears in its new parent, and then the LiveView diff engine is just smart enough to notice and send a small diff.

However, in your case you are working with Steffen’s example using streams of LiveComponents, meaning you have to actually instruct the old parent to delete the node (stream_delete) and then insert it into the new parent (stream_insert) and so on. As we’ve discussed (at length), this is not so simple.

But there’s something else to keep in mind: if you’re triggering these deletes and inserts via send_update/3 to various LiveComponent nodes, I’m not entirely sure what guarantees LiveView makes with respect to when those patches are actually made.

If you imagine a case where a delete went through, a diff was sent to the client, and then an insert went through, the old node would have been deleted before the new one was inserted, and therefore the state needed to do a “small” diff would be lost on the client.

This sort of weirdness is in fact why I was very surprised to learn that it’s possible to use LiveComponents inside streams at all, and I don’t understand the code deeply enough to know what will happen here (though I should note that Steffen most certainly does understand it, as a maintainer - but Steffen’s example did not allow for nodes to be moved).

I am curious to see what will be sent over the wire if you actually try this, as I’m not sure what to expect.

That’s because I’m (gradually) wrapping my head around what LiveView is doing server-side and starting to believe I can get it to manage the data whether kept in session memory or in streams (a virtually infinite list-ike construct of which sometimes none is actually in memory) to leverage its capabilities and avoid its shortcomings. I am starting to see how to map the conceptual tree operations onto LiveView (functions and components, their rspective mounts. callbacks like update and render and PubSub-compatible data for send_update and general process send calls). I’m sure forming a map of the landscape but there’s a big blank area on there marked “HBD” (for Here Be Dragons) and that’s around the morphdom side of things.

The DOM is unavoidably a tree in itself, so however a recursive tree maps onto DOM elements must be mappable to equivalent DOM operations. The embuggerence is that in between my recursive tree and the DOM’s recursive tree there’s two layers of software to that does not explicitly cater for recursive nesting. It caters for nesting, yes, but especially from your experiences some of the assumptions made by at least treams appears to break it’s efficiencies when it comes to recursive data. I am merely trying fill in the blanks in my understanding so I can design the behaviour I need rather than arbitrarily getting it to work on the LV level only to discover that it all gets lost in translation between LV and the DOM.

Just a little further down in the same documentation you pointed out, there is an interesting and encouraging observation about nesting and using breadth-first algorithms. It’s noteworthy because I (for one) have assumed it makes no difference and used depth-first traversal throughout. I doubt I’ve written even one breadth-first algorithm in 10 years. It’s still vague and broad strokes but I can start seeing how that would make a difference to diffing and dom operations. In the documentation it is stated that using the breadth-first approach is what facilitates nesting. Can’t profess to fully understanding the how and wherefore of that but I hope to get there soon.

So far, my grasp of it amounts to (also based on that section of the documentation) the notion that provided you instruct (using the LiveComponent update mechanism) LiveView when to leave the children alone, clear them or move them with the node, the mere fact that each node is rendered where in the (DOM and LV) tree you tell it to render, that’s where the node will be rendered. I’ll test it of course, and it corresponds well with the list of morphdom options, but it seems plausible. If all that checks out, it would explain why Steffen, who has a much deeper and more functional understanding of these things, straight-away knew that to update a parent without updating the children can be achieved by no-opping the children updates at the update level. Until you understand the consequences that’s something I would not have considered safe, but seeing it done helped me to better discriminate between the three different views of the data.

I know you’re not super familiar with frontend, but morphdom is essentially the “diff” part of React. All it does is take the current DOM and a copy of what you want the DOM to be and then makes changes to the DOM elements until they match. The reason React was designed to work this way is that, as we have discussed at length, declarative rendering is far superior from a developer experience perspective. So React’s diffing engine, and likewise morphdom, are designed to take an “example” of what you want the DOM to look like and make it so.

For the record, the actual algorithm used for this isn’t very complicated and it’s something you can implement pretty easily yourself (as a learning exercise). Here is a good article on the topic.

There are essentially two reasons to do this instead of just replacing the old DOM with the new DOM wholesale: First, it’s faster. Second, if you replace the whole DOM you might end up clobbering elements with implicit state (like form inputs).

Yeah, thanks, I read the documentation and saw more or less what it does. How it does it, though interesting, isn’t quite my concern at the moment. The real question was and remains about how to arrange/hack/trick/code/set it up so that LiveView and its supporting javascript code calls morphdom in such a way as to move a node from one place in the dom to another, children and all.

I have this one clue I’m working towards testing but got myself tangled up trying to master breadth-first algorithms the Elixir way. Seems like it might become important to have that in the toolbox or at least know how to spot it when I encounter it.

Before that journey down the rabbit hole I was pretty well on my way to creating a infinite virtual datasource for testing, i.e. a tree which just keeps delivering more data the deeper into it you navigate into which you only see a few levels at a time. for interest sake, a tree 64 levels deep and where each node has 64 children it has 6.5e113 nodes in it, that many googols of nodes and quite impossible to hold in memory, but the idea would be that you could navigate into any part of this tree without its virtual size becoming an issue.

The clue I hope to get back to soon is that there’s a chance that LV can in fact handle nested elements. Well, we know it can, but the question is if it can handle recursive nesting just as easily. The crux of the matter appears to be using a LiveComponent for each recursive “node” even though that might comprise many nested components and elements in its own right by ensuring that a unique and reliable dom-id is involved.

I came across an article but didn’t read it, which talks about streams as a source of truth, or if my quick scan is accurate, more in a “it’s a really bad idea but if you absolutely have to do it, here’s what you’d have to do” kind of way. Our lengthy debates about streams left me with half a suspicion that your woes with streams was a consequence of inadvertently leaning a little too far towards considering streams as a source of truth when every about it (being a stream) screams that it’s fluid. It carries data but it doesn’t hold on to it, ever. Used right, I still think they will be very useful and not broken at all. To use them correctly means to understand that there’s two sides to it - on the one end you must have a stable and reliable source of truth, like your database from which to source the exact records you need as you need them, with minimal waste and overhead. At the other end of the stream lives a set of DOM elements which holds a materialised view of those records that’s been deemed to be required on the client, i.e. what fits on the screen with a buffer above and below for scrolling. That loaded subset is controlled and managed by the server, based on user interactions and updates via PubSub but directly in relation to the source data, turning the relevant changes per client into update commands for which it supplies the relevant records in streams. It’s my current understanding that though the same stream name and state is used the content of the stream is always limited to only the records that needs updating and no others. On the client end of the stream, the javascript with morphdom acts like a post-office, basically taking the instructions received and routing the instruction and the relevant record(s) to the dom subtree living at the given address. The DOM is updated in-place. The DOM ends up with more, less and changed representations of data in memory, but at no point ever, does anyone, not the server, not the client, deal with the entire list of records. If they do, it’s because it’s a small list that does not extends beyond the window size, or the programmer didn’t implement pagination, but that’s not what streams are designed to do so even when they occasionally have all the records “in the stream” you best assume they’re not there anymore.

The part of the client that moves LiveComponents around in the way you desire is not morphdom - I believe it happens before morphdom is invoked. Here is a great article by Jose that might clear some things up for you:

We have discussed this before but I just want to emphasize: the problem you’re running in to is not a problem with recursive data. The actual issue is quite simple: LiveView does not properly diff :for comprehensions (lists) unless you use LiveComponents as the children of those lists. A tree view, by its nature, includes lists (of children for each node), and so you will run into problems with very large diffs unless you wrap the nodes in LiveComponents.

Recursive data is modeled just fine by the DOM and by LiveView because they both naturally form a tree structure.

I’m not entirely sure what you mean by “source of truth” here, but of course I am fully aware that streams don’t hold onto their state. That’s why, as I have repeated many times, they are not very good for building complex interfaces - because you have to hold onto the state in order to properly re-render a complex interface.

Obviously if your use case is very simple (like most streams examples) then you’ll probably be fine. I don’t dispute that.

Now that’s what I was talking about! That really is a great article explaining almost exactly the meat of what I was wondering about which isn’t how morphdom works but what are the right buttons to push to get LiveView to do things the most optimal way and the example I used was to move a large subtree on screen without rerendering it. With all that José shared in that article, I am confident that I can navigate my way around LiveView and my data.

What was also reaffirmed for me, is how important it is to have an effective source of truth for the data you present. It’s not only about the size of the data in memory, but having tracking changes as close to the source as possible. IF the LiveComponents involved are fed the actual changes that occurred they can do what they need to do to make those show up at the client. I great example of that is to know that a node moved or changed rahter than implementing it (as I’ve seen recommendations for) sending those as a delete and insert pair. Those are not equivalent, and even if LiveView can save the say and pick up many cases where that happens, it is still better to not put that on it and rather work off the facts.

Anyway, I’m out on my feet after a mind-breaking marathon over two days trying to wrap my rutted brain around breadth-first Elixr algorithms when my thinking had defaulted to depth-first for the logest time. Unfortunately (for me) the examples I found online about that lead me completely down the wrong path - they all used a queue (borrowed from Erlang’s :queue) to achieve the breadth first behaviour, so I presumed that it was a) hard, b) unnatural to Elixir and c) going to require that I do something similar. I must have written and discarded several thousand lines of code only to get nowhere. It felt like an intractible problem for two days. Then I woke up with an alternative idea and though it still took several hours of adjustment and even more restarts, I finally wrote code that constructed a tree in a breadth-first manner. It’s of no specific use to do it that way, it was just to wrap my head around it, but in the end the whole thing was less than 20 lines of rather straight forward code, and it ran in a fraction of the time my depth-first version had run.

I’ve now also written a tree generator that keeps the structure separate (in a nested list of tuples and lists) and a map for the label records. Next I’ve going to turn that generator into my infinite virtual tree that can scroll/navigate/zoom in any direction ad infinitum. yet keep a finite portion of the structure and only the referenced nodes in memory.

As an aside, it came to my attention that maps might well be streamed. I used to operate under the assumption that if you put a map on a stream that it would effectively be one entry but that might be untrue. The map of content nodes involved in any tree operation I wish to hand over to LiveView to handle would therefor make a great stream.

1 Like

I’m glad to hear you found the article helpful. It is indeed a very good explanation - Jose has a gift :slight_smile:

This is what I’ve been trying to tell you! As long as you update your tree and pass it to the LiveComponents they will diff it efficiently.

The requirement to worry about imperative operations (like inserts and deletes) comes from streams. Which is why you should avoid them unless absolutely necessary for performance.

The database is usually the source of truth, the master copy of the data that settles any disputes or confusion in what is held in memory with absolute authority.`

We famously disagree on the implication and interpretation of this sentiment, not its accuracy per se. I do not dispute that you have to hold onto the state in order to properly render a complex interface, and I never even suggested that it’s not the case. But what I am saying is that if you do it cleverly enough and don’t only rely on the magic of the platform you use, you can get a very long way with holding very small portions of the data as state. It is also my current opinion that once you do that (hold minimal byt essential state) then streams becomes your friend as a means of passing data via LiveView to the clients leveraging all the optimisations from Jose’s post. But I’ve been wronng before and have gigantic scars to prove it and no shame about making mistakes. I’m happy to continue on the streams bandwagon until I either tame them or understand why they cannot be tamed after all. Don’t break yourself over it, it’s my choice.

I hate the phrase, but I’m forced to use it. Let’s agree to disagree, at least for now. It’s a bone of contention in an otherwise very constructive conversation. I respond to what I understand not what I’ve been told. If streams are the wrong tool I really need to fully comprehend in terms of my own frame of reference why that is and under what conditions they are useful. That’s knowledge I can use, not just to avoid a painful bout or two in the short term, but for the rest of my life.