How can I get LiveView to tell morphdom to move a whole sub-tree to a different parent in the dom?

Phoenix Forum Questions / Help
, ,
1

Hi, there’s a whole long TL:DR going on about Recursive Trees in LiveView for context, but in the process of formulating a solution there is one thing that can be done to a tree that I’m struggling to figure out how it could map efficiently onto the wire (for component updates as well as via PubSub) and the dom operations it triggers client side.

As the above discussion has started doing let’s stick to using file-system directory structure as commonly understood metaphor. In that vein, the operation I’m having trouble with would be similar to

mv /dir1/dir2/dir3 /dir1/dir15/dir3

meaning to reposition the dir3 folder and all of its contents from being a subdirectory of /dir1/dir2 to become a subdirectory of /dir1/dir15.

I don’t have any morphdom experience or prior exposure, and most certainly don’t understand how LiveView uses it, but from a naive observer’s perspective there does appear to be primitives in morphdom that could achieve this.

In the best case scenario, it would automatically be taken care of if I simply update the dir3 node as a child of dir15, but have no idea if the parent relationship is visible or addressible anywhere.

I also thought of creating a new empty temporary node at the destination and make that look like the source node through updates in the hope that diffing and morphdom will notice its the same content and move it but i run into the same problem as above with added undertainty if it’s even possible to change the id of a LiveView component after it’s rendered.

This may seem trivial, and perhaps it is or could be, but in deeply recursive trees the impact on the user would be comparible to miving a directory through the GUI (especially windows) or from the command line with the mv command. The latter being completed instanteneous since it’s really just a pointer change and no uswer interface needs to update, while the former tracks progress as it traverses through the structure to collect what copies to make, shows progress as it copies each file and once done it completely refreshes the display. I’d love to avoid that approach if possible, and I think the brower’s DOM itself is quite capable of displaying it post-change very quickly, but I need help figuring out a reliable high-levlel approach to “trick” LiveView into giving morpdom the correct instruction.

Even if it means dispatching a custom message to the client which directly instructs morphdom to apply the move directly to the DOM before makeing the equivalent update to the LiveView which will then look to morphdom as already done.

Or perhaps the resulting JS on the client would just brute-force change the id for the root of what consitutes the dir3 node to a temporary value so that when we insert a new child into dir15’s LiveComponent with that same ID morphdom would pick up that content and move it into the new node. Then we might change the id back eiher through the LiveComponent or directly.

Or, it strikes me now (this is an edit), I could do a send_update from the LiveCompoent of either dir2 or directly dir3 to the component for dir15 to insert the content as a child, and hope/trust/know/why/how liveview diffing and morphdom will pick it up.

I don’t know, I’m just throwing out ideas, and would absolutely love hearing from others more familiar with the environment about how this an be acheived. It’s not the end of the world if such a change results in a big rerender and update, as I believe it shouldn’t happen all that often and if it does it might not be a bad thing if it happens slowly so the user can follow what’s happening, but it would be good to have some idea beforeharnd about if and how it could be done.

P.S. Is the “parent reference” in nested Phoenix.Component and/or Phoenix.LiveComponent instances stored explicitly or is it effectlively only kept on the call stack as these components invoke each other?

2

When you’re using LiveComponents, they are assigned (by you!) a static id which uniquely identifies that component on the page. If you take a look at the LiveComponent docs and scroll down a bit you will see the following:

Two live components with the same module and ID are treated as the same component, regardless of where they are on the page. Therefore, if you change the location […] it won’t be remounted.

What this means is that LiveView is smart enough to send down a minimal diff notifying the client that a LiveComponent has been moved to a different place in the DOM, so the client can move that entire subtree wholesale without the whole thing being sent over the wire again.

The above does not apply to normal components, which would be sent again in full.

I do want to point out that you show a lot of concern for morphdom in your post (in the title, even), but your main concern should be the diffing that happens in LiveView before the diffs are sent over the wire. The morphdom part is really just a “final step” that you shouldn’t concern yourself with too much, as it’s plenty fast unless you are passing in truly huge diffs (in which case you will always run into wire problems first anyway).

1 Like
3

Now there is something else to note here, and I’m keeping this in a separate comment because it’s pulling in context which I have from our previous discussions on this topic.

I’ve only ever done this in the “declarative” way (without streams) as we’ve discussed. When I render my trees and a subtree has been moved, I reload the entire tree (only the structural parts, as I’ve mentioned) and then render it back out declaratively into the LiveComponents. This means that the entire tree state transition is, for me, handled by the LiveComponent diffs. Because each node has a static (database) id, the node by its very nature disappears from its old parent and appears in its new parent, and then the LiveView diff engine is just smart enough to notice and send a small diff.

However, in your case you are working with Steffen’s example using streams of LiveComponents, meaning you have to actually instruct the old parent to delete the node (stream_delete) and then insert it into the new parent (stream_insert) and so on. As we’ve discussed (at length), this is not so simple.

But there’s something else to keep in mind: if you’re triggering these deletes and inserts via send_update/3 to various LiveComponent nodes, I’m not entirely sure what guarantees LiveView makes with respect to when those patches are actually made.

If you imagine a case where a delete went through, a diff was sent to the client, and then an insert went through, the old node would have been deleted before the new one was inserted, and therefore the state needed to do a “small” diff would be lost on the client.

This sort of weirdness is in fact why I was very surprised to learn that it’s possible to use LiveComponents inside streams at all, and I don’t understand the code deeply enough to know what will happen here (though I should note that Steffen most certainly does understand it, as a maintainer - but Steffen’s example did not allow for nodes to be moved).

I am curious to see what will be sent over the wire if you actually try this, as I’m not sure what to expect.

4

That’s because I’m (gradually) wrapping my head around what LiveView is doing server-side and starting to believe I can get it to manage the data whether kept in session memory or in streams (a virtually infinite list-ike construct of which sometimes none is actually in memory) to leverage its capabilities and avoid its shortcomings. I am starting to see how to map the conceptual tree operations onto LiveView (functions and components, their rspective mounts. callbacks like update and render and PubSub-compatible data for send_update and general process send calls). I’m sure forming a map of the landscape but there’s a big blank area on there marked “HBD” (for Here Be Dragons) and that’s around the morphdom side of things.

The DOM is unavoidably a tree in itself, so however a recursive tree maps onto DOM elements must be mappable to equivalent DOM operations. The embuggerence is that in between my recursive tree and the DOM’s recursive tree there’s two layers of software to that does not explicitly cater for recursive nesting. It caters for nesting, yes, but especially from your experiences some of the assumptions made by at least treams appears to break it’s efficiencies when it comes to recursive data. I am merely trying fill in the blanks in my understanding so I can design the behaviour I need rather than arbitrarily getting it to work on the LV level only to discover that it all gets lost in translation between LV and the DOM.

Just a little further down in the same documentation you pointed out, there is an interesting and encouraging observation about nesting and using breadth-first algorithms. It’s noteworthy because I (for one) have assumed it makes no difference and used depth-first traversal throughout. I doubt I’ve written even one breadth-first algorithm in 10 years. It’s still vague and broad strokes but I can start seeing how that would make a difference to diffing and dom operations. In the documentation it is stated that using the breadth-first approach is what facilitates nesting. Can’t profess to fully understanding the how and wherefore of that but I hope to get there soon.

So far, my grasp of it amounts to (also based on that section of the documentation) the notion that provided you instruct (using the LiveComponent update mechanism) LiveView when to leave the children alone, clear them or move them with the node, the mere fact that each node is rendered where in the (DOM and LV) tree you tell it to render, that’s where the node will be rendered. I’ll test it of course, and it corresponds well with the list of morphdom options, but it seems plausible. If all that checks out, it would explain why Steffen, who has a much deeper and more functional understanding of these things, straight-away knew that to update a parent without updating the children can be achieved by no-opping the children updates at the update level. Until you understand the consequences that’s something I would not have considered safe, but seeing it done helped me to better discriminate between the three different views of the data.

5

I know you’re not super familiar with frontend, but morphdom is essentially the “diff” part of React. All it does is take the current DOM and a copy of what you want the DOM to be and then makes changes to the DOM elements until they match. The reason React was designed to work this way is that, as we have discussed at length, declarative rendering is far superior from a developer experience perspective. So React’s diffing engine, and likewise morphdom, are designed to take an “example” of what you want the DOM to look like and make it so.

For the record, the actual algorithm used for this isn’t very complicated and it’s something you can implement pretty easily yourself (as a learning exercise). Here is a good article on the topic.

There are essentially two reasons to do this instead of just replacing the old DOM with the new DOM wholesale: First, it’s faster. Second, if you replace the whole DOM you might end up clobbering elements with implicit state (like form inputs).

6

Yeah, thanks, I read the documentation and saw more or less what it does. How it does it, though interesting, isn’t quite my concern at the moment. The real question was and remains about how to arrange/hack/trick/code/set it up so that LiveView and its supporting javascript code calls morphdom in such a way as to move a node from one place in the dom to another, children and all.

I have this one clue I’m working towards testing but got myself tangled up trying to master breadth-first algorithms the Elixir way. Seems like it might become important to have that in the toolbox or at least know how to spot it when I encounter it.

Before that journey down the rabbit hole I was pretty well on my way to creating a infinite virtual datasource for testing, i.e. a tree which just keeps delivering more data the deeper into it you navigate into which you only see a few levels at a time. for interest sake, a tree 64 levels deep and where each node has 64 children it has 6.5e113 nodes in it, that many googols of nodes and quite impossible to hold in memory, but the idea would be that you could navigate into any part of this tree without its virtual size becoming an issue.

The clue I hope to get back to soon is that there’s a chance that LV can in fact handle nested elements. Well, we know it can, but the question is if it can handle recursive nesting just as easily. The crux of the matter appears to be using a LiveComponent for each recursive “node” even though that might comprise many nested components and elements in its own right by ensuring that a unique and reliable dom-id is involved.

I came across an article but didn’t read it, which talks about streams as a source of truth, or if my quick scan is accurate, more in a “it’s a really bad idea but if you absolutely have to do it, here’s what you’d have to do” kind of way. Our lengthy debates about streams left me with half a suspicion that your woes with streams was a consequence of inadvertently leaning a little too far towards considering streams as a source of truth when every about it (being a stream) screams that it’s fluid. It carries data but it doesn’t hold on to it, ever. Used right, I still think they will be very useful and not broken at all. To use them correctly means to understand that there’s two sides to it - on the one end you must have a stable and reliable source of truth, like your database from which to source the exact records you need as you need them, with minimal waste and overhead. At the other end of the stream lives a set of DOM elements which holds a materialised view of those records that’s been deemed to be required on the client, i.e. what fits on the screen with a buffer above and below for scrolling. That loaded subset is controlled and managed by the server, based on user interactions and updates via PubSub but directly in relation to the source data, turning the relevant changes per client into update commands for which it supplies the relevant records in streams. It’s my current understanding that though the same stream name and state is used the content of the stream is always limited to only the records that needs updating and no others. On the client end of the stream, the javascript with morphdom acts like a post-office, basically taking the instructions received and routing the instruction and the relevant record(s) to the dom subtree living at the given address. The DOM is updated in-place. The DOM ends up with more, less and changed representations of data in memory, but at no point ever, does anyone, not the server, not the client, deal with the entire list of records. If they do, it’s because it’s a small list that does not extends beyond the window size, or the programmer didn’t implement pagination, but that’s not what streams are designed to do so even when they occasionally have all the records “in the stream” you best assume they’re not there anymore.

7

The part of the client that moves LiveComponents around in the way you desire is not morphdom - I believe it happens before morphdom is invoked. Here is a great article by Jose that might clear some things up for you:

8

We have discussed this before but I just want to emphasize: the problem you’re running in to is not a problem with recursive data. The actual issue is quite simple: LiveView does not properly diff :for comprehensions (lists) unless you use LiveComponents as the children of those lists. A tree view, by its nature, includes lists (of children for each node), and so you will run into problems with very large diffs unless you wrap the nodes in LiveComponents.

Recursive data is modeled just fine by the DOM and by LiveView because they both naturally form a tree structure.

I’m not entirely sure what you mean by “source of truth” here, but of course I am fully aware that streams don’t hold onto their state. That’s why, as I have repeated many times, they are not very good for building complex interfaces - because you have to hold onto the state in order to properly re-render a complex interface.

Obviously if your use case is very simple (like most streams examples) then you’ll probably be fine. I don’t dispute that.

9

Now that’s what I was talking about! That really is a great article explaining almost exactly the meat of what I was wondering about which isn’t how morphdom works but what are the right buttons to push to get LiveView to do things the most optimal way and the example I used was to move a large subtree on screen without rerendering it. With all that José shared in that article, I am confident that I can navigate my way around LiveView and my data.

What was also reaffirmed for me, is how important it is to have an effective source of truth for the data you present. It’s not only about the size of the data in memory, but having tracking changes as close to the source as possible. IF the LiveComponents involved are fed the actual changes that occurred they can do what they need to do to make those show up at the client. I great example of that is to know that a node moved or changed rahter than implementing it (as I’ve seen recommendations for) sending those as a delete and insert pair. Those are not equivalent, and even if LiveView can save the say and pick up many cases where that happens, it is still better to not put that on it and rather work off the facts.

Anyway, I’m out on my feet after a mind-breaking marathon over two days trying to wrap my rutted brain around breadth-first Elixr algorithms when my thinking had defaulted to depth-first for the logest time. Unfortunately (for me) the examples I found online about that lead me completely down the wrong path - they all used a queue (borrowed from Erlang’s :queue) to achieve the breadth first behaviour, so I presumed that it was a) hard, b) unnatural to Elixir and c) going to require that I do something similar. I must have written and discarded several thousand lines of code only to get nowhere. It felt like an intractible problem for two days. Then I woke up with an alternative idea and though it still took several hours of adjustment and even more restarts, I finally wrote code that constructed a tree in a breadth-first manner. It’s of no specific use to do it that way, it was just to wrap my head around it, but in the end the whole thing was less than 20 lines of rather straight forward code, and it ran in a fraction of the time my depth-first version had run.

I’ve now also written a tree generator that keeps the structure separate (in a nested list of tuples and lists) and a map for the label records. Next I’ve going to turn that generator into my infinite virtual tree that can scroll/navigate/zoom in any direction ad infinitum. yet keep a finite portion of the structure and only the referenced nodes in memory.

As an aside, it came to my attention that maps might well be streamed. I used to operate under the assumption that if you put a map on a stream that it would effectively be one entry but that might be untrue. The map of content nodes involved in any tree operation I wish to hand over to LiveView to handle would therefor make a great stream.

1 Like
10

I’m glad to hear you found the article helpful. It is indeed a very good explanation - Jose has a gift :slight_smile:

This is what I’ve been trying to tell you! As long as you update your tree and pass it to the LiveComponents they will diff it efficiently.

The requirement to worry about imperative operations (like inserts and deletes) comes from streams. Which is why you should avoid them unless absolutely necessary for performance.

11

The database is usually the source of truth, the master copy of the data that settles any disputes or confusion in what is held in memory with absolute authority.`

We famously disagree on the implication and interpretation of this sentiment, not its accuracy per se. I do not dispute that you have to hold onto the state in order to properly render a complex interface, and I never even suggested that it’s not the case. But what I am saying is that if you do it cleverly enough and don’t only rely on the magic of the platform you use, you can get a very long way with holding very small portions of the data as state. It is also my current opinion that once you do that (hold minimal byt essential state) then streams becomes your friend as a means of passing data via LiveView to the clients leveraging all the optimisations from Jose’s post. But I’ve been wronng before and have gigantic scars to prove it and no shame about making mistakes. I’m happy to continue on the streams bandwagon until I either tame them or understand why they cannot be tamed after all. Don’t break yourself over it, it’s my choice.

12

I hate the phrase, but I’m forced to use it. Let’s agree to disagree, at least for now. It’s a bone of contention in an otherwise very constructive conversation. I respond to what I understand not what I’ve been told. If streams are the wrong tool I really need to fully comprehend in terms of my own frame of reference why that is and under what conditions they are useful. That’s knowledge I can use, not just to avoid a painful bout or two in the short term, but for the rest of my life.

13

Right, the database is the source of truth for your data in “normalized” form. But the purpose of your application is to provide useful “denormalized” views of it.

For example, your tree might be stored in the DB as a list of rows, but your application code transforms that into a tree structure, and perhaps also provides pagination and so on.

In this sense, your application is the “source of truth” for the denormalized form of the data. The transformations may be done by SQL, or your own code, or likely both. For example, with a tree, you write a recursive CTE to grab the correct rows and then you write an Elixir function to build a tree structure.

The question is whether you want that “denormalized” state to be stored in the LiveView, or only in the client’s DOM. The latter is where you would use streams, and so in that case I suppose you could consider streams the “source of truth”.

Obviously I don’t think that’s a very good idea, because it means you have to know in advance which parts of the state you want to update. But for simple stuff it can work.

Streams can make use of some of the optimizations in Jose’s post (like sharing statics), but remember: because they don’t store state on the server, they also can’t diff the dynamics at all - any diffing optimizations are out the window.

When you do a stream_insert the whole item you render as a result of that has to be sent over the wire - even if only a single field actually changed. Depending on your use case this might be better or worse.

If your items have very few fields, then the inserts are very efficient. If your items have many fields and you update few of them, your inserts are very inefficient, and using LiveComponents to diff the fields would be better.

Note that you can technically use LiveComponents inside streams, which is weird to me but it can be done. I would imagine this results in the LiveComponents’ states being stored on the server anyway, which I would think defeats the purpose, but I could be missing something.

14

Definitely don’t take anything I say as a reason not to learn or experiment! Of course I encourage you to try things for yourself and see how you feel about them.

But if you do start to run into some of the problems I’ve described (and I think you have run into a couple already), hopefully my replies can at least help shed some light onto where things can go wrong.

15

There’s a piece of core wisdom that Academia and the School of Hard Knocks wholeheartedly agree upon yet is quite common knowledge and (for reasons I can wax quite cynical about so I won’t) doesn’t feature much on the Youtube School of Programming that goes like this: Changing the normalisation of data invariably requires procedural logic which meant to say it cannot be done in SQL. Some variants and techniques in modern SQL (like CTE, PIVOT etc) appear to have overcome that basic principle, but those mechanisms are merely procedural logic disquised as SQL and are almost always uni-directions.

If your premise is that the job of your software is to usefully present normalised data in denormalised form you’re trapped in a world of hurt, or as you’ve put it, intractible complexity. It’s easy to spot the design flaw in in a model that associates People with Companies and Companies with Addresses (so as to serve the gods of normalisation) because in no time you’ll be showing your users a denormalised version whereby each Person is shown to have an Address. Sounds good until they want to change someone’s address, but when they try they also change the addresses of everyone of the same company. It might be what the author thought would be a powerful feature but it conflicts with the users’ perspectives because the underlying model does not reflect their reality accurately enough. Like I said, it’s easy to spot (and correct) this mistake when we have sufficient domain knowledge to put into the data model, but as the domain grows more complex, unknown or variable, it can become terribly hard to spot these mismatches between the modelled data and reality. The usual result being that there develops a set of things each system simply cannot be made to do or where doing them imposes such a heavily performance penalty that it can only be done offline or in special reports, not on the live data.

So no, the job of your program, and therefor yours, is not to present normalised data un useful denormalised forms but to find and expose the user to data so smartly normalised that it perfectly aligns with their reality.

Then, if you are going to tackle the rarified challenge of converting between normal forms on the fly, you’d better have a thorough appreciation of what that challenge entails and arrive at the party with a pretty solid understanding of how you will solve the difficulties and a steady stream of brilliant ideas on how to adapt your approach when what once seemed a good plan falls apart in the face of changes to how you understand the needs or the needs themselves. In short, you need to be able to recognise and solve the problems yourself (if need be) before you should try to get any platform, library or tool to solve them for you (failing which, needs be that you do it yourself).

My suggestion, if you are indeed faced with a situation where you are forced to present and store your data at two different levels of normalisation, one of the better approaches is to encapsulate that (unavoidably procedural) translation in a shared server process. Most likely the whole database won’t fit into memory, so you’d have to write a pretty clever server that retains only the key mappings between the normalised and denormalised data. What would be unwise is to duplicate (which is different from replicate) that process for each user session (or to hand that part to a library you hope would handle it correctly. Hope is not a strategy).

Ultimately, your users need your application to have the smarts to when they want to change what looks to them as someone’s address (or it’s far more subtle and complicated real-life counterparts) to ask if they want to change only that person’s address or that of the company they work for, and then facilitate the correct change through the shared server, to the database and propagate it out to affected parties from there.

This is, to a large extent, exactly aligned with that Phoenix and LiveView is doing, which is why I love it. But it can’t and won’t automatically solve all your problems for you. You still need to a) understand what you’re asking it to do, b) know what the solution entails, c) know which problems the tools can be made to solve, d) know how to map the solution onto the nomenclature(s) or the tool(s) you’re using, e) be able and willing to step in and fill the gaps between the solution you need and what the tools can do. Which is a fair summary of the process I am going through with LiveView at the moment. I’ve designed and implemented several systems in my career which made all the difference to the large organisations I did it for, and the principle was always to encapsulate as much as humanly possible of the actual reality of the domain you’re dealing with in the data model. That didn’t mean no procedural part, only that the procedural part we’re well-defined and supported by the data. Following all that impact I had for companies’ benefit I chose to write another, most meaningful one, for my own benefit which includes the benefit of every other human being. Which meant that I couldn’t hope to come up with a data model that matches each and every domain known or unknown, so for my enterprise work I developed some generic model principle which ended up working brilliantly. What I’m doing at the moment continues the evolution of those generic simplifications that worked into more complex adaptations that work, a.k.a. Gall’s Law.

Bottom line is that you do as you think is right, I’ll do the same. It’s not about who knows better or who’s right. It’s about implementing solutions to real world problems.

16

I think I agree with you here. The example I gave was just such a case: it is not even possible for SQL to return a tree structure - only rows!

But this is not a “denormalized” version of the data. This is a bug! A person’s company’s address is not the same thing as a person’s address.

If you were storing the person’s address in normalized form - a person_addresses table, with {person_id, address} tuples - then presenting this to the user as you described would be perfectly reasonable. In fact, I’m not sure how else you would do it?

Would you present an end user a list of {person_id, address} tuples? Surely not.

Trying to return to the topic of discussion here, what I was talking about was the following. If you have a tree structure in a relational db, it is probably stored like this:

%Node{id: 1, folder_id: 1, file_id: nil, parent_id: nil}
%Node{id: 2, folder_id: nil, file_id: 1, parent_id: 1}
%Node{id: 3, folder_id: nil, file_id: 2, parent_id: 1}

%Folder{id: 1, name: "elixir stuff"}

%File{id: 1, name: "elixir.jpg"}
%File{id: 2, name: "jose.png"}

But when you present this file tree to the user, you would not render a set of three tables. You would render a tree! So you would first transform your data to this:

%{
  name: "elixir stuff",
  children: [
    %{name: "elixir.jpg"},
    %{name: "jose.png"},
  ],
}

To perform this “transformation”, between the normalized rows in the DB and the “denormalized” (there is probably a better term) data in your application, you would use a combination of SQL and Elixir code. I only mentioned this because I was trying to respond to what you said about streams being a “source of truth”, and why I agreed that was not a great idea.

17

Of course, and when I quotes the example I I mentioned that it’s am obvious mistake in that setting, but when it’s something deep down and subtle the obviousness disappear and it’s no longer so easy identify who or what is to blame for the nagging mismatch between that users want and what they have.

We’ve clearly been talking way past each other on this. There are a lot of terms for it by I usually talk about the data model or and when i\ve referred to it in the Phoenix context I’ve called it associated data because Ecto calls it associations. In pure SQL domains you’d here talk of relations and joins but it all boils down to the same thing. I’ve not gone counting but I estimate me app currently uses around 90 of these one way or another. But there is nothing denormalised about them at all. Normalisation is a definite but often abused concept from relational databases and set theory which your normalise has nothing to do with. In fact, what you’re hopefully looking at is hopefully highly accuractely normalised data which is what makes the data so complex. The essential characteristic as far as the database being a source of truth is concerned, is that is direct or one-to-one mapping between every piece of data you present to the user and its represenetation in the database. When that’s not the case anymore, you’re mangling the data which is where the world of hurt comes into play. The other Ecto term used for that are preloaded data - where preloading is a way to automatically (declaritively) construct these in-memory structures from the data. I’ve had to deal with a few ORMs in my life and from that perspective how Ecto does it is brilliant. It does an excellent job of maintaining tracking the metadata of exactly how the data in memory correlates to the data in the dataset to tje point that you can point at any loaded record and simply instruct Ecto to build a changeset for it for you and save changes to it to the database. If you had actual denormalised data, that would not be possible. But like I said, your job as designer and programmer is to ensure that the structure of the data in the database and on the screen with all the layers of abstraction in between matches the user’s problem domain exactly.

That also sets the scene for me to reiterate what I’ve said about the structure of my data since you probably missed it based on the unfamiliar terms I used. I have data that’s fundamentally recursive - meaning that the same constellations of associated records repeats, but between those recursions points there may be a great many other records who’s data adds up to fully display a node. My objective is to split the actual recursive data, in essence just the records involvrd in the parent-child relationship’s id’s, in a separate structure as the cheap and efficient version of the tree that I can afford to keep more of in memory to help me guide LiveView towards making only the changes that are required without having to keep big data in memory for it to do the comparisons itself.

Just like it’s known and understood between us that I’m not a front-end fundi, I think it’s becoming fair comment to say while your frame of reference is skewed towards the front-end you tend to solve problems there rather than at the back-end where your skill-set is less pronounced.

1 Like
18

This comment of yours just caught my attention (tab left open). I’ve always agreed with you that for a tree you can afford to keep in server memory that will work. My entire dilemma stems from having a tree that won’t even fit once in memory so forget about keeping the version each user is seeing (lots of shared content but each user tracks a personalised take on it and has vastly different access based on complex conditionals) so the tree materialises differently. I do intend to keep a portion of the big tree as realised for each user in memory, but I’m looking at ways to reduce the memory footprint per user even further.

I understand that with all the data displayed for a user visible to LiveView it will do a fine job of detecting and optimising what can be done. I also understand that using streams for some data could limit it’s abilities and force LV to brute force some things because it doesn’t have the data for comparison to optimise it. My premise is that if I take something away from LV I must compensate for that in another way. LV is fantastic to figure out differences automatically when it has all the data and it would be foolish of me to try replicate rather than simply use that. But I have opportunities LV doesn’t have - I understand my backwards, I know every change that occurs right where it happens and can calculate the ripples it causes, often with relative ease but sometimes it can have a lot of ripples affecting almost everything and everyone, in which case though still simple in principle, I know in advance when a change is made to the data what the impact of the change is going to be, and make an informed choice about how to go about pushing those updates.

Let’s say for argument’s sake that I have a certain type of data which determines access rights to all data for all users, and these recur in countless places in the data. If a node like that changes, it could affect any number of users, from 0 to everyone. The trick is that I can tell what will be affected and if it’s more than a threshold value I am comfortable with (at the time) I have the option of bypassing all the cascading changes and trigger the reload for all affected users that it was eventually going to result in anyway. That’s what I mean with my understanding of my data giving me an advantage over LiveView that has to detect what’s happening and require the old and new data in order to do so. That example is somewhat extreme, but the same principles also apply at a lower level of recursion because at the core of it, I (my code) knows the impact of every change and follow simple rules about what can be handed off to LV to figure out on its own because it has the relevant data, when to relieve LV from trying to figure it out and when to arrange for LV to have the key bits of data to do the right things with.

It’s that last part where I still maintain a glimmer of hope that streams would be a key enabler for me in that it will allow me to hand LV the relevant (sub)set of records it would need to consider. At the moment it looks quite doable by keeping a fraction of the tree’s structural part in LV memory, understanding when it will detect a hash value change for a node and making sure that it has access to the records it will require to rerender that part of the tree effectively.

I’ve got a lot to do still, but I’m getting there. It might seem like I’m overcomplicating things and should just let LV do what LV is designed to do. But I’m not overcomplicating anything. Instead, I am taking complications that already exist in spades on board and am trying to extract simplicity from it again. That’s different from making simplifying assumptions (like ignoring the effect of friction for this exercise) and more like the opposite. In the field of Cybernetics - the study of complexity of which I am a student, the term simplexity has been used to describe “simplicity after complexity” meaning to encapsulate high complexity inside simpler wrappers which still fully represent the now hidden complexity but has become easier to deal with as a separate concern with known interactions with the rest of the problem. In computing this is roughly equivalent to breaking down a problem into smaller problems. Unfortunately in world of computing the concept had been over-used and watered down until it lost the key requirement of encapsulation with the result that all too often the re-combination of the sub-solutions has cascading complexity instead of cascading capability with steady or decreasing simplicity.

I love Erlang, Elixir, Phoenix and LiveView because not only was it created by a set of people with an inate (or acquired) understanding of encapsulation but also because as a result it responds well to efforts to keep doing more of the same on top of it, which is what I am attempting.

19

Of course I am not advocating for keeping your entire database in-memory for each user (this would be absurd). I want to be very clear that almost everything I’ve written on this topic thus far has been about what to do after you’ve queried only the parts of the tree that you want to display to the user. How you do that is too domain-specific for me to even comment on, and I’m sure you have a handle on it anyway!

But once you have the “small” part of the tree, there are two paths: you can store it on the server and let LiveView diff it, or you can store it on the client via streams. If you choose to do the latter, it is up to you to make sure you update the correct parts of the state when they change. As I’ve said many times, if you have a very tight (constrained) use case this is tractable. If you have a very complex UI, it’s not. Only you can make that call, I’ve simply tried to give some perspective.

Just be careful to make sure you fully understand how LiveView diffs those nodes. I recommend spending some time looking at the diffs sent over the websocket in your browser dev tools if you haven’t already, and experimenting with ways to shrink them (LiveComponents, streams, etc).

One thing I will note: the “hashing” optimizations are related to conditional branching, it’s a bit different from how the dynamics are diffed. When you call assign() LiveView will track whether the value you assign is actually different, and if so it essentially marks all usage of that value as dirty so that those dynamics will be re-sent.

If you are using streams the above is different; there is no diffing, because there is nothing to diff since the previous data is obviously not even present. Instead you are relying on the fact that you only perform a stream_insert() call if something has changed, and you are telling LiveView to re-send that particular stream item.

1 Like
20

This is a digression, but it’s funny because I remember Google going to great lengths to deal with this exact problem in one of the Zanzibar papers. They had some elaborate caching scheme - I don’t remember exactly how it worked.