We’ll discuss this again once you’re able to see how I’ve handled that including the deterministic order. The order of children forms part of the structure part of the data, i.e. of how they are related and should be represented on screen relative to each other that is extracted from the recursive data but not referred to during the rendering of the data portion so it’s not subject to diff calculations at that level. Changes to the structure including order of children and where in the tree nodes should appear affects the structure data for which a relevant portion (rectangular region) is kept in memory per client session. That part is for getting live changes to currently displayed data as efficiently to the clients as possible but overall tracking of how the data in the database changes over time is not a matter for the UX but for the business logic that tracks how users change the data. It would be insane to try to detect what changes has occurred in a massive recursive database by comparing different time-based snapshots of the database, which is a vulgarisation of what you’re suggesting should be done. If your app’s business logic does not keep track of what changes it makes to the data that’s where your trouble starts. If it does track that, then it’s not complex at all to follow the associations in the data to bubble up changes or cascade them as required, as long as it is being done on the server with business logic and not attempted in UX code such as LiveView and JS. The key being to not “let go” of the event which initiates a change until all the consequential changes and updates had been dealt with. If you drop the ball your database and servers will have to work really hard to recover it, but if you keep it in hand basically all the operations you need to be doing can be done quickly with key-only access because you know where to look for things. It’s a bit of a mixture beween relational type thinking and event-driven thinking, but it’s well worth the effort to ride that edge.
MarthinL
garrison
The purpose of my reply was mainly to clarify what LiveView is natively capable of WRT diffing, in response to that tweet.
There are of course other ways you can accomplish these things, like what you’re doing.
It is insane to do this against the entire database, yes. But it is not insane to use this approach against a view of a portion of the database (which as I understand is what you’re working with anyway).
In general, the purpose of these declarative/reactive frameworks like React or LiveView is to allow people to build interfaces in such a way that it’s easy to localize and incrementalize the updates to the interface without going down the “bad path” (diffing the entire thing), but while still maintaining the mental model of diffing the entire thing.
Of course, the result of that careful design is that most people using React and similar frameworks don’t actually know how they work. A blessing and a curse.
As an analogy, database isolation often works a similar way: you are allowed to think of transactions as being isolated and serialized, even though they are actually concurrent over the same storage space!
MarthinL
Yes, but exclusively for a carefully controlled minimal subset of the structural portion of the data. That part only changes when users reorder children, move parts of the tree around, add or remove nodes of the tree. As long as every client session manages ensures that it only subscribes to changes to things that are actually represented on the browsers of the clients they’re representing it’s fairly trivial to get those from the business logic layer through to the sessions and for them to get it through to their clients’ screens.
Which brings a second aspect into play. The business logic should also have a fairly easy time of distinguishing between changes to the structure and changes to the content of the nodes. If only the structure changes then the HTML rendering for the content can be reused without a hassle on one condition - the nodes need to have been rendered flat in the DOM, i.e. none of the recursive hierarchy should be present in the HTML. I initially thought that wasn’t likely to be possible and that I’d have to keep track of the nesting as well in order to control the updates with surgical accuracy. But I discovered since that there are quite a few ways of keeping the HTML flat and therefore for the nodes, even though they represent deeply nested / recursive data, to have no overlap with each other. In fact, though nesting is a very common thing in HTML and DOM the bulk of its primitives are fully geared towards big sets of peer elements. Only a handful of the higher level constructs in HTML and CSS has any support for nesting at all, but it’s usually quite a mission to get it to work as you’d want even if you know in advance all the levels of nesting you need to handle. When you can’t know in advance it tends towards insanity. I was very relieved when I figured out how to map a indefinitely recursive dataset onto a stock standard flat structure like a table, list or grid, which is what I am using at this point. That’s right! I’m showing the hierarchy in a grid. Looks and works brilliantly. The parts that are order dependent as you mentioned as important are isolated into a small structure that calculates quickly and cheaply and takes almost no memory because it only deals with how the recursive data maps onto the flat substrate.
The nodes themselves, freed from overlapping at the UI level, becomes very easy to deal with as well. At the moment the prototype don’t even use any LiveView component for that because I’m still only showing a single label field, but in time I have the option of using either a function component or a LiveComponent to manage the contents. Any and all of them being super simple because it doesn’t have any burden from the recursive nature of the data to carry or represent. Each node is on its own little island, visually, unaware that to the user they appear in a hierarchy brought about and managed without their involvement.
I’m not entirely sure I can agree with your sentiment here. By my reckoning whether it’s mental or actual, diffing “the entire thing” is simply the wrong way to think about the problem and no UX tools like LiveView or React has a legitimate role to play in the solution of that level of problem. UX tools and frameworks should be used to represent data and the how people may manipulate it and that is it. Nothing else. Effecting the changes requested by users via the UX tools, handling the knock-on effects of changes and getting word out to any subscriber to data that had been touched in the process is at the heart of the business logic which I strongly believe should live on the server (what Phoenix calls context apps, I think) i.e. not in files that live in the …Web directory. Sure, while you’re still experimenting with different options you’d maybe do stuff in a controller to make it quick and easy to work in a single file that changes a lot and often. But as it becomes apparent what the operations you’re going to need to make work for the data to be managed properly, those would disappear from the controller logic into context apps where they have standardised access to the database and things like pub sub management can be dealt with without repeating yourself. That’s my view anyway.
I’ll take your word for it. Managing a complex recursive dataset is still in my view not a problem any UX toolset will do for you even if you understand perfectly how it works. They’re just the wrong type of tool for that job.
That’s an interesting analogy. I used to be a fully qualified Oracle (Tuning) DBA and was the “owner”
of many other databases for big companies over the years as well. Strange thing about transaction isolation is that it’s almost never properly implemented. I was once asked by a new employer to recommend which database I’d want to do the new system on and I went straight for Oracle. Because of the price that raised many eyebrows and I was of course asked to motivate and defend my recommendation as there are so many cheaper alternatives. It took me less than 15 minutes, most of which was spent firing up databases, to run the same few statements from different terminal sessions against each of the databases. Oracle has was the only one that produced the correct result, all the others failed in the same way, essentially not honouring transaction boundaries and allowing data that has not been committed yet to be seen by others who could make their own changes to the database using the changed data which stuck even when the original transaction was rolled back. It not only made my case but also explained issues they were facing in systems that didn’t use Oracle. I’ve not had license to run Oracle in more than a decade now, but I have respect for it. Your comment implying that transaction isolation is essentially a myth doesn’t really hold true. It just takes a really well designed and stable database to actually implement it as specified in the standards, but it really isn’t as fuzzy and unpredictable as most non-Oracle users tend to experience transaction isolation. But I get your point which I read as referring to using tools in their abstract form while trusting them to turn the abstract into reality in a predictable manner. I’m just saying especially when it gets to the bigger concerns it can be hard to find tools that do their abstract forms justice when it comes to practice.
garrison
I enjoyed your Oracle story 
But yes, to be clear, my point was that database transactions give the illusion of being serialized when they are actually concurrent. They do this while preserving correctness through proper concurrency control.
Of course there are some databases (many…) which are literally broken, but that’s not what I meant.