A positive step forward, thank you.
For context, I’m not looking at hundreds of thousands items but hundreds of millions per regional federation server cluster. Every user has theoretical access to a substantial portion of it, but not all at once.
It’s largely small records but they add up quickly, about a quarter the data tracks each user’s effective access rights while the remainder is again split down the middle between structural elements (theoretically graphs but in practice usually trees with the odd loop turning it into a sort of directed graph) and the displayed informational content (not implying the rest doesn’t carry information or doesn’t impact what user see).
I don’t merely wish to avoid keeping all that data in server memory, it would be physically impossible. I have no alternative than to load the data for each user in segments controlled to a reasonable size, still using a starting point but limiting levels rather than rows.
But reality is slightly less dramatic than the picture I paint. While it’s all accurate there are natural and practical limitations in play on several levels which drastically reduce the portion of data loaded from the database and presented to each user. That’s all taken care of by the aligning how and where I store the data with reality, being aware of and able to accommodate exceptions (intra-regional references) efficiently and presenting a user interface which essentially navigates each user through the vast expanses of data using a (comparatively) tiny window or cursor which contains only enough pre-loaded data to avoid excessive delays when drilling deeper or backing out along the path. It’s “classic” infinite scrolling but for recursive/hierarchical content.
Let’s agree for the moment that to stream or not to stream is not the question. The issue at hand is this: Although I’ve reduced the amount of data per user session the server and/or client handles at any time the size of the DOM updates being sent to the client still makes a material difference. I could argue that the worst case scenario, i.e. that for every tiny change the entire active dataset being re-rendered and sent to the client is acceptably small and “shouldn’t be a problem” which seems a fair assumption until you a) consider the impact of undetected runaway data caused by malicious users or by accident and b) consider the combined impact on the servers from a load and traffic perspective. Then it becomes imperative (as in essential and urgent) to implement the rendition of the data and its changes in whatever manner will ensure that as far as can be arranges only the parts that actually changed ever gets sent to the client. LiveView already bares an impressive portion of this responsibility with it’s diffing and tracking of what parts of what assigns determined which portion of the HTML so that using the assigned DOM ids the affected parts of the DOM may be patched.
Now we know that the (as you put it) canonical use-case of LiveView involves linear sets of independent records as obtained from (either pure or streamed) lists provided to the component/rendering function as assigns. To flatten a recursively preloaded Ecto schema into a linear set of records and back again into the nested structures upon use, is rapidly turning from a solvable to a solved problem for me. That takes care of the linear list part. At first glance it seems that by happenstance or foresight the way LiveView identify the parts of the DOM requiring updates create the opportunity to succinctly identify and propagate the patches to each node in the tree (given to LiveView as a list). Ostensibly, (and this could be crux of the matter) it matters not where in the DOM a node’s HTML is placed, it will patch that node’s HTML based on the ID with the re-generated HTML.
If the above were true in every sense it would have simplified matters a great deal. Still might, but it all rests on how LiveView updates the DOM.
Any assumption of independence between elements being given IDs as they are rendered in preparation for DOM updates could mean that when that portion of the DOM is updated it is replaced with the new HTML that just arrived. If that element/node however included other content for which there or don’t need to be newly generated HTML, will the new DOM tree be patched with the other content from the original DOM or will it be discarded? Between what I’ve seen happening myself, what is described in the documentation and presentations and your feedback about streams and their issues, I’m getting lots of mixed signals. I’ve been warned of and have witnessed containers’s existing content hanging about indefinitely after an update. One particular example of this I can remember involved table rows I incorrectly added as headings without the appropriate identifier (data-phx-id) being impossible to get rid of. On the other hand you’re mentioning that (neither standard assigns or stream assigns in) LiveView handles container updates correctly. That’s more than a little confusing. Perhaps (though unlikely) it all true and the diffing and DOM update code in Phoenix works or doesn’t work at random but more likely it is merely opinionated about when it can and should do what. Whether it pertains to bugs, missing functionality or documented caveats, it should be possible to grasp what LiveView would and would not be able to handle under what conditions and then see to it that those conditions are met when writing the code.
Best course of action I can think of right now is to consult with the expert, which will do its own reply so he doesn’t have to suffer through all of the above.
