Before I compose a reply aimed at this discussion in particular, I need to clarify once more that the criticisms of streams I laid out in the previous thread are not restricted to the recursive use-case. I provided a non-recursive example (see my comments about the hypothetical chat app, a canonical streams use-case) to illustrate this.

The reason that these issues manifest particularly strongly with recursive data is that recursive UIs are particularly complex, and all complex UIs built with streams (recursive or not) will suffer because imperative APIs make it difficult-to-impossible to reason about how changes in your state will affect the resulting render.

The reason I brought this up in the context of recursive data is that your previous thread is about recursive data, and so that was the context in which I was writing. Since it seems you have also made this thread about a similar topic, I’m sure it’s going to come up again, but just keep in mind that these issues are not limited to a recursive context.

For a general argument (which is not aimed at LiveView in particular), I will again cite this article:

This, right here, is where the streams approach will go off the rails. The problem is that streams are only capable of performing this mapping one-to-one, but for any nontrivial use case (like a recursive tree), the mapping from state to rendered UI is arbitrarily many-to-many. It is not possible to know in advance, for a nontrivial UI, which parts of the rendered UI need to change when a single struct or field is updated. You have to render the whole thing from scratch to find out.

A simple example is when a node is moved from one position in the tree to another. Hierarchical trees are stored in a relational DB using a parent_id field on each child node, which points back to another node in the same nodes table.

When a node is moved, a single parent_id field is updated. However, in the rendered UI, the situation is very different! We have to remove the node from its old parent and add the node to its new parent. Already we have broken the one-to-one mapping which, as you said, streams provide.

But the pain won’t stop there. First of all, you may end up having to handle removing and re-inserting all of the child nodes of the moved node. I think the streams APIs may happen to save you from some of this work (only because DOM nodes themselves happen to form trees).

Once you start adding more features (your tree probably actually does something, right?), the pain will increase exponentially. A real example from my work: I wanted the nodes to be collapsible, but I didn’t want to render the expand/collapse icon if the node was an empty folder.

Think about how trivial this feature is, and then think about how annoying it would be to implement with streams. You now have to write special-case code that updates the parent of a child when the child is the last child and it is removed from the parent (plus when a first child is added). This is just one tiny feature!

You are very wrong about this. It seems to me that you misunderstood my previous advice and have essentially constructed some sort of strawman to argue against instead (you did this on your last thread too). I can’t allow you to misrepresent my posts like this: what you’ve written here is entirely your own invention.

If you have any questions about what I’ve written you are welcome to ask and I am happy to answer. I am probably one of relatively few people in the world who has actually done the exact thing you are posting about (complex recursive UI in pure LiveView, no client rendering) and I’d be glad to discuss it further.

You should not expect more from it since that is exactly what it is for, just like for normal assigns including record sets in assigns, you provide a mapping between the data and its HTML, and that mapping is in that sense one-to-one.

For normal assigns and streams alike, the code performs the magic it’s been taught to perform which in this case means reflecting on the assigns using the metadata to do the diff and look for opportunities to minimise the updates resulting from changes by re-rendering only the changed parts and sending those DOM fragments to the client. But when it cannot isolate the changes it has to rerender the whole assign, list member or stream element. While that’s usually not too heavy a price to pay, really large assigns, list elements within assigns or stream elements such as what happens with recursive structures will mean that when the code reverts to rerendering the outer structure it has an explosive impact. So the impact is big and the chances of it happening increased because of the complexity of the data. Together that means complex data has a high risk of causing unwieldy updates. I seem to recall that both the documentation and the code when it can detect issues at compile time warns about several things in this regard.

It can for example only pick up on the strcuture of the streamed content if you reference the stream in the prescribred way inside your components.

As another example it warns (with explanations) to not manipulate assigns inside the heex template (but beforehand if you need to).

So yes, absolutely, true and justified, without additional metadata neither LiveView nor Streams can do a whole lot more than it is already doing. It simply hasn’t been taught the additional magic required. When we come with our recursive data, it is up to us to find a way to reduce what we give to tools to what they can handle, and that means it’s up to us to limit what we put in a stream strictly to things for which a suitable one-to-one mapping to HTML has been declared and stick to the rules that allows LiveView and Streams isolate independent changes for which it knows how to propage the changes. I’ve done some of that in my app and you’ve chosen to go about it a different way the actual details of which escapes me still.

Now my quest is to see if maybe Streams can be taught some of the additional magic I’ve been using so that it can handle recursion in stream data more efficiently.

Though it doesn’t qualify as a feature implementation (it’s simply something that has to be there) I happen to have implemented that behaviour recently and it was a complete non-event. In the heex template I render the node either as a (<a>-based) leaf item or as a (<details><summary><a/><ul/></details>-based) node.

Of course that by itself does nothing to address the issue of recursion, but I genuinely do not see where this complex body of special case code you keep referring to is meant to be inserted or what it looks like.

If I’m misunderstanding and/or misrepresenting you it’s without malice and purely because I cannot picture where and in what context or even language these special cases you’re talking about lives. I keep hearing you say that it lives in the frontend and you make frequent references to React, DOM manipulation and even jQuery as the old way to do it. To me that implies the code you’re talking about is hand-written Javascript (or using something like React) but either way running on the client and directly manipulating the DOM. The way I use LiveView and Streams I don’t have the foggiest of ideas how to manipulate the DOM and neither would I want to have one.

The only DOM manipulation in my world is done through what I can let Phoenix and LiveView do by providing them with the appropriate mappings. When I cannot provide such mappings the onus is on me to rearrange the server data so that I have data for which such one-to-one mapping can be defined. Based on what I read in the documentation, articles and this forum, that’s more or less what everyone does and keeps doing until the problem has been broken down into small enough sub-problems to pre-calculate the data in a form for which the mappings to HTML is simple enough for LiveView and Streams to effectively keep track of. It sounds ike you’re doing something else, or while you were using Streams ended up doing something different which broke the assumptions of Stream and forced it to rerender massive chunks everytime a coconut drops.

Between your words and my frame of reference I understood that the way you responded to (or at least tried to for a while) that (Streams resulting in runaway update sizes) was to do (a whole lot of) additional DOM manipulation somehow related to using the Stream API. If that’s not the case by all means correct with what you actually did.

OK, you keep quoting this. Apart from having an issue with throwing articles at people to make your point for you, this article is particularly far outside my range. Just from the title I know that I am in absolutely no position to even try reading the article, it’s just too far outside my frame of reference.

Remember, I’m not an experienced frontend programmer trying my hand at designing and building systems but a experienced Systems Architect (and visionary) forced by circumstance (of my own doing) to produce the frontend my system needs by means whatever most appropriate. I’ve never used React, never installed or wrote a line of jQuery code, try my best to avoid bringing any additional frameworkds like React into Phoenix LiveView equation. I dabbled a bit with the Vue concepts back in the day when I thought I could postpone writing a proper server and build some Vue-based SPA with a backend in Laravel, but then I migrated my efforts to Phoenix and replicated a year’s worth of progress within a week so I pivoted to write a proper backend and produce a frontend coded in the same language.

Which is a long way to say I haven’t read the article and won’t be reading it cause I can’t. You’re the one who understands the artiicle so just tell me what you wanted the article to tell me. Who are “we”, why would react need saving, what makes it worth saving, is it not saved enough by others like LiveView (in your words) having adopted its most useful parts as its own?

Again we’re at risk of wandering off topic. I really don’t want to get caught (again) in a discussion about how bad Streams are, and/or DOM manipulation code. I want to make my data better aligned with Streams so I can utilise its (paging) benefits for the intrinsically recursive data my app needs to present to users in HTML.

Perhaps for you issues with Streams, and because I misrepresent you, you’ll be better off starting your own discussion focussed on that. I don’t want this war of words to distract from the constructive discussion I’m trying to initiate.

The mapping between the state and the rendered template is not one-to-one in any nontrivial case. I think it’s best if I illustrate this in code.

This mapping is one-to-one: one message, one div. If you use streams for this, you can replace the messages with a stream and do a stream_insert/4 whenever a message is updated. This is the canonical use-case for streams, and it will work fine.

assign(socket, :messages, [%Message{...}, %Message{...}, ...])
defp add(socket, message) do
  assign(socket, :messages, socket.assigns.messages ++ [message])
end
# OR
stream(socket, :messages, [%Message{...}, ...])
defp add(socket, message) do
  stream_insert(socket, :messages, message)
end

~H"""
<div :for={msg <- @messages}>{msg.text}</div>
"""

But imagine we change the template to include a “message count” somewhere in the UI.
The mapping between %Message{}s and the template is no longer one-to-one. There are parts of the state (the number of messages) which are an arbitrary function (in this case a count) of the messages.

~H"""
<div>You have {length(@messages)} messages!</div>
<div :for{msg <- @messages}>{msg.text}</div>
"""

The issue with using streams for this is that when you try to use stream_insert, it only knows how to update a single message, but it doesn’t understand that it also has to update the count. So you would have to write what I’ve been calling “special-case code” to handle that specific case. Something like:

defp add(socket, message) do
  if message_is_new?(socket, message) do
    socket
    |> stream_insert(:messages, message)
    |> assign(:message_count, socket.assigns.message_count + 1)
  else
    stream_insert(socket, :messages, message)
  end
end

In the case of our little example the special-case code is still rather simple, but as we add more features the special cases will start to increase exponentially as they will all have weird dependencies on each other. That’s what the article I linked you was about.

So what does this have to do with recursive data?

Well, in the example I gave (the expand/collapse node), we want to render the button conditionally depending on whether a folder has children or not. The problem is that, when we update a child, the stream will not, on its own, understand that it has to update the parent. So we would have to write some sort of special case.

Note that this is pseudo-code, and the real code would be more complicated.

defp update_node(socket, node) do
  old_parent = get_old_parent_node(node)
  new_parent = get_new_parent_node(node)
  if length(old_parent.children) == 1 do
    # old parent will now be empty
    stream_insert(socket, :nodes, %Node{old_parent | ...})
  else
    socket
  end
  # same for new parent, and so on...
end

And then you have to write more code like this for many other features.

However, if you avoid streams and just write this as a normal LiveView, you will not have this problem. This is why I advised you to be wary of streams for this use-case.

I would like to hear more about this. In particular, how are you handling updates when the state (the underlying tree structure) changes?

This is dangerously close to being too far off-topic but let’s see if we can put this to bed quickly.

I don’t know if you just chose a bad example, but from the one you gave at least one problem seems obvious. The count of messages should not be calculated based on stream operations. It’s normal to calculate and maintain the count closer to the context app which set up the stream and other assigns the frontend should render from. “Unread” counts can be expensive to keep extracting from the database but there are other techniques to cache them per user and simply invalidate each affected cache when a message is sent or read. Not saying that what you should do, only that you need to consider the data required to implement these “features” at a higher level with the right tools. Doing it within the stream event callbacks, will drive you bonkers, and probably did.

You’re right, the stream by itsetself cannot figure that out but it will have figured out that your heex code referenced parent.children and mark it as requiring an update. It errs on the side of caution and usually (as you’ve described experiencing) regenerates far too much of the HTML just to be safe. The objective with better support for recursion in streams is to enable the stream version of the diffing LiveView does for assigns anyway to detect changes only where they actually occur and deal with them in isolation. In your example that would mean that even in the complete implementation of a display that depends the presence of children or not would result at most two records will change resulting in new HTML being generated and sent - the newly inserted node and the parent node which now happens to contain that one new node, but whatever it contains doesn’t matter because those additional children are not on the client yet (or else the children count would not have been 0) and has to be sent there. It would be able to detect that the rest of the tree did not change.

I now have a clearer understanding of what you’ve tried to warn me about, and I thank you once again for your kindness. I think I’ll be OK. For the places where I’ve used streams so far I’ve done the right thing and for the bigger problem involving large recursive datasets I soon will be doing the right thing in terms of being stream-friendly.

Unfortunately avoiding streams and using regular assigns is far more likely to expose the user to unacceptable and/or unmanged performance when the data cascades. Regular assigns make even harsher assumptions about the programmer ensuring that the assignes don’t get too big, and if you stick a variable sized assign element in there which preloaded recusrive associations would be your poor users will face a variable if not unpredictable experience. Regular assigns were designed for small variables or simple lists of small variables, or large variables broken down into lists of small or large variables, but the whole variable is processed. But dont worry, I’ll figure out how to trick streams into handling indefinitely recursive structures effectively.

That example is stripped down as much as possible because its purpose is to build intuition (for you). There are a great many ways one could implement this feature but what remains true is that there is a non-linear relationship between the state and the rendered template. If you want to learn more about this you can read the article I linked - I think a deeper explanation would veer, as you said, off-topic.

I’m a bit concerned by this reply. Streams can’t do any of this! The fact that they don’t do this is, in fact, the entire reason I’m warning you against using them.

It is not possible for streams to diff the parent because they do not even hold the parent in memory! It’s deleted after being used to render the template!

It sounds to me almost like you believe streams work the same was as normal renders - if this is the case then it is no wonder you have met my criticism with disbelief.

Funny, it sounds like your idea is converging on what I (briefly) proposed in a past comment as a possible solution. But I think it’s important to highlight two things: first, this has nothing to do with recursion (it’s a general problem with streams). And second, the changes required to fix this would make streams unrecognizable (and surely break backwards compatibility). In other words, it would necessary to call that new API something else.

You are quite right, and this is the problem I had to solve. In my very first reply to you I described how I did it.

To elaborate, there are two performance problems:

The first is that, because LiveView does not properly diff containers (or nested containers), the diffs sent down the socket are excessively large. The correct solution is for LV to diff containers (I think they’re working on it), but as a temporary fix you can wrap your nodes in LiveComponents to shrink the diffs down to constant. This is the performance advice I gave you.

The second is that you have to hold the entire tree in-memory on the server. This is the price of using LiveView and I accept it, so I made no attempt to fix this. If this is a real problem (you have a truly large tree, hundreds of thousands of nodes), then you will have to resort to some very advanced tactics. You could probably virtualize (i.e. paginate, in a nested fashion) the tree on the server. Note that you still don’t need streams to do that.