Performances when using live view with large table

Phoenix Forum Questions / Help
,
1

I am using live view to render a table:

  <table>
    <thead>
      <tr>
        <th>#</th>
        <%= for c <- @columns do %>
        <th>
          <%= tr(c.display_name, 32) %>
        </th>
        <% end %>
        <th></th>
      </tr>
    </thead>
    <tbody>
      <%= for {r, rnum} <- @rows_index do %>
      <tr class="<%= if rnum == @selected_y do "selected" end %>">
        <td><%= rnum + 1 %></td>
        <%= for {val, cnum} <- r  do %>
        <td phx-click="select_cell"
          phx-value-x="<%= cnum %>"
          phx-value-y="<%= rnum %>"
          class="<%= if cnum == @selected_x && rnum == @selected_y do "selected" end %>">
          <%= val %></td>
        <% end %>
        <td></td>
      </tr>
      <% end %>
    </tbody>
  </table>

I have about 150 columns and 10 000 rows.

The problem is that when I do the select_cell call, and update selected_x and selected_y the update takes a few seconds.

Is there a way to optimize this with live view, or should I move to JS?

4 Likes
2

You should use temporary assigns for the rows and you can either use phx-update="append" and append the two updated rows (the old row which is now unselected and the new selected row), or you can use a component for reach row, and do a send_update to the old selected row, and newly selected row. Both options will send the minimal diff down and not require holding 10k rows in your server state.

8 Likes
3

If I get this right, I should:

  • move data to temporary assigns
  • create a row component
  • put phx-target to myself on the row component and handle the event in the row component
  • in the event handler, I use pubsub to broadcast the row change selection
  • in handle_info of the live view, I send_update to the two affected rows

Is that the way to go?

4

You don’t need the pubsub step and instead of targeting the component, let the parent LV handle the event, then it performs the send_update to both the component children. The parent LV has the selected state, so it can use that to send_update the selected component, then the handle_event params can contain the desired selected row, which you will use to send_update the newly selected component. Make sense?

2 Likes
5

Yes, it is working with send_update from the live view.

I did create a component for each rows, but with large tables (>100 columns), even updating a single row takes a noticeable time.

Is it possible to nest components? Like having a component for the row that would just render <tr class="..."> and only handle the switch of class on the the row while using a for loop to render sub components? Would that just work?

Also, I have a “tab” selector above my tables, and when I change tab, I need to re-render the whole table. Right now, live view is sending a “huge” diff with every rows when changing tabs. Is it possible to tell live view to just re-render the container and replace the whole container content in one operation?

6

I have quite similar application, using send_update can get hairy at some point (like jQuery approach, sometimes it needs other components updated too e.g. resetting [ui] state for the rest, and only update a component)

I have a very big tree component (recursively rendered), so yes nested components is possible. However, the way out often is rethinking of state design. It’s quite tricky to get minimal diff payload for large DOM nodes.

I am moving towards more pagination, less DOM nodes is better. For the “tab”, you could possibly remodeling your assigns, moving assigns that make it marked as “changed” out, when switching tab 1 -> tab 2 -> tab 1 (this stays unchanged), but if you want always fresh data, I don’t think there is a way around other than reducing DOM nodes. (And I think toString on javascript side is not that performant … something around Array.from that is slow)

Also be aware of memory usage per process (per browser tab) too!

7

I tried to have a top component, like this:

... here html for a tab selector that change @table assign

  <%= live_component @socket,
        CardioDataWeb.DBLive.TableComponent,
        id: :main,
        table: @table,
        sel: @sel
  %>

This above component render a tag with thousands of cells.

Now, when I change the @table in the top component, instead of just having 1 single big diff like wrapper.innerHTML = "some big chunk of html" I have thousands and thousands of diffs which takes tens of seconds to be applied. (if I render the HTML normally without live view, rendering in the browser is instant)

My tab selector could be regular links, but I want to keep the state between tab changes.

My use case is some sort of spreadsheet editor. I could use some JS lib, but I tested native HTML rendering, and everything is butter smooth with thousands of rows, browsers are really good at managing large pages and I thought I could leverage this with live view, but live view needs to have minimal diff.

8

I also tried nesting live views, but when the child live view is unmounted/remounted, there is also a huge diff which takes a very long time to be processed (much longer than just doing similar to innerHTML = newdata).

9

My untested idea is that you could do something like what infinite twitter feed does, only render rows fit within screen viewport, not sure if it would be clunky or not. For example, 300 rows. The downside is native search on browser will suck.

That’s the only solution (pagination) I stick with, I gave up minimizing diff payload in other way since it’s going to have at least a bunch of one character keys (a lot of them alone without content is huge)

Or just give up view layer on server for those cells, just push_event with pure data and use whatever lib to render the cells on client.

10

Also I guess your sel: @sel sit on every row (somewhere like <%= if @sel, do: "" %>), its change always mark the whole table (i.e. every row) as change and produce diff payload of every row even if @table data is unchanged.

11

Well “smart partial client side rendering” is possible, but I want to avoid that because rendering the whole table works very well even with huge table.

And I don’t want to minimize diff size, what I’d like is to have a single large diff for the table when the table change, not a million small diffs for all element in the table.

As I said if I re-render the whole table, I’d like the container’s content to be swapped with the new content, in one single operation. I don’t want a million diff be applied, which is very slow.

So I am wondering if this is possible with live view.

12

I am currently trying a mixed approach where I would still use live view for rendering.

Basically, instead of rendering the whole table, I render only the visible part (in a live view), and I push the coordinates when scrolling.

It is still early stage but I think this road should work while requiring very minimal JS.

13

Ok, I ended up with the following solution:

  • set a hook on the container like so:

Hooks.table = {
  mounted () {
    this.resize()
    this.resizeListener = () => this.resize()
    this.scrollListener = (evt) => this.scroll(evt)
    window.addEventListener('resize', this.resizeListener)
    this.el.addEventListener('wheel', this.scrollListener)
  },
  destroyed () {
    window.removeEventListener('resize', this.resizeListener)
    this.el.removeEventListener('wheel', this.scrollListener)
    this.resizeListener = null
  },
  resize () {
    this.rect = this.el.getBoundingClientRect()
    this.pushEventTo('#table-component', 'resize', this.rect)
  },
  scroll (evt) {
    let d = evt.deltaY
    if (d === 0) {
      return
    }
    switch (evt.deltaMode) {
      case WheelEvent.DOM_DELTA_PIXEL:
        break;
      case WheelEvent.DOM_DELTA_LINE:
        d *= 16;
        break;
      case WheelEvent.DOM_DELTA_PAGE:
        d *= this.rect.height / 2;
        break;
    }
    let off = {x: 0, y: 0}
    if (evt.shiftKey) {
      off.x = d
    } else {
      off.y = d
    }
    this.pushEventTo('#table-component', 'scroll', off)
  }
}

have a simple handler like this:


  @impl true
  def handle_event("resize", %{"width" => width, "height" => height}, socket) do
    {:noreply,
     assign(socket, viewport: %{width: width, height: height})
     |> do_fetch_data()}
  end

  @impl true
  def handle_event("scroll", %{"x" => x, "y" => y}, socket) do
    offset = socket.assigns.offset

    offset = %{
      x: offset.x + x,
      y: offset.y + y
    }

    {:noreply, assign(socket, offset: offset) |> do_fetch_data()}
  end

Then in the do_fetch_data I clamp the offset, and I gather the data rows for the given view.

I then just render that “window” on my large data table.

This approach doesn’t provide native scrolling, but it is very fast and I can stream over my large table. The good side is very minimal javascript code (of course I would need a bit more for scrollbars, this is a proof of concept).

Live view is very fast at re-rendering the table on scroll events.

2 Likes
14

I think the viewport / sliding window approach you have landed on is nice!

Pretty sure it’s the only graceful way to handle this without too much data on either the client, or the server…

15

That’s what I said earlier!

Is your app server in the same country or region? It’s nice to hear that rendering per scroll event is fast (I assume do_fetch_data hit database?), I’d love to adopt this in my app as well, I was afraid of latency in worst case scenario (e.g. EU ↔ NZ).

16

Yeah, I didn’t want to smart render only the required region because I was surprised at how good browser were at handling large table “out of the box”, but sadly it doesn’t play well with live view diffing mechanism.

As for latency, yeah, the server is located on the same country. I guess the only way to cope with larger latency would be to send more data to the client at once, and to scroll client side within that “larger data”.

1 Like
17

I found this tutorial helpful for those looking to work with temporary assigns:
Updating and Deleting from Temporary Assigns in Phoenix LiveView. It’s a bit older but it checks out.

18

I’m currently running into the same issue. I have a table with some data, but not a large amount. If I get over 100 rows it starts to slow down a lot. I’m looking to put 2000 rows. I would really like to have all of that on one page.

It is slow even on the initial rendering of the live view. Outside of updating does anyone know how to just render it quickly? It all seems to be rendering time that is killing it all.

Does anyone have any idea?

19

Oh, I just learned about profiling with live view.

This might help me find it.

2 Likes