Infinite scroll issue - scrolling up leads to new data being prepended to the top of the stream which breaks ordering

I am trying to use a stream to show a list of people and use “infinite scroll” to allow a user to scroll up and down the list. The list of people is fetched from the database sorted by first name, last name and then an id key.

I am using the example code in the “Scroll Events and Infinite Stream Pagination” almost exactly: Bindings — Phoenix LiveView v0.20.14 (hexdocs.pm) but I am experiencing some behaviour that is really hurting my brain.

If I scroll down the produced list of names everything is fine. The next page loads in correctly, the new data is appended to the end of the stream and if needed, data at the top of the stream is replaced.

When I scroll up to go to a previous page the behaviour is very different. The data for the previous page is fetched correctly but the new data is prepended to the very top of the stream which immediately breaks the ordering of the list.

The new data gets added to the top of the stream which means the user will scroll up past the left over stream items that have first names beginning with say, a K and then hit the newly fetched items that might begin with an M.

You can see the result here:

The first 20 results are the product of the “scroll up” pagination query and the ones below were pre-existing from the previous “scroll down” queries.

I’m clearly doing something wrong but I’m not sure what. I don’t know if I’m supposed to be compensating for the existing items in the stream when I calculate the offset on a scroll up or if I’m supposed to be re-ordering the stream manually in some way to ensure cohesion in the list. I’m fairly new to Streams and this is my learning case!

My code is almost verbatim the code from the documentation:

The template:

<div
    :if={@live_action == :index}
    id="user_container"
    phx-update="stream"
    phx-viewport-top={@page > 1 && "prev-page"}
    phx-viewport-bottom={!@end_of_timeline? && "next-page"}
    phx-page-loading
  >
    <div
      :for={{dom_id, user} <- @streams.user_search_results}
      id={dom_id}
      class="mt-8 grid grid-col-1 gap-12"
    >
      <%= user.first_name %> <%= user.last_name %> (<%= user.employee_number %>)
    </div>
    
    <div :if={@end_of_timeline?} class="mt-5 text-[50px] text-center">
      🎉 You made it to the beginning of time 🎉
    </div>
  </div>

the pagination functions (per_page is currently 20):

  defp paginate_results(socket, 1, filters) do
    filters = Map.delete(filters, :offset)
    users =
      Users.find_users(filters)

    socket =
      stream(socket, :user_search_results, users, reset: true)
      |> assign(:filters, filters)

    socket
  end

  defp paginate_results(socket, new_page, filters) when new_page >= 1 do
    %{per_page: per_page, page: current_page} = socket.assigns
    filters = Map.put(filters, :limit, per_page) |> Map.put(:offset, (new_page - 1) * per_page)

    users =
      Users.find_users(filters)

    {users, at, limit} =
      if new_page >= current_page do
        {users, -1, per_page * 3 * -1}
      else
        {Enum.reverse(users), 0, per_page * 3}
      end


    case users do
      [] ->
        assign(socket, end_of_timeline?: at == -1) |> assign(:filters, filters)

      [_ | _] = users->
        socket
        |> assign(end_of_timeline?: false)
        |> assign(:page, new_page)
        |> assign(:filters, filters)
        |> stream(:user_search_results, users, at: at, limit: limit)
    end
  end

  def handle_event("next-page", _, socket) do
  end

  def handle_event("prev-page", %{"_overran" => true}, socket) do
    {:noreply, paginate_results(socket, 1, socket.assigns.filters)}
  end

  def handle_event("prev-page", _, socket) do

    if socket.assigns.page > 1 do
      {:noreply, paginate_results(socket, socket.assigns.page - 1, socket.assigns.filters)}
    else
      {:noreply, socket}
    end
  end

It seems like the stream guide uses a stream limit (i.e. in-DOM limit) of 3x the page size, meaning that when you scroll up it’s meant to overwrite the first couple pages in the DOM before the prepends start taking place (since stream/4 upserts into the DOM).

I’m not sure I fully understand your description of the behavior, but it seems like maybe the previous page is getting prepended when it’s supposed to be overwriting the existing rows for the first couple of pages (as described above). If so, perhaps the culprit is unstable dom_id values?

It’s not clear to me what the id column of your user structs is. The code you posted shows an employee_number value, which also appears in the DOM as the id - are you using stream_configure/3 for this, or do these structs also have an id?

Also, your :for attribute refers to user_search_results but then you access a user variable instead - I’m assuming this was just a transcription error?

Hi garrison, you’re correct. The dom_id values are derived from an id key value in a stream_configure:

stream_configure(:user_search_results, dom_id: &"user-card-#{&1.user_id_key}")

as the structs do not have an “id” field. The user_id_key is a unique value so it was my understanding that as long as the dom_ids are unique their structure doesn’t matter. Is that not correct?

The behaviour is a bit awkward to describe properly.

  • My initial page is 20 items.
  • I scroll down to page 2 and get the next 20 items with a query offset of 20. This is appended to the items returned from page 1.
  • I scroll down again to page 3 and get the next 20 items with a query offset of 40. This is appended to the items returned from pages 1 and 2
  • I scroll down again to page 4 and get the next 20 items with a query offset of 60. This is appended to the items returned from pages 2 and 3. page 1 disapears from the dom as I have hit my stream limit of 60 items.
  • I scroll up from page 4 to page 3.The query thats fired has an offset of 40 that represents page 3 of the list. These items are prepended to the stream container which means my container now looks like: 20 items from page 3, 20 items from page 2, 20 items from page 4.

Note that data in the pagination queries is ordered in-query:

from q in query, order_by: [asc: q.last_name, asc: q.first_name, asc: q.user_id_key]

I hope that is more descriptive. I think it suggests that when I scroll up I need to query for a larger offset to account for the fact I already have records in the DOM but the documentation example suggests that the code snippet is all I need for infinite scroll.

I’ve corrected the transcript error, good spot.

This was a very clear description of the problem! What you’re describing is what I figured was going on, which is that the items are being prepended when they’re supposed to be replaced in-DOM for that page (since they already exist).

Suspecting a bug, I did some brief spelunking into the LiveView commit history and it appears this behavior was changed/fixed earlier this year:

So I assume you’re on an outdated version - update your deps and try again :wink:

1 Like

After updating Liveview everything is working as expected. Thanks for the pointer, it literally never occurs to me that a problem might not be my fault.