Wallaby Acceptance tests pass individually, fail when run together

I’ve posted this to StackOverflow as well in someone would prefer to answer there for the points

Can you reproduce it in a sample application? As long as you call Wallaby.start_session per setup and have async: false, you should be covered.

I’ll try to reproduce it in a sample application, but in the meantime will note that I was able to resolve the issue (and get all Wallaby tests passing when run together with mix test test/acceptance) by manually logging out (click(link("Log Out"))) at the end of each test case.

I store an authentication token in LocalStorage, and when I print it to the page (and take a screenshot in each test case), I can see that the token is the same for all of my test cases, even though I’m calling Wallaby.start_session before each test case. I would assume that, as a result of this PR, Wallaby sessions should not be sharing LocalStorage:

https://github.com/keathley/wallaby/pull/93

I’ll continue investigating…

I believe this may be a bug with Wallaby. When I IO.puts both the session.id and result of execute_script("return localStorage.getItem('token')" at the start of each test, it’s clear that each test is run in a differens phantomjs session (the ids differ) but that the token I’m setting with localStorage.setItem is persisting across sessions.

I’ve reproduced this behavior in a minimal sample app: GitHub - stevegrossi/wallaby_test: A test of how Wallaby handles localstorage across sessions · GitHub and have submitted an issue to Wallaby: localStorage persisting across sessions · Issue #274 · elixir-wallaby/wallaby · GitHub

@stevegrossi I am encountering the same error where if i run the E2E feature test alone they pass but fail if run in bulk (all together). After seeing all of the links you shared on differen platform i came to conclusion that it can be resolved by adding logout in each of the feature test case. But i have tried that and still getting the error. Also read in github that it was resolved but where i am doing wrong that it is failing for me.
below is the feature test case being run in a docker container

  feature "pagination is preserved after deleting", %{session: session} do
    # Create 15 two factor form detectors to ensure pagination
    two_factor_form_detectors =
      Enum.map(1..15, fn i ->
        {:ok, two_factor_form_detector} =
          TwoFactorFormDetectors.create_two_factor_form_detector(%{
            name: "Delete Pagination Test #{String.pad_leading("#{i}", 2, "0")}",
            input_fields_selector: "input[name='code']",
            submit_button_selector: "button[type='submit']",
            is_multi_input: false
          })

        two_factor_form_detector
      end)

    # Get a detector from page 2 (not the last one on the page)
    # With descending ID sort, page 2 has IDs 5,4,3,2,1. Use detector #4 (index 3)
    target_detector = Enum.at(two_factor_form_detectors, 3)
    first_page_detector_name = "Delete Pagination Test 15"

    session =
      session
      |> visit("/app/page1")
      |> wait_for_element(css("[data-testid='two-factor-form-detectors-table'], table"))

    # Navigate to page 2 - wait for pagination to be available first
    session =
      session
      |> wait_for_element(
        css("[data-testid='pagination-page-2'], div[phx-click='forward_to_page'][phx-value-page_number='2']")
      )
      |> click(css("[data-testid='pagination-page-2'], div[phx-click='forward_to_page'][phx-value-page_number='2']"))
      |> wait_for_page_load()
      |> wait_for_element(css("[data-testid='two-factor-form-detectors-table']"))
      |> assert_has(css("[data-testid='two-factor-form-detectors-table'] tbody tr", text: target_detector.name))

    # Delete a detector from page 2
    session =
      session
      |> execute_script("window.confirm = function() { return true; }")
      |> click(css("#two_factor_form_detector_actions_container_#{target_detector.id} button[aria-haspopup='true']"))
      |> assert_has(css("#delete_two_factor_form_detector_button_#{target_detector.id}"))
      |> click(css("#delete_two_factor_form_detector_button_#{target_detector.id}"))
      |> wait_for_flash_message()
      |> assert_page_contains("Two factor form detector deleted")
      |> wait_for_element(css("[data-testid='two-factor-form-detectors-table']"))

    # Verify we're still on page 2 (not reset to page 1)
    # The first page detector should not be visible, and we should still see other page 2 detectors
    # The deleted detector should not be visible anymore
    session
    |> wait_for_element(css("[data-testid='two-factor-form-detectors-table']"))
    |> wait_for_element_to_disappear(css("[data-testid='two-factor-form-detectors-table-row-#{target_detector.id}']"))
    |> refute_has(css("[data-testid='two-factor-form-detectors-table']", text: first_page_detector_name))
    |> assert_has(css("[data-testid='two-factor-form-detectors-table']", text: "Delete Pagination Test"))
  end

And its failing with the message

** (Wallaby.ExpectationNotMetError) Expected to find 1 visible element that matched the css ‘[data-testid=‘two-factor-form-detectors-table’] tbody tr’ and contained the text ‘Delete Pagination Test 04’, but 0 visible elements were found.

code: |> assert_has(css(“[data-testid=‘two-factor-form-detectors-table’] tbody tr”, text: target_detector.name))

Below is the setup

defmodule AppWeb.WallabyCase do
  use ExUnit.CaseTemplate

  alias Ecto.Adapters.SQL.Sandbox

  using do
    quote do
      use Wallaby.Feature

      import Ecto.Query
      import App.AccountsFixtures
      import AppWeb.WallabyCase

      # Tag all Wallaby-based tests as End-to-End so they can be included/excluded via --only/--exclude e2e
      @moduletag :e2e

      # The default endpoint for testing
      @endpoint AppWeb.Endpoint
    end
  end

  setup tags do
    # Clear Redis caches before each test to ensure isolation
    clear_redis_caches()

    pid = Sandbox.start_owner!(App.Repo, shared: not tags[:async])

    on_exit(fn ->
      Sandbox.stop_owner(pid)
      # Clear Redis again after test completes
      clear_redis_caches()
    end)

    metadata = Phoenix.Ecto.SQL.Sandbox.metadata_for(App.Repo, pid)

    # Start Wallaby session with debugging
    IO.puts("Starting Wallaby session...")
    {:ok, session} = Wallaby.start_session(metadata: metadata)
    IO.puts("Wallaby session started: #{inspect(session)}")

    # Try to get session info to verify it's working
    try do
      session_info = Wallaby.Browser.current_url(session)
      IO.puts("Session info retrieved: #{inspect(session_info)}")
    rescue
      e -> IO.puts("Error getting session info: #{inspect(e)}")
    end

    {:ok, session: session}
  end

  defp clear_redis_caches do
    # Clear all Redis-based caches to ensure test isolation
    try do
      App.SessionCache.flush()
    rescue
      _ -> :ok
    end

    try do
      App.RedisCounterCache.flush()
    rescue
      _ -> :ok
    end
  end
end