I’m using Wallaby (0.17.0) for acceptance tests with phantomjs (2.1.1), and the latest Elixir (1.4.5) and Erlang (20). I’ve written a few acceptance tests, each of which passes when run individually, but when I run them together (via mix test test/acceptance) they fail because things that should be on the page aren’t. It’s as if all the tests are running in the same browser, even though Wallaby should “manage multiple browsers for you.”
I believe I’ve set everything up according to the setup section of the README, although I’m not running the tests with async: true because my app uses sockets and Phoenix.Ecto.SQL.Sandbox doesn’t play well with async. To be clear, the tests also fail with async: true, I just also get DBConnection.OwnershipErrors in each when attempting to connect to the socket.
Is there anything else necessary to ensure each test is run in its own phantomjs browser, or to clear out sessions between test runs?
1 Like
I’ve posted this to StackOverflow as well in someone would prefer to answer there for the points 
Can you reproduce it in a sample application? As long as you call Wallaby.start_session per setup and have async: false, you should be covered.
I’ll try to reproduce it in a sample application, but in the meantime will note that I was able to resolve the issue (and get all Wallaby tests passing when run together with mix test test/acceptance) by manually logging out (click(link("Log Out"))) at the end of each test case.
I store an authentication token in LocalStorage, and when I print it to the page (and take a screenshot in each test case), I can see that the token is the same for all of my test cases, even though I’m calling Wallaby.start_session before each test case. I would assume that, as a result of this PR, Wallaby sessions should not be sharing LocalStorage:
https://github.com/keathley/wallaby/pull/93
I’ll continue investigating…
I believe this may be a bug with Wallaby. When I IO.puts both the session.id and result of execute_script("return localStorage.getItem('token')" at the start of each test, it’s clear that each test is run in a differens phantomjs session (the ids differ) but that the token I’m setting with localStorage.setItem is persisting across sessions.
I’ve reproduced this behavior in a minimal sample app: https://github.com/stevegrossi/wallaby_test and have submitted an issue to Wallaby: https://github.com/keathley/wallaby/issues/274
1 Like
@stevegrossi I am encountering the same error where if i run the E2E feature test alone they pass but fail if run in bulk (all together). After seeing all of the links you shared on differen platform i came to conclusion that it can be resolved by adding logout in each of the feature test case. But i have tried that and still getting the error. Also read in github that it was resolved but where i am doing wrong that it is failing for me.
below is the feature test case being run in a docker container
feature "pagination is preserved after deleting", %{session: session} do
# Create 15 two factor form detectors to ensure pagination
two_factor_form_detectors =
Enum.map(1..15, fn i ->
{:ok, two_factor_form_detector} =
TwoFactorFormDetectors.create_two_factor_form_detector(%{
name: "Delete Pagination Test #{String.pad_leading("#{i}", 2, "0")}",
input_fields_selector: "input[name='code']",
submit_button_selector: "button[type='submit']",
is_multi_input: false
})
two_factor_form_detector
end)
# Get a detector from page 2 (not the last one on the page)
# With descending ID sort, page 2 has IDs 5,4,3,2,1. Use detector #4 (index 3)
target_detector = Enum.at(two_factor_form_detectors, 3)
first_page_detector_name = "Delete Pagination Test 15"
session =
session
|> visit("/app/page1")
|> wait_for_element(css("[data-testid='two-factor-form-detectors-table'], table"))
# Navigate to page 2 - wait for pagination to be available first
session =
session
|> wait_for_element(
css("[data-testid='pagination-page-2'], div[phx-click='forward_to_page'][phx-value-page_number='2']")
)
|> click(css("[data-testid='pagination-page-2'], div[phx-click='forward_to_page'][phx-value-page_number='2']"))
|> wait_for_page_load()
|> wait_for_element(css("[data-testid='two-factor-form-detectors-table']"))
|> assert_has(css("[data-testid='two-factor-form-detectors-table'] tbody tr", text: target_detector.name))
# Delete a detector from page 2
session =
session
|> execute_script("window.confirm = function() { return true; }")
|> click(css("#two_factor_form_detector_actions_container_#{target_detector.id} button[aria-haspopup='true']"))
|> assert_has(css("#delete_two_factor_form_detector_button_#{target_detector.id}"))
|> click(css("#delete_two_factor_form_detector_button_#{target_detector.id}"))
|> wait_for_flash_message()
|> assert_page_contains("Two factor form detector deleted")
|> wait_for_element(css("[data-testid='two-factor-form-detectors-table']"))
# Verify we're still on page 2 (not reset to page 1)
# The first page detector should not be visible, and we should still see other page 2 detectors
# The deleted detector should not be visible anymore
session
|> wait_for_element(css("[data-testid='two-factor-form-detectors-table']"))
|> wait_for_element_to_disappear(css("[data-testid='two-factor-form-detectors-table-row-#{target_detector.id}']"))
|> refute_has(css("[data-testid='two-factor-form-detectors-table']", text: first_page_detector_name))
|> assert_has(css("[data-testid='two-factor-form-detectors-table']", text: "Delete Pagination Test"))
end
And its failing with the message
** (Wallaby.ExpectationNotMetError) Expected to find 1 visible element that matched the css ‘[data-testid=‘two-factor-form-detectors-table’] tbody tr’ and contained the text ‘Delete Pagination Test 04’, but 0 visible elements were found.
code: |> assert_has(css(“[data-testid=‘two-factor-form-detectors-table’] tbody tr”, text: target_detector.name))
Below is the setup
defmodule AppWeb.WallabyCase do
use ExUnit.CaseTemplate
alias Ecto.Adapters.SQL.Sandbox
using do
quote do
use Wallaby.Feature
import Ecto.Query
import App.AccountsFixtures
import AppWeb.WallabyCase
# Tag all Wallaby-based tests as End-to-End so they can be included/excluded via --only/--exclude e2e
@moduletag :e2e
# The default endpoint for testing
@endpoint AppWeb.Endpoint
end
end
setup tags do
# Clear Redis caches before each test to ensure isolation
clear_redis_caches()
pid = Sandbox.start_owner!(App.Repo, shared: not tags[:async])
on_exit(fn ->
Sandbox.stop_owner(pid)
# Clear Redis again after test completes
clear_redis_caches()
end)
metadata = Phoenix.Ecto.SQL.Sandbox.metadata_for(App.Repo, pid)
# Start Wallaby session with debugging
IO.puts("Starting Wallaby session...")
{:ok, session} = Wallaby.start_session(metadata: metadata)
IO.puts("Wallaby session started: #{inspect(session)}")
# Try to get session info to verify it's working
try do
session_info = Wallaby.Browser.current_url(session)
IO.puts("Session info retrieved: #{inspect(session_info)}")
rescue
e -> IO.puts("Error getting session info: #{inspect(e)}")
end
{:ok, session: session}
end
defp clear_redis_caches do
# Clear all Redis-based caches to ensure test isolation
try do
App.SessionCache.flush()
rescue
_ -> :ok
end
try do
App.RedisCounterCache.flush()
rescue
_ -> :ok
end
end
end
