I’ve recently added end-to-end testing to my phoenix application. I quickly tried setting up hound but then ended up using wallaby which I’m quite happy with, but it’s still not clear to me which one the better one for the future is.
I found some discussions that hinted at the fact that wallabys development has stopped but it looks like there is some recent activity.
Because of this I wanted to ask the community what their experiences with E2E hex packages for elixir is and what the most future proof library of them is.
I am one of the current maintainers of Wallaby, development is still active. Feel free to jump into the #wallaby channel in the Elixir slack if you have any questions about Wallaby .
Hi @mhanberg First of all thanks for taking over the maintenance of Wallaby, I think maintainers don’t receive enough love for what they do so here you go .
Since you’re already here I wanted to ask you about the strong and weak points of Wallaby compared to other solutions (I’m ok with a biased answer).
I have never used Hound, so I can’t really comment on the differences in anger. All I know is the API’s for each are pretty different, with Wallaby being more pipeline operator driven.
I have used Cypress and I did not like using it to be honest, but I’ll note my experience was minimal. One key difference is you won’t have access to any of your elixir code when using Cypress, so if you need to seed your database, you’ll need to figure out a way to do that.
Cypress does seem to be backed by a product company (based around Cypress) and seems to have major support in the JS community, so I would expect that it would continue being well supported.
This phrasing kind of hints that maybe your E2E tests are not purist black-box style testing and instead rely on some prior state to be present in the database. If that’s so, I’d suggest re-examining that testing design choice. E2E tests should generally not be aware of or reliant on application internals, just the behavior a user/client can see and interact with directly.
@shanesveller I’m indeed not performing purist black box testing. I do have specific test cases that rely on some previous state being in the database. How the data gets there is another test case which is also covered.
The test setup code that I have is mostly around setting up users and setting up some basic records in the database. It wouldn’t be a huge effort to migrate to something like Cypress since all the setup would be doable from the UI (I only use my public API methods for setting up the test cases anyway).
So the main reason for staying with an elixir lib is to not have another tool that I need to integrate into the (pet) project. Given that Wallaby provides me with all the features I need I see no need to switch for this particular project.
While I can understand the notion of E2E being everything running like in production there’s also merit in using those tools “just” to make sure things don’t break in a browser. With wallaby/hound + ecto sandbox you can have isolated tests, where setup is done via plain elixir code, while only the actually subject under tests is dealt with by remote controlling a browser. One can for sure use cypress, but that either means tests depend on state of previous tests, or setup tasks can become expensive as all the setup must be done via the controlled browser. And I question the merit of creating a tenant via the UI over and over again because 97% of tests need a tenant.
I am drawing a distinction between integration tests that happen to exercise a browser and actual black-box testing. You’ll note that I intentionally used the word “purist” in that reply to acknowledge where my stance falls on the pragmatic<->pedantic spectrum.
Piggybacking on this thread - I’m still using Hound, but lately all the tests are getting “invalid session id” (which I assume is being pushed back from Chromedriver). But it looks like hound hasn’t been updated in four years - so I’m wondering whether hound is still viable and whether I should even try to figure this issue out, or just switch to Wallaby.
Any further thoughts on the state of the art in `24?
That error means your chrome and chromedriver are not on the same version. You need to make sure they are. Afaik there’s now even a chrome version for testing so it doesn’t autoupdate anymore and can be easier matched with a chromedriver version. Doesn’t matter if you use hound or wallaby to call the chromedriver.
Hound still works, but they changed something in recent Chrome or Chromedriver and for headless mode to work I needed to call Hound.start_session([additional_capabilities: %{browserName: "chrome"}]) now
.
