What's the state of the art E2E testing mix package for elixir?

Hi there,

I’ve recently added end-to-end testing to my phoenix application. I quickly tried setting up hound but then ended up using wallaby which I’m quite happy with, but it’s still not clear to me which one the better one for the future is.

I found some discussions that hinted at the fact that wallabys development has stopped but it looks like there is some recent activity.

Because of this I wanted to ask the community what their experiences with E2E hex packages for elixir is and what the most future proof library of them is.

Thanks for your answers already!

I am one of the current maintainers of Wallaby, development is still active. Feel free to jump into the #wallaby channel in the Elixir slack if you have any questions about Wallaby :smile:.

4 Likes

Hi @mhanberg First of all thanks for taking over the maintenance of Wallaby, I think maintainers don’t receive enough love for what they do so here you go :smile:.

Since you’re already here I wanted to ask you about the strong and weak points of Wallaby compared to other solutions (I’m ok with a biased answer).

We actually ended up using Cypress and write our E2E tests in JS. Cypress is well maintained and has a fairly friendly API.

2 Likes

I have never used Hound, so I can’t really comment on the differences in anger. All I know is the API’s for each are pretty different, with Wallaby being more pipeline operator driven.

I have used Cypress and I did not like using it to be honest, but I’ll note my experience was minimal. One key difference is you won’t have access to any of your elixir code when using Cypress, so if you need to seed your database, you’ll need to figure out a way to do that.

Cypress does seem to be backed by a product company (based around Cypress) and seems to have major support in the JS community, so I would expect that it would continue being well supported.

1 Like

Thanks @mhanberg. Cypress isn’t an option for me since I don’t want to have to deal with duplicating test setup code.

For now I’ll probably stick with wallaby. Thanks for your time!

1 Like

This phrasing kind of hints that maybe your E2E tests are not purist black-box style testing and instead rely on some prior state to be present in the database. If that’s so, I’d suggest re-examining that testing design choice. E2E tests should generally not be aware of or reliant on application internals, just the behavior a user/client can see and interact with directly.

2 Likes

@shanesveller I’m indeed not performing purist black box testing. I do have specific test cases that rely on some previous state being in the database. How the data gets there is another test case which is also covered.

The test setup code that I have is mostly around setting up users and setting up some basic records in the database. It wouldn’t be a huge effort to migrate to something like Cypress since all the setup would be doable from the UI (I only use my public API methods for setting up the test cases anyway).

So the main reason for staying with an elixir lib is to not have another tool that I need to integrate into the (pet) project. Given that Wallaby provides me with all the features I need I see no need to switch for this particular project.

1 Like

While I can understand the notion of E2E being everything running like in production there’s also merit in using those tools “just” to make sure things don’t break in a browser. With wallaby/hound + ecto sandbox you can have isolated tests, where setup is done via plain elixir code, while only the actually subject under tests is dealt with by remote controlling a browser. One can for sure use cypress, but that either means tests depend on state of previous tests, or setup tasks can become expensive as all the setup must be done via the controlled browser. And I question the merit of creating a tenant via the UI over and over again because 97% of tests need a tenant.

2 Likes

I am drawing a distinction between integration tests that happen to exercise a browser and actual black-box testing. You’ll note that I intentionally used the word “purist” in that reply to acknowledge where my stance falls on the pragmatic<->pedantic spectrum.

1 Like