How important are acceptance tests in Phoenix?

Phoenix Forum Questions / Help
1

Hello everyone,

First post, so please let me know if I make a faux pas :slight_smile: I didn’t find what I was looking in the forum search but may have missed something.

I have been building hobby projects for the last few months, following the example of Programming Phoenix, and have been really enjoying the experience of working with ex_unit and writing unit and integration tests. I have been checking out acceptance/end-to-end tests over the last 24 hours (Wallaby and Hound) and have found the experience a bit difficult, mostly around all the caveats, PhantomJS issues etc. I’m taking a bit of a step back and wondering whether I should even be going down this road.

Part of the issue here is I am working on ‘selling’ Elixir to Ruby on Rails developers who consider the use of Cabybara essential to their processes.

How important is acceptance testing in Phoenix vs something like Rails? Does functional code require less high level testing because the individual pieces are generally less tightly coupled? How do you folks approach high level testing? (if at all?)

2

Hi,

On the frontend you want use only html views or other javascript frameworks(angular vue react etc)?

In any case cypress seems to be better than Cabybara.

A link with relevant info https://docs.cypress.io/faq/questions/general-questions-faq.html

One more link with comparassion between the two https://stackshare.io/stackups/capybara-vs-cypress

I hope this helps you.

3

Just .eex and .leex for me! I just want to use the server rendered solutions that are provided out of the box (and with LiveView).

One of the big appeals for me is avoiding the front end complexity.

4

I have been looking at Cypress! Looks very slick and I’m going to give it a test drive today.

It is kind of curious that the best solution is not an Elixir one. Is acceptance testing a lower priority? I guess I am asking if I should be acceptance testing as well as how.

Do you write end-to-end tests wolfiton?

5

At the moment NO, I prefer integration tests more.

Also I am new to elixir i come form a nodejs background.

6

Found this article on how to integrate cypress with phoenix https://bego.hr/cypress-elixir-phoenix-ecto-sandbox/

Another one here https://medium.com/@svengehring/phoenix-end-to-end-testing-in-real-life-efdba3a85be1

2 Likes
7

Very interesting, I have been playing with Cypress for the last few hours and I can see why it is so popular.

I can also see that I am not going to be integrating it into my workflow any time soon. The process of starting it is clunky, slow, and involves a lot of manual steps. The opposite of the rest of my Elixir testing experience!

Perhaps if I was working on a huge app with a large client facing component this would be worth it.

I am really curious if anyone out there is using it and what they are using it for!

1 Like
8

I got it working! It is single threaded, so not overly fast, but it is running with a single mix command now.

I wrote up a little post if anyone is curious, I think that most people could get this up and running in under 10min.

https://www.alanvardy.com/posts/36

I am now going to see how it fares on CircleCI :slight_smile:

3 Likes
9

I would suggest that the articles, you used to learn about cypress should be referenced in your article, because you didn’t have the knowledge before you started writing you article about this technology.

That seems fair to me

10

I’m super happy to provide credit when I use other resources (as you can see on my previous posts), but I didn’t use either of those two.

Both checked sandbox connections out via API, and were written for previous versions of Ecto. I didn’t want to go down this road.

I used the Cypress documentation, a couple of bash scripting tutorials, and a quick perusal of the Mix task documentation. This means I create a 4th environment, with a 4th database that I can reset before every system test. A part of the challenge for this newbie was figuring out how to start the server before the test and shut it down after while still keeping the exit code.

I think that linking to documents that I didn’t use, and are not referenced in my blog post, would be a little silly. :slight_smile:

Thank you for the suggestion though, did you read the post? I updated it with how I got it working with CircleCI

11

I used Wallaby testing an Elm frontend with Phoenix and had a good experience. Definitely use chrome headless instead of phantomjs though. Were your issues related to phantomjs or with the API that Wallaby provides?

12

Please read carefully what i wrote

I never said that the articles i provided are the one that should be put in your reference section I said in general.

1 Like
13

How important is acceptance testing in Phoenix vs something like Rails? Does functional code require less high level testing because the individual pieces are generally less tightly coupled?

Overall I don’t think there’s much difference to be honest? Both Ruby and Elixir are dynamically typed so even if your application modules are nicely structured and tested and decoupled, having some higher-level tests can definitely help.

How do you folks approach high level testing? (if at all?)

For server-side rendered HTML apps I’d write mostly Phoenix integration tests. These are easy to write, separated (thanks to Ecto Sandbox), pretty fast (no browser communication latency) and thanks to recycling the conn can be higher-level

If the app has a lot of client-side javascript, or/and if I need to test how the app renders in a real browser, I’d pick a bigger gun. Also I think even when most of the things are tested in Phoenix ConnCases, it’s good to test some of the most business-critical application paths with a real end-to-end testing framework.

The most common choice are WebDriver-based frameworks (“Selenium”). I would build something myself using the official Selenium WebDriver client libraries in Java, Ruby, etc, or use a nice framework with added conveniences like Capybara etc. There’s also Elixir Hound and Elixir Wallaby (which I’m currently working with), if you want to write the tests in Elixir. Either way, I wouldn’t run these tests with PhantomJS - it’s a dead project, and Chromedriver with headless Chrome is also pretty fast and can be used on the CI. For testing different browsers and mobile, there are platforms like Browserstack, Saucelabs or Crossbrowsertesting.

Using the “Chrome DevTools Protocol” for testing also seems appealing (maybe you’ve heard of puppeteer, which is a client for it), so if I had to test only Chrome and wanted to play with the “hot new tech” I’d try out Taiko. Seems a bit closer to the browser, it could be faster while being as capable as Selenium. Other browsers also could implement this protocol, Firefox is a WIP I think.

Overall while I don’t personally love it, JavaScript seems to be a good choice for end-to-end testing thanks to being event-driven (no need for “wait for” loops and less tinkering with custom timeouts).

Cypress seems like a nice choice with cool tooling and I’d like to try it out someday too. But Cypress runs entirely in the browser alongside the application code, so… I don’t know, it’s somewhere in the middle between Phoenix ConnCases and WebDriver? Not “truly” externally ran and end-to-end, can’t be used to test things like payment iframes or clicking through external services.

5 Likes
14

I see, sorry but there is some ambiguity to your previous post.

I interpreted this as you saying that I learned about Cypress through these articles, and that they should be referenced. No harm no foul :slight_smile:

I have already referenced the Cypress.io site at the beginning, which provided 80% of what I needed to get this working through some very nice documentation. I specifically wrote the post because the last 20% was mostly through experimentation, and didn’t exist anywhere else that I could find.

I think that I will reference this thread as my initial inspiration too!

15

I admit that I gave up pretty quickly on Wallaby (after about an hour) when I found that

  • My web searches were just recommending Hound or Cypress instead.
  • The project is looking for a new maintainer: Redirecting…
  • The setup instructions where for Ecto 2
  • The instructions were for installing PhantomJS, which doesn’t seem to want to install from the repository on Arch and is not being developed any more.

I’m not saying it isn’t an awesome project, the current maintainer is obviously a stand up guy, and the syntax looks great! I just encountered enough friction to make me look at other options.

Oh I really love the ex_unit experience, and want 95% of my tests to be within its scope. The tests are crazy fast, nice to write, and are just fantastic all around. My intention is to rely on ex_unit and then add a happy path acceptance test to keep the coworkers happy :slight_smile:

Oh cool, I will check this out. Thanks!

I am also not a huge fan of JavaScript, but have found the experience really painless with Cypress and its jQuery selector syntax for the tests. Definitely not a replacement for ex_unit integration tests, but I hope it covers a couple of the things that ex_unit cannot (like making sure forms are fillable and buttons are visible).

2 Likes
16

Very important if you want to sleep at night. Ultimately acceptance tests verify that your end users see what they need to see from your software. They are priceless and return dividends fast.

I use Hound to write these tests against chrome headless webdriver.

2 Likes