End To End Testing With Hound and Github-Actions

or alternatively, I’m open to swapping to Wallaby if anyone has had luck there.

I was reading some posts last night on Cypress as well, and it seems very nice, but keeping everything in the exunit/elixir world is something I’d like to do if possible to make it one less thing to onboard others to the project.

I’m using Wallaby together with GitHub Actions.
It’s a bit tricky to set up, but this elixir.yml GitHub Actions workflow should get you going

For setting up any Wallaby tests, I’m using this FeatureCase test helper:

For the rest, I simply followed the Setup Tutorial of Wallaby.

@PJUllrich thanks so much for sharing that. That gave me a great head start for figuring out caching if nothing else so totally worth it taking my build/test time down!

One question, I don’t see anything about starting chromedriver after its installed, are you starting that in your mix test task or something else?

name: Phoenix - Cypress Acceptance Tests

on: push

jobs:
  test:
    runs-on: ubuntu-latest

    services:
      db:
        image: postgres:11
        ports: ['5432:5432']
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v1.0.0
      - uses: actions/cache@v1
        with:
          path: deps
          key: ${{ runner.os }}-mix-cypress-${{ hashFiles('**/mix.lock') }}
          restore-keys: |
            ${{ runner.os }}-mix-cypress-
      - uses: actions/cache@v1
        with:
          path: _build
          key: ${{ runner.os }}-build-cypress-${{ hashFiles('**/mix.lock') }}
          restore-keys: |
            ${{ runner.os }}-build-cypress-
      # - run: cp config/dev.secret.exs.example config/dev.secret.exs
      - uses: actions/setup-elixir@v1.0.0
        with:
          otp-version: 22.1
          elixir-version: 1.9.1
      - run: mix deps.get
      - run: npm install --prefix assets
      - run: npm run deploy --prefix assets
      - run: mix phx.digest
      - run: mix cypress.ci

I am using Cypress for my acceptance tests, and had to run a few extra commands to get the JavaScript running properly on CI. I suggest trying

      - run: npm install --prefix assets
      - run: npm run deploy --prefix ./assets
      - run: mix phx.digest

Right before your run your tests and see what happens. I have no idea what the internals of Hound look like, but hopefully this helps.

I have been really happy with Cypress, and recommend it as an option.

Glad I could help

Yes, chromedriver is started by Wallaby. You don’t have to start anything manually.

Cypress as @alanvardy mentioned is also really, really nice! I can highly recommend it if you want to test your system as a black box, so exclusively test through the UI. Writing UI tests is a breeze using Cypress. In my case however, I needed to do “grey box”-testing, so, I had to both call some Elixir functions like e.g. insert(:post) and “click through the UI”. Since wallaby is written Elixir I could easily do both with it. I’m not sure about this, but maybe @alanvardy knows whether it’s possible to call Elixir functions using Cypress?

Awesome. I managed to get everything working last night, thanks for sharing those github action spec, that helped a ton at getting it working.

After adding a line to start chromedriver and background it, it seems to be working as expected.

I’m intrigued by cypress, it looks like a good approach and looks very clean. I’m going continue to explore there, but for now I’m glad to have hound rolling. I mostly wanted to stay in the elixir world due to the gray-box testing you mentioned, specifically for my project some “shopping cart”-type functionality that happens over several steps with items/quantities and persists between sessions. Its nice to be able to write assertions around what should be in the database.

Thanks so much again for the help!

I am not able to call any elixir functions within the code (I think), and pretty much black box the whole thing.

I have Cypress register an account, log out, log in, create and view resources etc. I made a 4th environment called :system_test and then can mock out anything I need to within the code (i.e. not requiring a confirmation email on sign up).

I try to cover as much as possible with the unit and integration tests because they are a lot faster, and then have Cypress for covering the happy path and the things that are impossible to test any other way… like if LiveView is still working after my last change