End To End Testing With Hound and Github-Actions

Hi all,

Does anyone have a working setup to do end to end testing using hound with github-actions?

I’ve got it working locally :roll_eyes:, and its a great way to sanity check the multi-step liveview features on this project but when running on the github-action I get a

** (RuntimeError) invalid element state for each of those tests.

When using hounds fill_field function.

I’ve tried several different combinations of things last night and this morning and seem to be at a deadend, and I’m unsure what else to try.

My workflow.yaml is below.

name: Master PR

on: 
  pull_request:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-18.04

    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: larder_test
        ports:
        - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-elixir@v1.1.0
      with:
        otp-version: 22.x
        elixir-version: 1.9.x
        
    - name: Install Dependencies
      run: |
        mix local.rebar --force
        mix local.hex --force
        mix deps.get


    - name: Setup Node
      uses: actions/setup-node@v1

    - name: Install ChromeDriver
      run: |
        CHROME_VERSION=$(google-chrome --version | cut -f 3 -d ' ' | cut -d '.' -f 1) \
          && CHROMEDRIVER_RELEASE=$(curl --location --fail --retry 3 http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROME_VERSION}) \
          && curl --silent --show-error --location --fail --retry 3 --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_RELEASE/chromedriver_linux64.zip" \
          && cd /tmp \
          && unzip chromedriver_linux64.zip \
          && rm -rf chromedriver_linux64.zip \
          && sudo mv chromedriver /usr/local/bin/chromedriver \
          && sudo chmod +x /usr/local/bin/chromedriver \
          && /usr/local/bin/chromedriver --version
          
    - name: Run Tests
      run: |
        export DISPLAY=:99
        /usr/local/bin/chromedriver --port=9515 &
        curl localhost:9515
        sudo xvfb -ac :99 -screen 0 1280x1024x24 > /dev/null 2>&1 &
        mix compile 
        mix test
      env:
        # use postgres for the host here because we have specified a container for the job.
        # If we were running the job on the VM this would be localhost
        DB_USER: postgres
        DB_PASS: postgres
        DB_NAME: larder_test

My Hound config lines:

config :hound, driver: "chrome_driver", browser: "chrome_headless"

config :hound, app_host: "http://localhost", app_port: 4002

config :hound, retry_time: 500

Can anyone point me in the right direction?

1 Like

or alternatively, I’m open to swapping to Wallaby if anyone has had luck there.

I was reading some posts last night on Cypress as well, and it seems very nice, but keeping everything in the exunit/elixir world is something I’d like to do if possible to make it one less thing to onboard others to the project.

I’m using Wallaby together with GitHub Actions.
It’s a bit tricky to set up, but this elixir.yml GitHub Actions workflow should get you going :slight_smile:

For setting up any Wallaby tests, I’m using this FeatureCase test helper:

For the rest, I simply followed the Setup Tutorial of Wallaby.

2 Likes

@PJUllrich thanks so much for sharing that. That gave me a great head start for figuring out caching if nothing else so totally worth it taking my build/test time down!

One question, I don’t see anything about starting chromedriver after its installed, are you starting that in your mix test task or something else?

name: Phoenix - Cypress Acceptance Tests

on: push

jobs:
  test:
    runs-on: ubuntu-latest

    services:
      db:
        image: postgres:11
        ports: ['5432:5432']
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v1.0.0
      - uses: actions/cache@v1
        with:
          path: deps
          key: ${{ runner.os }}-mix-cypress-${{ hashFiles('**/mix.lock') }}
          restore-keys: |
            ${{ runner.os }}-mix-cypress-
      - uses: actions/cache@v1
        with:
          path: _build
          key: ${{ runner.os }}-build-cypress-${{ hashFiles('**/mix.lock') }}
          restore-keys: |
            ${{ runner.os }}-build-cypress-
      # - run: cp config/dev.secret.exs.example config/dev.secret.exs
      - uses: actions/setup-elixir@v1.0.0
        with:
          otp-version: 22.1
          elixir-version: 1.9.1
      - run: mix deps.get
      - run: npm install --prefix assets
      - run: npm run deploy --prefix assets
      - run: mix phx.digest
      - run: mix cypress.ci

I am using Cypress for my acceptance tests, and had to run a few extra commands to get the JavaScript running properly on CI. I suggest trying

      - run: npm install --prefix assets
      - run: npm run deploy --prefix ./assets
      - run: mix phx.digest

Right before your run your tests and see what happens. I have no idea what the internals of Hound look like, but hopefully this helps.

1 Like

I have been really happy with Cypress, and recommend it as an option.

1 Like

Glad I could help :slight_smile:

Yes, chromedriver is started by Wallaby. You don’t have to start anything manually.

Cypress as @alanvardy mentioned is also really, really nice! I can highly recommend it if you want to test your system as a black box, so exclusively test through the UI. Writing UI tests is a breeze using Cypress. In my case however, I needed to do “grey box”-testing, so, I had to both call some Elixir functions like e.g. insert(:post) and “click through the UI”. Since wallaby is written Elixir I could easily do both with it. I’m not sure about this, but maybe @alanvardy knows whether it’s possible to call Elixir functions using Cypress?

1 Like

Awesome. I managed to get everything working last night, thanks for sharing those github action spec, that helped a ton at getting it working.

After adding a line to start chromedriver and background it, it seems to be working as expected.

I’m intrigued by cypress, it looks like a good approach and looks very clean. I’m going continue to explore there, but for now I’m glad to have hound rolling. I mostly wanted to stay in the elixir world due to the gray-box testing you mentioned, specifically for my project some “shopping cart”-type functionality that happens over several steps with items/quantities and persists between sessions. Its nice to be able to write assertions around what should be in the database.

Thanks so much again for the help!

I am not able to call any elixir functions within the code (I think), and pretty much black box the whole thing.

I have Cypress register an account, log out, log in, create and view resources etc. I made a 4th environment called :system_test and then can mock out anything I need to within the code (i.e. not requiring a confirmation email on sign up).

I try to cover as much as possible with the unit and integration tests because they are a lot faster, and then have Cypress for covering the happy path and the things that are impossible to test any other way… like if LiveView is still working after my last change :slight_smile:

1 Like