How can I control every step of tests running with ExUnit

I’m developing web automation tools using wallaby.
One of the challenges in web automation is that sometimes the tests are failing for no reason and when I run it again it passes.
When I run a set of tests especially when I run it virtual machine I need a way to run a set of tests at one time and run some of them again in case of failure.

Another thing, because I’m using VM to run the tests I need a way to send messages to Slack so I can easily monitor the results.
I’m sending a few kinds of messages: before all the tests are started there is a message that the running start, before and after every test start I’m sending a message that the tests starts/ends and the result (failure/success),
finally, after all the tests completed I need to send a summary of the results.

Right now I’m managing all things with a bash file that gets the list of the tests and runs all of them one by one.
The messages before and after every test I’m sending from ExUnit formatter.

I’m looking for a better solution to manage this system without the bash file by using just the elixir system.

What I’m looking for is a way to run the all tests in one run in the order that I want and to get access to the action before the tests started between them and before they finished.

I would love your help.
THX

When I run a set of tests especially when I run it virtual machine I need a way to run a set of tests at one time and run some of them again in case of failure.

If you want to retry failed tests a few times, instead of running mix test, you can run mix test || mix test --failed || mix test --failed

Another thing, because I’m using VM to run the tests I need a way to send messages to Slack so I can easily monitor the results.

Your CI provider can probably handle this for you. If you have rolled this by hand, you’ll need to build that yourself.

What I’m looking for is a way to run the all tests in one run in the order that I want and to get access to the action before the tests started between them and before they finished.

Not really sure what you’re asking here, but there are ExUnit callbacks you can hook into for different parts of the test lifecycle: ExUnit.Callbacks — ExUnit v1.11.3

1 Like

I believe if you use test --seed 0 your tests will run in the same order each time. Perhaps that will help? From mix help test:

*  --seed - seeds the random number generator used to randomize the order
    of tests; --seed 0 disables randomization

To be precise any seed will run the tests in the same order, what is special about --seed 0 is that they are not shuffled

are you passing the same seed as when the tests failed? That is key to debugging tests

If you want to retry failed tests a few times, instead of running mix test , you can run mix test || mix test --failed || mix test --failed

This solution is not helping me because I one to run multiple tests with one short command, the settings that test will run more than ones should be in the code.

Your CI provider can probably handle this for you. If you have rolled this by hand, you’ll need to build that yourself.

I still didn’t find a way to send a message before the list of tests or even files are running

Not really sure what you’re asking here, but there are ExUnit callbacks you can hook into for different parts of the test lifecycle: ExUnit.Callbacks — ExUnit v1.11.3

I’m looking for a way to add code after the ExUnit started but before the tests are starting and the same in the end.
(maybe some kind of listener the can do action according to ExUnit messages - similar to formatters but a little bit dipper)
The callbacks don’t help me because they are just per test (and I also have to set them for each one)

Maybe this is naive but at least the former is somewhat done by having code in the test/test_helper.exs file which seems to be the entry point of all tests before they are loaded / started. Worth the shot?

I thought about it but I didn’t find the way to run tests from there or to access the tests that are supposed to run

Not sure I understood from your previous messages: why must you enforce a precise test order execution?

I’m running a set of tests (regression tests) in one command, most of them should run again if they are failed but few of them can only run ones.
I also send the slack a message that the set of the tests are starting now.

Still not sure I understand completely. Could you give one more illustrated example?

And I know this is not exactly what you need, but have you tried mix test.watch? It also allows plugging additional mix tasks which should cover your Slack message sending requirement.

It’s hard to do an example because it’s big system and a lot of code

I think what you may probably need is to run your tests programmatically.

Read this thread,
ExUnit tests from a module instead of command line/mix

and the implementation of the test task in Mix. (look for the run/1 function)
elixir/test.ex at master · elixir-lang/elixir · GitHub

3 Likes

It seems complicated and he never updated if he ever gets something working for this but it seems the right direction.
I’ll be happy to get some more details and example but in the meantime I’ll try it.
THX

1 Like

If I were you i would copy Mix.Task.Test and strip it down to only load tests and run them. From there you can extended to do what you have described in this post.

1 Like

More on the subject,
If you are hacking Mix.Tasks.Test
read this article by @dnlserrano
dnlserrano.github.io/2019-05-26-exunit-deep-dive.markdown at master · dnlserrano/dnlserrano.github.io · GitHub

Also a recent library from @devonestes seem to do something similar.
muzak/runner.ex at 63166c0a2526cda60787c22127be8645cc13e4dd · devonestes/muzak · GitHub

I haven’t tested it, but this is this looks like the minimum code required in this test in this library.
nimler/test_all.exs at master · wltsmrz/nimler · GitHub

How did I find out all this?
https://github.com/search?q="require_and_run"&type=code

3 Likes