How can I control every step of tests running with ExUnit

Questions / Help
,
1

I’m developing web automation tools using wallaby.
One of the challenges in web automation is that sometimes the tests are failing for no reason and when I run it again it passes.
When I run a set of tests especially when I run it virtual machine I need a way to run a set of tests at one time and run some of them again in case of failure.

Another thing, because I’m using VM to run the tests I need a way to send messages to Slack so I can easily monitor the results.
I’m sending a few kinds of messages: before all the tests are started there is a message that the running start, before and after every test start I’m sending a message that the tests starts/ends and the result (failure/success),
finally, after all the tests completed I need to send a summary of the results.

Right now I’m managing all things with a bash file that gets the list of the tests and runs all of them one by one.
The messages before and after every test I’m sending from ExUnit formatter.

I’m looking for a better solution to manage this system without the bash file by using just the elixir system.

What I’m looking for is a way to run the all tests in one run in the order that I want and to get access to the action before the tests started between them and before they finished.

I would love your help.
THX

2

When I run a set of tests especially when I run it virtual machine I need a way to run a set of tests at one time and run some of them again in case of failure.

If you want to retry failed tests a few times, instead of running mix test, you can run mix test || mix test --failed || mix test --failed

Another thing, because I’m using VM to run the tests I need a way to send messages to Slack so I can easily monitor the results.

Your CI provider can probably handle this for you. If you have rolled this by hand, you’ll need to build that yourself.

What I’m looking for is a way to run the all tests in one run in the order that I want and to get access to the action before the tests started between them and before they finished.

Not really sure what you’re asking here, but there are ExUnit callbacks you can hook into for different parts of the test lifecycle: ExUnit.Callbacks — ExUnit v1.11.3

1 Like
3

I believe if you use test --seed 0 your tests will run in the same order each time. Perhaps that will help? From mix help test:

*  --seed - seeds the random number generator used to randomize the order
    of tests; --seed 0 disables randomization
1 Like
4

To be precise any seed will run the tests in the same order, what is special about --seed 0 is that they are not shuffled

5

are you passing the same seed as when the tests failed? That is key to debugging tests

6

If you want to retry failed tests a few times, instead of running mix test , you can run mix test || mix test --failed || mix test --failed

This solution is not helping me because I one to run multiple tests with one short command, the settings that test will run more than ones should be in the code.

Your CI provider can probably handle this for you. If you have rolled this by hand, you’ll need to build that yourself.

I still didn’t find a way to send a message before the list of tests or even files are running

Not really sure what you’re asking here, but there are ExUnit callbacks you can hook into for different parts of the test lifecycle: ExUnit.Callbacks — ExUnit v1.11.3

I’m looking for a way to add code after the ExUnit started but before the tests are starting and the same in the end.
(maybe some kind of listener the can do action according to ExUnit messages - similar to formatters but a little bit dipper)
The callbacks don’t help me because they are just per test (and I also have to set them for each one)

8

Maybe this is naive but at least the former is somewhat done by having code in the test/test_helper.exs file which seems to be the entry point of all tests before they are loaded / started. Worth the shot?

9

I thought about it but I didn’t find the way to run tests from there or to access the tests that are supposed to run

10

Not sure I understood from your previous messages: why must you enforce a precise test order execution?

11

I’m running a set of tests (regression tests) in one command, most of them should run again if they are failed but few of them can only run ones.
I also send the slack a message that the set of the tests are starting now.

12

Still not sure I understand completely. Could you give one more illustrated example?

And I know this is not exactly what you need, but have you tried mix test.watch? It also allows plugging additional mix tasks which should cover your Slack message sending requirement.

13

It’s hard to do an example because it’s big system and a lot of code

14

I think what you may probably need is to run your tests programmatically.

Read this thread,
ExUnit tests from a module instead of command line/mix

and the implementation of the test task in Mix. (look for the run/1 function)
elixir/test.ex at master · elixir-lang/elixir · GitHub

3 Likes
15

It seems complicated and he never updated if he ever gets something working for this but it seems the right direction.
I’ll be happy to get some more details and example but in the meantime I’ll try it.
THX

1 Like
16

If I were you i would copy Mix.Task.Test and strip it down to only load tests and run them. From there you can extended to do what you have described in this post.

1 Like
17

More on the subject,
If you are hacking Mix.Tasks.Test
read this article by @dnlserrano
dnlserrano.github.io/2019-05-26-exunit-deep-dive.markdown at master · dnlserrano/dnlserrano.github.io · GitHub

Also a recent library from @devonestes seem to do something similar.
muzak/runner.ex at 63166c0a2526cda60787c22127be8645cc13e4dd · devonestes/muzak · GitHub

I haven’t tested it, but this is this looks like the minimum code required in this test in this library.
nimler/test_all.exs at master · wltsmrz/nimler · GitHub

How did I find out all this?
https://github.com/search?q="require_and_run"&type=code

3 Likes
18

@Yonis Did you ever find a solution to creating a specific order to the tests? I tried the --seed option, and I’ve been trying to figure out how to possibly create Wallaby tests and import them into a sort of ‘master_test.exs’ file but haven’t figured that out yet. My issue is that the concurrent tests interfere with each other and I want to use them in a more ordered manner. Any thoughts or possible solutions?

19

Have you considered just tagging those tests differently and invoking them separately e.g. mix test --only interactive?

20

I did try tagging them as well. I ended up finding a solution here:

Though - the Code.load_file/2 was deprecated and I’m using Code.require_file/2 instead.

I made a post on stackoverflow asking why that’s the case here but maybe that questions is better asked in elixirforum…so…

Why do test modules not import/alias like non-test modules and instead I have to use Code.require_file("test/path.exs) in the test_helper.exs in order for them to work?

21

You don’t need to do that. You can do this in your mix.exs:

  def project do
    [
      app: :your_app,
      version: "1.2.3",
      elixir: "~> 1.14.0",
      elixirc_paths: elixirc_paths(Mix.env()),     # 👈 NOTICE THIS
      compilers: [:phoenix] ++ Mix.compilers(),
      start_permanent: Mix.env() == :prod,
      aliases: aliases(),
      deps: deps()
    ]
  end

  defp elixirc_paths(:test), do: ["lib", "test/support"]
  defp elixirc_paths(_), do: ["lib"]

And then put everything that you want to have access to in tests in test/support. I don’t think you should put it in the root test/ directory however.

2 Likes