Running a single test in ExUnit and stopping a test suite prematurely

Questions / Help
1

In mutation testing, you want to run tests until you kill a mutant. After killing the mutant, you don’t need to run any more tests. You also have to run the whole test suite for each mutation, and there can be hundreds or thousands of mutations. This means that any time spent running test that don’t need to be run will make the test suite much, much longer.

I’m interested in the following questions about ExUnit (I’d like to be able to reuse Exunit for mutation testing):

  1. Is there a way of running a single test (by module name and test name) programatically? That way I could store in a database the single tests that kills a certain mutant and run that first. If the test fails, I’m done and I can proceed to the next mutation.

  2. Currently we can give a maximum number of failures, which will cause the test suite to be aborted. However, it isn’t aborted immediately after the first failure. It is aborted after the first test case finishes. Actually, from my cursory reading of the code it’s aborted after the longest running concurrent test case finishes. Shouldn’t we abort the whole test suite immediately?

2

You can call mix test path/to/the/test.exs and mix test path/to/the/test.exs:83 to target a specific file / file:line.

3

This might help you ExUnit tests from a module instead of command line/mix

1 Like
4

I’m actually finding it very hard to do something much simpler. Something like this:

I want to run the test suite, and after it finishes I want to run it again with a different global config value. Ideally, I’d like something like this:

for mutation <- mutations do
  set_mutation(mutation)
  ExUnit.run()
end

My problem is that ExUnit.run() seems incapable of detecting test files after running the test suite the first time, and no tests are found after the same mutation. What’s the way around it?

5

Did you try @rodrigues solution ? It is 5 lines of code.

6

I did try that. The tests are not rerun:

2:45:22.979 [info]  {:ok, %{excluded: 0, failures: 3, skipped: 0, total: 77}}

12:45:22.979 [info]  Testing mutant: {Mod, 0, 1}

12:45:22.989 [info]  {:ok, %{excluded: 0, failures: 0, skipped: 0, total: 0}}

12:45:22.989 [info]  Testing mutant: {Mod, 0, 0}

12:45:22.999 [info]  {:ok, %{excluded: 0, failures: 0, skipped: 0, total: 0}}

The first time they are run as they should. But after the first time it seems like the tests are not detected. I’ve tried using Code.unrequire_files(test_files) after running the test suite once, but it still doesn’t work.

7

Ok, I’ve found the problem. It turns out ExUnit has a very stateful API which reads the module names for the tests from the ExUnit.Server. This server is populated during compilation of the test suite. When you run the test suite, ExUnit takes values from the server. After running the test suite, the server is empty, so when you call ExUnit.run() no test are run.

Requiring the test files doesn’t work, although recompiling the test files work (I have to turn off compiler warnings so that I don’t get overwhelmed with warning messages telling me that modules were being overwritten.

So:

  1. I have to recompile the test suite to run the tests again
  2. I have to turn off compiler warnings
  3. There should be a way to get the state of the ExUnit.Server before running the tests and away to “refill” the modules again with the old state without compiling the test suite.

I’m having all this work to make my mutation testing framework (Darwin) work without recompiling the user code each time the test suite is run and I find out that I have to recompile the test suite each time I run it… I have to find out if there is a more performant way of doing this.

2 Likes