Feature Requests - ExUnit

I’d like to discuss two proposals to improve ExUnit and make it easier to write alternative testing frameworks which cooperate nicely with ExUnit.

Feature Request #1 - Access to the list of sync and async test modules

There should be a way of accessing the list of sync and async modules in the ExUnit.Server genserver.

Basically, the equivalent of the following code:

:sys.get_state(ExUnit.Server)

Motivation

Currently, the only way of telling ExUnit which modules to run is by adding modules to the stateful ExUnit.Server, using the add_sync_module/1 and add_async_module/1 functions.

This API is slightly inconvenient but very easy to use. However, as far as I can tell, there is no way of retrieving the sync and async modules that have been registered with ExUnit.Server. Those modules are registered when the test suite is compiled, and are “consumed” from the genserver when they are run. After the test suite has run, the genserver is empty and must be filled again. One way of refilling the genserver is to compile the test files again, but that has two problems:

  1. It’s very slow - sometimes running a test suite can take milliseconds and compiling the same test suite can take 1 or more seconds

  2. Compiling the same module a repeatedly can cause the BEAM’s literal allocator to consume too much memory and crash the BEAM (it happens after about ~250 recompilations in my case)

I’m having both of these problems in my mutation testing framework. I can generate hundreds of mutations in the same module, which means running the test suite must be very fast and can’t saturate the literal allocator.

It doesn’t change the semantics of ExUnit.

Implementation

This feature as it stands is trivial to implement.

One needs only a new genserver call in the ExUnit.Server module.

Criticisms

ExUnit.Server is probably meant to be an implementation detail.

I’m not sure we are meant to meddle with it or depend on it being present.

Feature Request #2 - Stop the test suite right after a test fails

Currently we can configure ExUnit to stop after a number of failures, using the :max_failures option. However, this option doesn’t do what one might think it does. Instead of stopping the test suite (including modules that are being tested in parallel) just after the test has failed, it will only stop the test suite after the current test case has run.

I think the test suite should stop after the number of failures has been reached instead of the current behavior.

Motivation

Same motivation as above. I’m writing a mutation testing framework which has to run a test suite hundreds or thousands of times. If I can abort the tests as soon as a single test has failed, it can save a lot of time.

This is also true for other situations, like continuous integration pipelines.

Implementation

There are several ways of implementing this. The obvious one is to spawn all processes related to the test suite from a single master process and send a kill signal to that process, but there might be others.

Criticisms

This feature probably requires some radical refactoring of ExUnit’s code base. It also changes the semantics of ExUnit and I don’t think it can be done without at least changing the map returned by ExUnit.run(). Currently the map has the following shape:

%{excluded: 0, failures: 86, skipped: 0, total: 161}

We would have to add a new key to the map to describe tests that weren’t run because one of the tests failed.

5 Likes

Pretty good proposal overall. I’ll ask a couple questions where things were unclear for me.

Feature Request #1

Could you keep track of the modules yourself? For example, test’s could choose to use fuzzing with something like:

defmodule MyTest do
  use ExUnit.Case
  use Fuzzing
end

And then, in the after compile callback, Fuzzing could keep track of the modules itself. Then it could presumably resubmit the modules to ExUnit. Might still need a proposal since add_sync_module/1 and add_async_module/1 aren’t public API.

Feature Request #2

Why not put them in skipped?

2 Likes

Yes, but I think it would be better if I could run the tests from the “outside” instead of injecting my own code. But just making the add_*_module() functions public would probably be enough.

They could be put in skipped. It’s the same to me.

EDIT: OTOH, maybe the best solution is to fork ex_unit into Darwin, place it under the Darwin namespace and turn it into my own test runner! No need to change ExUnit.

1 Like

I’d like to have a way to determine whether the test failed or not in the ‘on_exit’ callback.

1 Like

@tmbb Did you settle on a different approach or are you still trying to get something like what you proposed into ExUnit?

I’m still trying to make this work, but unfortunately I cant compile Elixir on my machine supposedly because of memory issues, but probably something else (my machine has plenty of memory), and at the moment I’m not interested in tying to solve the compilation problems. This means I won’t submit any PRs to ExUnit for the time being.

But I’m still trying to find a way of making the test suite stop when the first test fails. I’m running Darwing on the Elixir Enum module (actually a copy of the Enum module called Enom, because if I mutate the real Enum module – which I can do – I break the language). Many mutations cause the functions in the Enum module to enter infinite loops, which is very inconvenient. If I have 60 tests which fail with a timeout of 3s, It’s ~3mins untill all the test finaly fail. This means I waste 3mins killing that particular mutation (Darwin fins ~300 mutations in the Enum module). If I could fail the whole test suite after the first test, I could kill the mutation in 3s…

1 Like

On a more positive note, I’ve just noticed I can detect whether a test fails using a Formatter. So, if there’s a way of killing the test suite “remotely”, I could do that from the formatter… I have to look into it.

2 Likes

On a negative note again, it’s NOT possible to kill the test suite inside the formatter because the formatter is async… This means I can’t guarantee I’m closing the right test suite. It’s possible that Darwin is already on the next test suite, and killing it causes errors.

This is my conclusion for running a very primitive “test suite killer” in preactice…

1 Like