I believe I need to design a custom test runner (ExUnit) but I am not sure I really need to, hence asking here for guidance.
Say I have an arbitrary set of elements (defined externally, a supervised worker, etc.) and I want to permute them. Then:
for every permutation, the resulting value will need to be passed to every test file (i.e. /test/**/*_test.exs) when the tests are executed
repeat, until the permutations are consumed.
Say, I have this: Enum.zip([[1, 2, 3], [:a, :b, :c]])
The resulting list of tuples: [{1, :a}, {2, :b}, {3, :c}], will then be used one by one, for every test run. All the tests will run 3 times, in this case. Trying to obtain the same behavior as I would’ve run mix test three times, and for every run, I’d provide a custom value, for a given variable. Every tuple in the list will have to be available to every test, via a custom context variable, say: :my_tuple
Simply put, I’d like to be able to run all my tests multiple times, and for every run, the tests will receive a different values via some custom variables, variables defined ideally in a ExUnit.CaseTemplate.
I’ll paste here what I am going to ask on Slack, related to this question, for brevity. Thank you, for your attention.
Hi there - need to pick your brain about a design issue; ExUnit. I’ll oversimplify, but the core of the problem I am trying to solve is relatively straight forward (methinks’)
Say, hypotetically, I could run mix test in a “each” loop, for example:
run_tests = fn val ->
# Run assertions and collect the pass/fail results
# assert val > 0
Task.async(fn ->
:timer.sleep(:rand.uniform(1_000))
IO.puts(val) end)
end
(1..3)
|> Enum.map(&(run_tests.(&1)))
|> Enum.map(&Task.await/1)
The run_tests/1 would be responsible for running all the tests (for a given project), as if I would’ve run $ MYVAL=val mix test from different bash sessions, simultaneously, and for each round of tests the different “val” would be accessible to every test (as a context var, etc.)?!
The idea is, I want to run them from my app, not from a shell instance, so that I can collect the pass/fail results, exchange stats about tests, and so on.
The questions are:
Can this be done better, and without “re-inventing”/“stealing” the Mix.Compilers.Test.require_and_run/3 functionality?
Are there OS projects already doing something like this, or frameworks?
Right now, my plan is to implement the strategy above, as a mix task and Task.async bsically this: System.cmd "mix", ["test" | args], .... But I hope there is a better, or more elegant, solution?!