ExUnit: changing test's context values at run-time, before running all the tests

Hi there,

I believe I need to design a custom test runner (ExUnit) but I am not sure I really need to, hence asking here for guidance.

Say I have an arbitrary set of elements (defined externally, a supervised worker, etc.) and I want to permute them. Then:

  • for every permutation, the resulting value will need to be passed to every test file (i.e. /test/**/*_test.exs) when the tests are executed
  • repeat, until the permutations are consumed.

Say, I have this: Enum.zip([[1, 2, 3], [:a, :b, :c]])

The resulting list of tuples: [{1, :a}, {2, :b}, {3, :c}], will then be used one by one, for every test run. All the tests will run 3 times, in this case. Trying to obtain the same behavior as I would’ve run mix test three times, and for every run, I’d provide a custom value, for a given variable. Every tuple in the list will have to be available to every test, via a custom context variable, say: :my_tuple

Simply put, I’d like to be able to run all my tests multiple times, and for every run, the tests will receive a different values via some custom variables, variables defined ideally in a ExUnit.CaseTemplate.

I was trying to find some inspiration in this thread: Testing a Mix app with ExUnit: changing config values set at compile time, but in my case the values received by tests will be set dynamically, not at compile time.

Thank you,
Florin

I’ll paste here what I am going to ask on Slack, related to this question, for brevity. Thank you, for your attention.

Hi there - need to pick your brain about a design issue; ExUnit. I’ll oversimplify, but the core of the problem I am trying to solve is relatively straight forward (methinks’)

Say, hypotetically, I could run mix test in a “each” loop, for example:

run_tests = fn val -> 
  # Run assertions and collect the pass/fail results
  # assert val > 0
  Task.async(fn -> 
    :timer.sleep(:rand.uniform(1_000))
    IO.puts(val) end)
end

(1..3) 
|> Enum.map(&(run_tests.(&1)))
|> Enum.map(&Task.await/1)

The run_tests/1 would be responsible for running all the tests (for a given project), as if I would’ve run $ MYVAL=val mix test from different bash sessions, simultaneously, and for each round of tests the different “val” would be accessible to every test (as a context var, etc.)?!

The idea is, I want to run them from my app, not from a shell instance, so that I can collect the pass/fail results, exchange stats about tests, and so on.

The questions are:

  • Can this be done better, and without “re-inventing”/“stealing” the Mix.Compilers.Test.require_and_run/3 functionality? :slight_smile:
  • Are there OS projects already doing something like this, or frameworks?

Right now, my plan is to implement the strategy above, as a mix task and Task.async bsically this: System.cmd "mix", ["test" | args], .... But I hope there is a better, or more elegant, solution?!

many thanks!

Updating this thread with the summary of what I implemented, until someone else will stumble on it and come up with a different/better solution.

  • mix task to run System.cmd "mix", ["test" | args ..., as a worker, in a :pollboy pool
  • passing the test suite specific args via Mix’ env
  • wrote a custom ExUnit formatter, for collecting the tests stats; pas/fail/etc. Mostly based on elixir/lib/ex_unit/lib/ex_unit/cli_formatter.ex

it works well so far, and can run parallel test-suites; which is what I was looking for :slight_smile:

HTH

2 Likes