ExUnit: changing test's context values at run-time, before running all the tests

Hi there,

I believe I need to design a custom test runner (ExUnit) but I am not sure I really need to, hence asking here for guidance.

Say I have an arbitrary set of elements (defined externally, a supervised worker, etc.) and I want to permute them. Then:

  • for every permutation, the resulting value will need to be passed to every test file (i.e. /test/**/*_test.exs) when the tests are executed
  • repeat, until the permutations are consumed.

Say, I have this: Enum.zip([[1, 2, 3], [:a, :b, :c]])

The resulting list of tuples: [{1, :a}, {2, :b}, {3, :c}], will then be used one by one, for every test run. All the tests will run 3 times, in this case. Trying to obtain the same behavior as I would’ve run mix test three times, and for every run, I’d provide a custom value, for a given variable. Every tuple in the list will have to be available to every test, via a custom context variable, say: :my_tuple

Simply put, I’d like to be able to run all my tests multiple times, and for every run, the tests will receive a different values via some custom variables, variables defined ideally in a ExUnit.CaseTemplate.

I was trying to find some inspiration in this thread: Testing a Mix app with ExUnit: changing config values set at compile time, but in my case the values received by tests will be set dynamically, not at compile time.

Thank you,

I’ll paste here what I am going to ask on Slack, related to this question, for brevity. Thank you, for your attention.

Hi there - need to pick your brain about a design issue; ExUnit. I’ll oversimplify, but the core of the problem I am trying to solve is relatively straight forward (methinks’)

Say, hypotetically, I could run mix test in a “each” loop, for example:

run_tests = fn val -> 
  # Run assertions and collect the pass/fail results
  # assert val > 0
  Task.async(fn -> 
    IO.puts(val) end)

|> Enum.map(&(run_tests.(&1)))
|> Enum.map(&Task.await/1)

The run_tests/1 would be responsible for running all the tests (for a given project), as if I would’ve run $ MYVAL=val mix test from different bash sessions, simultaneously, and for each round of tests the different “val” would be accessible to every test (as a context var, etc.)?!

The idea is, I want to run them from my app, not from a shell instance, so that I can collect the pass/fail results, exchange stats about tests, and so on.

The questions are:

  • Can this be done better, and without “re-inventing”/“stealing” the Mix.Compilers.Test.require_and_run/3 functionality? :slight_smile:
  • Are there OS projects already doing something like this, or frameworks?

Right now, my plan is to implement the strategy above, as a mix task and Task.async bsically this: System.cmd "mix", ["test" | args], .... But I hope there is a better, or more elegant, solution?!

many thanks!

Updating this thread with the summary of what I implemented, until someone else will stumble on it and come up with a different/better solution.

  • mix task to run System.cmd "mix", ["test" | args ..., as a worker, in a :pollboy pool
  • passing the test suite specific args via Mix’ env
  • wrote a custom ExUnit formatter, for collecting the tests stats; pas/fail/etc. Mostly based on elixir/lib/ex_unit/lib/ex_unit/cli_formatter.ex

it works well so far, and can run parallel test-suites; which is what I was looking for :slight_smile: