Test *file* run order with mix test

Questions / Help
,
1

Hi all,

I recently added a test that mistakenly modified some global state and caused failures in later tests. This failure occurred on CI but not locally. I want to recreate it locally by running:

$ mix test test/c_test.exs test/b_test.exs --trace

Where c_test.exs contains the state-modifying code. But the tests are always run with b first and c second. Even renaming c_test.exs to a_test.exs did not change the order! :exploding_head:

I expected that mix test would honour the order given on the command line but I guess it ends up being re-ordered internally?

This question was already asked in 2017 but that thread became derailed. However, in my case the shared state was accidental and it seems necessary to be able to specify test file order to track this problem down. Is there a workaround?

2

Afaik the order of tests is determined by the seed. So you should be able to get deterministic order by setting the same seed between invocations.

1 Like
3

Take a look on the seed option/flag in the documentation of mix test — Mix v1.14.1 and ExUnit — ExUnit v1.15.0-dev

4

Thank you @LostKobrakai and @Marcus for suggesting setting the seed. My understanding is that the seed only affects ordering of the tests run within the file. The documentation says:

--seed 0 disables randomization so the tests in a single file will always be ran in the same order they were defined in

I’d never really noticed before but the test files run with --trace do seem to always run in the same order but you can see that the individual tests are in random order between runs.

5

AFAIK the order of files is out of control, as ExUnit uses Kernel.ParallelCompiler.require/2 - here

The workaround that worked for me was to add Process.sleep/1 right after defmodule, i.e.

defmodule Core.Test01 do
  Process.sleep(1_000)
  use ExUnit.Case
  ...
end