I recently added a test that mistakenly modified some global state and caused failures in later tests. This failure occurred on CI but not locally. I want to recreate it locally by running:
$ mix test test/c_test.exs test/b_test.exs --trace
Where c_test.exs contains the state-modifying code. But the tests are always run with b first and c second. Even renaming c_test.exs to a_test.exs did not change the order!
I expected that mix test would honour the order given on the command line but I guess it ends up being re-ordered internally?
This question was already asked in 2017 but that thread became derailed. However, in my case the shared state was accidental and it seems necessary to be able to specify test file order to track this problem down. Is there a workaround?
Thank you @LostKobrakai and @Marcus for suggesting setting the seed. My understanding is that the seed only affects ordering of the tests run within the file. The documentation says:
--seed 0 disables randomization so the tests in a single file will always be ran in the same order they were defined in
I’d never really noticed before but the test files run with --trace do seem to always run in the same order but you can see that the individual tests are in random order between runs.