mix test --failed --max-failures 1 --seed 0 --trace
Mix alternates between two tests, always the same two. It looks like it may be clock based as the test changes every 1 or 2 seconds (hard to measure since the command takes some time).
For info, if I run mix test I have the following stats: 46 tests, 12 failures, 1 skipped.
But the two different tests that the seeded command runs are always the same two.
Those two tests are from different files. I would not mind too much if the randomization would still help me to work on the same module.
--seed - seeds the random number generator used to randomize the order of tests; --seed 0 disables randomization so the tests in a single file will always be ran in the same order they were defined in
since your tests are in separate files, the order in which they run isn’t related to disabling seeds using --seed 0
--trace essentially disables async tests to run the test files in what I believe is alphabetical order. That should mean that with --seed 0 the first file executes all tests and it should wait to execute the next file, top down.
It’s not clear what you’re trying to test to know if there are side effects bubbling between the executions. Your tests in each file could have issues executing sequentially. The output of trace should group the failures to each file and test. Using --failed will only execute the failed tests and it’s possible that option is throwing things off. Try running the original command without it: mix test --seed 0 --trace. That’ll run all tests sequentially top down and output which tests are failing.
I have that command bound to a shortcut (with rotation of the seed but let’s keep it simple). It’s a very efficient workflow. It’s generally doing what I want, meaning always running the same failed test until it passes, and the goes to a next test and stick to it until it passes, and so on.
But from Elixir 1.17 (I guess) I have been seeing that new behaviour on occasion, very rarely. But today it’s consistently happening.
@srcoulombe indeed the docs do not tell that the seed option impacts the order of files. But it seems to do so.
@w0rd-driven yeah it seems related to the use of --failed but the workflow is based around that specific flag.
I think the only reasonable way to debug such an issue would be to try to reproduce it in a side-effect free small project, otherwise there is a very high chance the problem lies in the codebase itself and not in the elixir testing functionality.
Do you think it would be possible to reproduce this issue in a project we could take a look at?
Not sure as it’s not happenning on any other project at this moment, and I use this kind of command on at least 20 repos. Though it’s not the first time.
And yes it seems related to the codebase so no luck.
I guess I’ll do some debugging on ExUnit codebase when I have time to try to understand.
