Hi!

As some may already have heard, a coworker and me have taken over a rather large codebase a while ago and as there is no-one walking as through it, we do a lot of discovery on our own.

And even though there is a nominal test coverage of ~90% the following problems exist:

We already learned, that for some reason a mix test --stale runs several tests again that are unrelated and we assume that this is due to massive dependency cycles within the codebase. This is not the problem I want to discuss here.

Sadly, we also discovered, that sometimes after a change mix test --stale is not running relevant tests. We assume that this is due to massive (ab)use of mocking.

Even worse: We removed whole modules (implementations) by accident, and the full test suite still passed, and we therefore lost some, if not all, confidence in the suite.

To regain confidence I wanted to run some mutation testing and tried exavier (by @dnlserrano) and muzak (by @devonestes). Sadly neither has received any updates in the last 18+ months, and neither worked with Elixir 1.14 (which we upgraded to at work).

Is anyone aware of an actively developed mutation testing library that is compatible with 1.14 and does not require massive code changes?

Also, exaviers direct mapping between test files and modules would be something that is actually more a hinderence than a benefit, as even though (mostly) the mapping exists in a 1:1 mannor, it doesn’t match exaviers inference and we had to do a lot of manual overrides (there are 700 modules, my guess is that only 100 could be infered by exavier).

And a semi related question: Do I understand (the idea behind) mutation testing correctly, that it is made exactly for this kind of situation, to regain/strengthen confidence in the testsuite?