Mutation testing on Elixir 1.14?

NobbZ

Hi!

As some may already have heard, a coworker and me have taken over a rather large codebase a while ago and as there is no-one walking as through it, we do a lot of discovery on our own.

And even though there is a nominal test coverage of ~90% the following problems exist:

We already learned, that for some reason a mix test --stale runs several tests again that are unrelated and we assume that this is due to massive dependency cycles within the codebase. This is not the problem I want to discuss here.

Sadly, we also discovered, that sometimes after a change mix test --stale is not running relevant tests. We assume that this is due to massive (ab)use of mocking.

Even worse: We removed whole modules (implementations) by accident, and the full test suite still passed, and we therefore lost some, if not all, confidence in the suite.

To regain confidence I wanted to run some mutation testing and tried exavier (by @dnlserrano) and muzak (by @devonestes). Sadly neither has received any updates in the last 18+ months, and neither worked with Elixir 1.14 (which we upgraded to at work).

Is anyone aware of an actively developed mutation testing library that is compatible with 1.14 and does not require massive code changes?

Also, exaviers direct mapping between test files and modules would be something that is actually more a hinderence than a benefit, as even though (mostly) the mapping exists in a 1:1 mannor, it doesn’t match exaviers inference and we had to do a lot of manual overrides (there are 700 modules, my guess is that only 100 could be infered by exavier).

And a semi related question: Do I understand (the idea behind) mutation testing correctly, that it is made exactly for this kind of situation, to regain/strengthen confidence in the testsuite?

11 comments

#testing #mutation-testing

1 1579 11

2022-12-08 07:37:41 UTC

First Post!

Show Best Posts
Show All (oldest first)
Show All (newest first)

al2o3cr

I don’t think mutation testing is necessarily going to help much given the problems you’ve described - the changes it makes (random example: ROR3 from Exavier) are mostly function-scale. That kind of mutation is good for making sure your tests cover the < and the = situations for an <= comparison, but “we removed a whole module and the tests still passed” is a bigger issue.

My interpretation of your situation is that the codebase has fallen into a mocking trap; there are real implementations that are replaced with a mock EVERYWHERE and not tested individually. Some ways to address that:

write specific tests for the thing that’s being mocked everywhere. Ideally there would be a corresponding test for every scenario that’s set up in the mocks, to demonstrate that the real thing actually does the what the mocks are pretending to do.
write higher-level integration tests that don’t use mocks. For a legacy codebase, the “happy path” is a good place to start. These will be slower than isolated unit tests, so you may want to tag them and run them as a separate CI step.

Post #1

Most Liked

devonestes

You’re right that Muzak isn’t as supported as I’d like it to be (for reasons), but I am releasing a new version this weekend that works for 1.14, and should hopefully work for all future Elixir versions. I’m hopeful that I’ve gotten it off of any private APIs at this point.

Yes, the situation that you describe where you can essentially delete the “code under test” entirely and the tests still pass is basically the canonical example of what mutation testing helps with.

Post #10