Exavier - Mutation Testing library for Elixir

dnlserrano

I created this lib to learn more about code compilation in Elixir, about ex_unit and also as an excuse to experiment working with the AST in Elixir.

It’s still very much a PoC, but I’d be happy to discuss about it. I think this can be useful to run as part of your CI pipeline if we get it to a good enough state, which is probably not where it’s at right now.

The work is inspired by mutant and pitest, but obviously less powerful ATM. Good thing is we can get there!

I have blogged about exavier here. The library GitHub repo with other info and ways of contributing (if you find it mildly interesting) is available here. Lots of good simple additions are in the works.

Thanks.

15 comments

#library #testing #exunit

13 2697 15

2021-01-04 16:52:03 UTC

Most Liked

Show Best Posts
Show All (oldest first)
Show All (newest first)

tmbb

This is very interesting, and I’ve been thinking of doing the same. From a cursory reading of the code, it looks like you mutate all operators in the file at the same time instead of one by one. Is that so? Shouldn’t one mutate one operator at the time to get more precise results?

To reduce the need to recompile a module lots of times, I’ve also tried to do it in a different way by mutating the Erlang AST in a way that allowed me to toggle mutations on and off without recompiling the code (see here: Introducing PureScript Erlang backend), but I couldn’t get very good error messages on my first try. It also made it harder to generate relevant mutations (and it made it impossible to generate mutations on macros)

Post #1

tmbb

When I tried to do it it didn’t seem as easy as you might think. I tried to traverse the AST while keeping a counter, so that I knew which operator to mutate, but I didn’t manage to make it work. I must have been doing something wrong.

Yes, that’s the main reason to mutate one at a time.

Recompiling your modules will in general be much slower than running the tests, so I think you’re optimizing for the right thing (i.e., reducing the need to recompile code, at the cost of possibly running more tests)

Because of what I said above, I don’t think you should even try to guess which tests to run…

Post #3

tmbb

I’ve just found out that recompiling modules lots of times (~ hundreds of times), even if the modules don’t change consumes memory from the BEAM’s literal allocator. This memory doesn’t seem to be garbage collected and it will fill up and cause errors. If you decide to generate mutants one at a time and each mutant requires a module recompilation, you might hit these errors…

I’m just warning you because I found out about this problem the hard way (i.e., when I tried to recompile my test suite about 300 times for 300 mutations in an elixir module). Fortunately, the way I’m generating mutants doesn’t require recompiling the code for each mutant, so I could work around it.

Post #7

Last Post!

luizfelipead

Hi @dnlserrano, sorry for not responding in a reasonable time.
But then I have to create, for each new test module, a new line in this configuration file? Is there a way for exavier to detect automatically these modules? (New idea for my future MR maybe? hehe)

Thanks for all the support.

Post #15