Darwin - Mutation Testing for Elixr

I’ve been working on a mutation testing library for Elixir for a while now. It’s named Darwin, because it generates mutants in your code, which are then killed by your test suite (the analogy to evolution by natural selection should be pretty obvious at this point). The docs aren’t yet as organized as they should be, and test reporting is still very incomplete, so I haven’t released it on Hex yet. But the functionality is mostly done, and I thiknk it’s time I created a new thread in the Libraries category.

You can find the code on GitHub, along with an example on how to use Darwin on a real Elixir module.

• GitHub for Darwin: GitHub - tmbb/darwin: Darwin mutation testing framework · GitHub
• GitHub for the demo project: GitHub - tmbb/demo_darwin: Demo project for the Darwin mutation testing framework · GitHub (an example that applies Darwin to a real module - a version of the Enum module, but renamed so that mutating it doesn’t break the language)

Why another mutation testing library?

We already have Exavier by @dnlserrano, so why another one? First, both libraries are still in a very early development stage, so it’s not clear at this point that one of them is clearly better and the other one should be abandoned. Second, both libraries follow very different design philosophies, so they can’t exactly be merged or something like that.

I plan on writing a series of posts comparing Exavier to Darwin and defending some of Darwin’s design decisions.

Can I use Darwin in my real project right now?

Probably yes. Code is only mutated in development, so your production code should be unaffected. On the other hand, Darwin has a number of problems:

1. Test reporting is not great
2. Running all your mutation tests might take a while
3. Darwin uses some nasty private ExUnit APIs (Exavier does the same, by the way; ExUnit is not really suitable for use as a library)

Can I use Darwin for Mutation Testing of Erlang Code?

Currenctly HTML reporting is not supported for .erl files (only .ex) files. This can be changed though. Darwin mutates the code at the level of the fully expanded Erlang AST, so it should be an excellent choice to mutate Erlang libraries. The main drawback in this use case is that you have to install Elixir and use ExUnit in order to test mutations in your own Erlang project.

It’s likely that Darwin will be easy to integrate with pure erlang test frameworks, as long as someone writes the necessary glue code. Sometimes I think of rewriting Darwin in Erlang, but Elixir’s conveniences make it much easier to write in Elixir than in Erlang, so I don’t think I’ll go that way until Darwin is stable enough.