Darwin - Mutation Testing library for Elixir (and Erlang!)

Your Libraries & OS Mentoring Libraries
,
1

Darwin - Mutation Testing for Elixr

I’ve been working on a mutation testing library for Elixir for a while now. It’s named Darwin, because it generates mutants in your code, which are then killed by your test suite (the analogy to evolution by natural selection should be pretty obvious at this point). The docs aren’t yet as organized as they should be, and test reporting is still very incomplete, so I haven’t released it on Hex yet. But the functionality is mostly done, and I thiknk it’s time I created a new thread in the Libraries category.

You can find the code on GitHub, along with an example on how to use Darwin on a real Elixir module.

Why another mutation testing library?

We already have Exavier by @dnlserrano, so why another one? First, both libraries are still in a very early development stage, so it’s not clear at this point that one of them is clearly better and the other one should be abandoned. Second, both libraries follow very different design philosophies, so they can’t exactly be merged or something like that.

I plan on writing a series of posts comparing Exavier to Darwin and defending some of Darwin’s design decisions.

Can I use Darwin in my real project right now?

Probably yes. Code is only mutated in development, so your production code should be unaffected. On the other hand, Darwin has a number of problems:

  1. Test reporting is not great
  2. Running all your mutation tests might take a while
  3. Darwin uses some nasty private ExUnit APIs (Exavier does the same, by the way; ExUnit is not really suitable for use as a library)

Can I use Darwin for Mutation Testing of Erlang Code?

Currenctly HTML reporting is not supported for .erl files (only .ex) files. This can be changed though. Darwin mutates the code at the level of the fully expanded Erlang AST, so it should be an excellent choice to mutate Erlang libraries. The main drawback in this use case is that you have to install Elixir and use ExUnit in order to test mutations in your own Erlang project.

It’s likely that Darwin will be easy to integrate with pure erlang test frameworks, as long as someone writes the necessary glue code. Sometimes I think of rewriting Darwin in Erlang, but Elixir’s conveniences make it much easier to write in Elixir than in Erlang, so I don’t think I’ll go that way until Darwin is stable enough.

7 Likes
2

Ooo it’s released, awesome!

1 Like
3

It’s not released. You can’t find it on hex yet. It’s just stable enough that I’ve decided to create a new thread in the forum

1 Like
4

Documentation is a bit better. Most mutators are still undocumented, but that’s something I’ll solve in time. I’m also writing a series of articles about Darwin’s internals for future contributors (and because Darwin’s ideas can be applied to other kinds of projects that manipulate Erlang’s source, or the source of any functional language, for that matter)

Darwin can now do the following:

  • Mutate arithmetic and comparison operators
  • Mutate some literals (not all literals yet)
  • Traverse Erlang AST and apply mutations recursively
  • Mutate code in guard by recompiling function clauses into a linear sequence of matches that doesn’t use any guards; it effectively moves guards into “normal code”, where you don’t have any restrictions on what kinds of expressions are allowed. It’s interesting because guards and pattern matches are actually not very composable… You have to turn them into (very inefficient) normal code in order to make them easier to work with. The inefficiency is probably not a dealbreaker because Darwin is only used as a test tool and not in production.

There are still some useful mutators missing. However, if you think you’re missing any mutator, you can just implement your own (mutators are behaviours, and you can add your own modules that follow the behaviour)

1 Like
5

Cool, I’ve been following along with the development and mutation testing seems like a neat idea. So is Darwin ready to try out on actual projects now?

6

Yes, you can try it right now, but the test reporters are still very primitive. They merely list the mutations and whether they have survived or not, and don’t gather any statistics yet

2 Likes
7

Test reporting is now better: it produces a pretty HTML file with mutation listed by line.

It can now perform more mutations too. It can now perform mutations inside pattern matched, which it couldn’t previously. It does so by rewriting pattern matching expressions as a chain of function calls, which is less efficient but I don’t think it matters much when it’s only used for testing.

3 Likes