Muex - the library for mutation testing

Thanks for open sourcing this, I’m playing with it now. It great to have this in Elixir.

v0.6.0 is out, all kudos go to @bglusman

Fixes:

• classify sandbox compilation errors as :invalid, not :killed
• auto-discover mutators + StatementDeletion & ReturnValue mutators

Features:

• parallel cross-file mutation testing with sandbox isolation

Very interesting project! Thank you. I left an issue on Github because it seems to have problem compiling my project that uses macros.

Yeah, thanks! I am on it already.

I had Claude Code left you a comment too, which may or may not be useful

Nah, the thing is I overestimated my ability to deal with a dozen of projects faster than my AI assistant (I am kinda competing with her on that matter )

Yesterday I messed up merging new features and in a rush I did not validate it 102%. Stay tuned, the working version is to be there soon.

v0.6.1 is out, thanks @lud for testing and reporting the bug.

Fixes:

• Less aggressive optimization without premature Code.compile_string/2

I’ll try as soon as possible

Hello,

So I let it run with mix muex --optimize --optimize-level aggressive --verbose and it still takes a lot of time. I believe JSV was a very bad match because there are 7000 tests. I’ll try to move the tests so I can target only the right subset (for now I have a generated/ tests directory inside test/jsv/ so I cannot target "all files except those in generated. But it could be nice to be able to pass exclusion flags to mix testto exclude a@moduletag`.

Then about the results I’m not so sure how to interpret them.

Survived Mutations:
--------------------------------------------------
lib/jsv/vocabulary/v7/applicator.ex:86
  ReturnValue: replace return value of validate_keyword with nil

If I return nil from the function defined I have 52 test failures when running manually.

lib/jsv.ex:374
  StatementDeletion: delete statement 63 of 138

This is a @doc group @doc_group. I also have results about removing a @doc false statement. Those should be ignored.

lib/jsv/resolver.ex:276
  StatementDeletion: delete statement 36 of 78

I’m not sure what is statement 36:

   283	  defp scan_subschema(list, parent_id, nss, meta, path, acc) when is_list(list) do
   284	    list
   285	    |> Enum.with_index()
   286	    |> EnumExt.reduce_ok(acc, fn {item, index}, acc ->
   287	      scan_subschema(item, parent_id, nss, meta, [index | path], acc)
   288	    end)
   289	  end

Some results file:line point to a defstruct statement. The app does not compile if I remove them.

So in the end it’s not really helpful for now with all those delete statement X of Y because I don’t know what that is. It could be really helpful to have a report file with patches so we can see what was the mutation, reproduce it, write a test to invalidate.

I’m trying to see if the html report has this, but I can’t find a file to generate actual mutations. I set a “limit 30” but it only generates “invalid mutations”. I’ll try to re-run the full output (still with aggressive though) later.

But it was already helpful in some way, so far I know that I have to test a @derive {Inspect, ... statement that is not tested at all.

So I believe this is promising but you need a way to make prioritize the mutations in some way.

Yes. And this is unavoidable. We (@bglusman and myself) put a lot of effort to the sophisticated optimization, but Elixir is still a compiled language.

That’s exactly what’s planned next.

Absolutely! Thanks for helping to test it!