Exavier - Mutation Testing library for Elixir

Hi @dnlserrano, sorry for not responding in a reasonable time.
But then I have to create, for each new test module, a new line in this configuration file? Is there a way for exavier to detect automatically these modules? (New idea for my future MR maybe? hehe)

Thanks for all the support.

Hey Luiz,

test_files_to_modules should have test file paths as keys and the actual module that test file is testing as value (i.e., not the test file module as you did). An example is in the exavier repo’s self-mutation testing as example here:

  test_files_to_modules: %{
    "test/exavier/mutators/aor1_test.exs" => Exavier.Mutators.AOR1,
    ...

Hey guys,

I’m just having another issue with the library. Can you give me a hand? For 99% of my test modules, I receive this message:

10:38:11.461 [error] Could not find module  defined in option :test_files_to_modules for test/gateway_web/controllers/participant_controller_test.exs.
10:38:11.461 [error] GenServer Exavier.Server terminating
** (MatchError) no match of right hand side value: :ok
    (exavier 0.3.0) lib/exavier/cover.ex:8: Exavier.Cover.lines_to_mutate/2
    (exavier 0.3.0) lib/exavier/server.ex:20: anonymous fn/2 in Exavier.Server.handle_call/3
    (elixir 1.11.2) lib/enum.ex:2181: Enum."-reduce/3-lists^foldl/2-0-"/3
    (exavier 0.3.0) lib/exavier/server.ex:18: Exavier.Server.handle_call/3
    (stdlib 3.14) gen_server.erl:715: :gen_server.try_handle_call/4
    (stdlib 3.14) gen_server.erl:744: :gen_server.handle_msg/6
    (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.94.0>): :xmen1

I have already tried reference them in the .exavier file like this:

%{
 test_files_to_modules: %{
   "test/address_key_worker_test.exs" => Gateway.AddressKeyWorkerTest,

But it doesnt work. It does not recognize my test modules and result in a similar error (and even if it worked, there would be a large work to put all test modules in the file and to educate the team to do so).
Do you know what can be done?

Thanks in forward.

Thanks a lot. I’m still new to elixir and was not able to find this debug option.
And certainly am going to try that MR and/or collaborate with anything I can.

Hey Luiz, thanks for your interest in exavier. For now I think you should be able to overcome that by setting EXAVIER_DEBUG=1.

MRs welcome to allow setting custom timeouts for each particular time-bound work.

Hi,

First, thanks for creating this library, I’m a personal fan of mutation tests and got really glad by finding exavier.

I’m actually having an issue running the mutation tests with the default timeout for mutate_module. Here is the stacktrace:

13:46:08.242 [error] GenServer Exavier.Server terminating
** (stop) exited in: Task.Supervised.stream(5000)
    ** (EXIT) time out
    (elixir 1.11.2) lib/task/supervised.ex:304: Task.Supervised.stream_reduce/7
    (elixir 1.11.2) lib/enum.ex:3461: Enum.reverse/1
    (elixir 1.11.2) lib/enum.ex:3054: Enum.to_list/1
    (exavier 0.3.0) lib/exavier/server.ex:59: Exavier.Server.handle_call/3
    (stdlib 3.13.2) gen_server.erl:706: :gen_server.try_handle_call/4
    (stdlib 3.13.2) gen_server.erl:735: :gen_server.handle_msg/6
    (stdlib 3.13.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.94.0>): :xmen

Is there a way to configure a longer timeout?

Thanks a lot in advance

I’ll look into it. IMO, if we are recompiling the same module, we shouldn’t consume new memory from the literal allocator. And I did try to clean up the code as much as I could. I’m not as motivated to get at the bottom of this, though, because my approach no longer depends on recompiling modules…

Lol, that’s awesome. ^.^

I wonder what is leaking, if you could reduce it to a simple test-case (recompilation loop?) and figure out if Elixir or OTP issue then a bug report can be submitted? I know they both store information out of band, wonder if it is ever cleaned up…

I’ve just found out that recompiling modules lots of times (~ hundreds of times), even if the modules don’t change consumes memory from the BEAM’s literal allocator. This memory doesn’t seem to be garbage collected and it will fill up and cause errors. If you decide to generate mutants one at a time and each mutant requires a module recompilation, you might hit these errors…

I’m just warning you because I found out about this problem the hard way (i.e., when I tried to recompile my test suite about 300 times for 300 mutations in an elixir module). Fortunately, the way I’m generating mutants doesn’t require recompiling the code for each mutant, so I could work around it.

Hey @tmbb. No, I haven’t tried “one mutation at a time instead of all at once”. I’m now focusing on trying to have test coverage by individual test instead of by each whole module. This would allow me to evaluate mutation coverage in a more fine-grained way and have more realistic coverage (not as low, in practice), for each of the modifications I make.

Currently, I run an (all at one) mutation on a module, run the whole test module on in and check failures. If some test passes, then I flag it as a survived mutant (bad). But sometimes, I may be changing only function X and not function Y, but testing functions X and Y as part of the same test module.

If I can have finer-grained test coverage, I could understand only function Y is covered by test Y, and only try and modify that on the mutation run. Same goes for X. Better, more accurate coverage would come out of this. But it’s not trivial to do without being a bit hacky. I’ll try and get there. I’ll keep on posting any relevant developments here. Right now I’m not even pushing changes to my remote, since it’s all very experimental at this stage.

Stay tuned.