Hey, so I’ve looked at protox a bit, and I’ve found a few things:
- I was able to reproduce some of those compilation errors, and they should have been caught and counted as a successful run (since the mutation was caught), so I’ll push up a fix for that in 1.0.2 in a little bit - thanks for the report there!
- I wasn’t able to reproduce this locally when I cloned down
protox to debug this. Is it possible you weren’t in the root directory when running that so the path was incorrect? This reminds me - I should probably check that the file exists when folks use --only so they can get some better feedback, so this is still super helpful!
- It looks like there’s a process that can be killed by some mutations, and when that process gets killed the tests don’t run. Here’s the mutation that caused the process to be killed:
%{
line: 26,
mutation: ["def", " ", "dvysgsomhx", "(", "", ":string", "", ")", ",", " ",
"do:", "", " ", "\"", "", "\""],
original: ["def", " ", "default", "(", "", ":string", "", ")", ",", " ",
"do:", "", " ", "\"", "", "\""],
path: "lib/protox/default.ex"
}
nonode@nohost: "Mutating file"
warning: clauses with the same name and arity (number of arguments) should be grouped together, "def default/1" was previously defined (nofile:13)
nofile:27
nonode@nohost: "Mutating completed"
nonode@nohost: "No compile dependencies to recompile"
nonode@nohost: "Tests starting"
Excluding tags: [conformance: true, properties: true]
== Compilation error in file test/protox_test.exs ==
** (FunctionClauseError) no function clause matching in Protox.Default.default/1
The following arguments were given to Protox.Default.default/1:
# 1
:string
(protox 1.2.2) nofile:13: Protox.Default.default/1
(protox 1.2.2) lib/protox/parse.ex:295: Protox.Parse.get_kind/3
(protox 1.2.2) lib/protox/parse.ex:212: Protox.Parse.add_field/5
(protox 1.2.2) lib/protox/parse.ex:203: Protox.Parse.add_fields/4
(protox 1.2.2) lib/protox/parse.ex:179: Protox.Parse.make_message/4
(protox 1.2.2) lib/protox/parse.ex:164: Protox.Parse.make_messages/4
(protox 1.2.2) lib/protox/parse.ex:128: Protox.Parse.parse_file/2
(protox 1.2.2) lib/protox/parse.ex:108: Protox.Parse.parse_files/2
(protox 1.2.2) lib/protox/parse.ex:18: Protox.Parse.parse/2
(protox 1.2.2) expanding macro: Protox.__using__/1
test/protox_test.exs:43: ProtoxTest (module)
(elixir 1.11.0) expanding macro: Kernel.use/2
test/protox_test.exs:43: ProtoxTest (module)
(elixir 1.11.0) lib/kernel/parallel_compiler.ex:416: Kernel.ParallelCompiler.require_file/2
(elixir 1.11.0) lib/kernel/parallel_compiler.ex:316: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7
nonode@nohost: "Tests finished"
08:12:28.398 [error] GenServer #PID<0.834.0> terminating
** (stop) killed
Last message: {:EXIT, #PID<0.833.0>, :killed}
State: %DynamicSupervisor{args: {:ok, %{extra_arguments: [], intensity: 3, max_children: :infinity, period: 5, strategy: :one_for_one}}, children: %{#PID<0.836.0> => {{GenServer, :start_link, :undefined}, :temporary, 5000, :worker, [GenServer]}, #PID<0.837.0> => {{GenServer, :start_link, :undefined}, :temporary, 5000, :worker, [GenServer]}}, extra_arguments: [], max_children: :infinity, max_restarts: 3, max_seconds: 5, mod: Supervisor.Default, name: {#PID<0.834.0>, Supervisor.Default}, restarts: [], strategy: :one_for_one}
I think this might actually be a really interesting piece of feedback that is being surfaced by muzak - that there’s a way to crash the application that puts it into a potentially unusable state. I’ll have to have a think about how best to capture this feedback and present it to the user so they can use it to improve their application. My first thought is to maybe allow the user to set a timeout for each mutation that, if passed, would mean something like “something has gone very wrong with your application so we’re going to print out a bunch of information about what happened while we were running mutation tests so you can figure out what happened to your application.”
How that output would look, however, is a tricky question that I’ll need some time to consider.
.
. Furthermore, when I apply one of the mutation manually (the first one in the extract above), my tests fail as expected. So, why did muzak says this mutation survived? What am I missing?
