Is there a way to generate an ignore file for all failing tests?

With dialyzer, you can create an ignore file so that once generated, you only get notified of new type errors. Is there an equivalent for mix unit tests? The context is we are taking over a codebase where the original author has since left, and it has ~150 failing tests out of 400. The system works but likely the tests are out of date or written in such a way that they are flaky.

1 Like

You can potentially use a tag and exclude them that way. See Introduction to Testing — Phoenix v1.7.12

1 Like

In all earnestness, I would figure out which the flakey ones are and skip them for now then delete everything else. If the system works then they are useless tests atm. You could just save the descriptions somewhere for when you have time to come back and re-implement them, if what they say is even true anymore.

1 Like

You could probably hack something based on GitHub - balance-platform/ex_unit_summary: "Extension" for ExUnit to generate a list to then then tag/ignore

2 Likes

After a full run of mix test you can invoke mix test --failed in combination with ex_unit_summary as @mayel pointed out and you can copy-paste the entire output and only leave the filename:line pairs with sed or awk. Should be a 15 minute project.

1 Like

Not sure I follow. Do I still need to manually tag each test that is failing?

Oops, yes, I didn’t specify. I meant that collecting the output of mix test --failed with a custom test result formatter can then be used to do a xargs + sed scripts that inserts f.ex. @tag :broken above each of the filename:line pairs and then configure ExUnit to exclude the :broken tag when running tests.

But in seriousness, this is way too brittle a solution and is error-prone. I know 150 tests are a lot but I’d just get some cake and coffee and manually tag each broken test manually over the course of 1-2h.

1 Like