A single test passes, but if I run the whole test suite, the same test fails

Questions / Help
, ,
1

Hello everyone, lately I have encountered this problem, I have seen it happen in several languages, any idea how to deal with it in elixir?

For example, when I run

mix test test/app_web/live/module/index_live_test.exs → pass

but if I do

mix test → only the above test fails

I want to clarify that in my case I use Task.async in that module and in the test I also use Task.async and send() to send those tasks.

P.S: Actually, this works but separating the problematic tests to another test module, I don’t know if this is considered bad practice.

P.S2: Sometimes it was solved with Process.sleep(300) or adding async: true to the ConnCase config.

Thank you very much in advance.

2

It’s impossible to know without seeing your code but this generally happens because one test is causing persisting side effects that another test is reading. In the past this was usually the database but Phoenix, and most other frameworks, rollback the db state on each test. Did you turn that off, per chance? Another possibility is that one test is writing a file that another is expecting not to be there. There are a lot of different possibilities but that’s the gist of it.

2 Likes
3

Assuming this is a learning project, are you able to share the full repro (maybe a repo on Github)?

4

Hi, unfortunately the project is private, and I can’t share it, so I wanted a more general overview of the problem.

5

Yes I would think they are side effects, because the error it generates can be described as: “it is not reached to send the Task.async therefore no such tasks exist and the test fails”, I have separated the tests that handle Task.async to another describe with a different setup and that works for now for me, thank you very much for the general context of the problem.

P.S: I have not moved any of the test configuration (rollback question).

6

General overview: your code has mysterious side-effects that make the tests flaky.

Potential steps to investigate:

  1. Isolate the tests into a reproducible and publishable piece of code.
  2. Hire a consultant, have them sign and NDA and debug it in your codebase.

I don’t think a productive discussion can be had with the details you’ve been able to provide so far.

2 Likes
7

Yes, sorry, ideally I would like to have a project with the reproducible bug, right now I solved it by using a different describe with a different setup for the problematic tests (in same file), in a future post I will be sure to follow the steps you recommend, thank you very much.

1 Like
8

Agree with @iarekk but my shot in the dark: I know you just said you are using Task.async and send but are you somehow sending to the same named process across tests? That’s all I got without more info.