Oh my, yes. And things continue to get strange. Let me explain the setup here just in case any of these details are important.
My test_helper.exs contains only this:
ExUnit.start(exclude: [:skip], async: false)
(that should force all tests to be run synchronously, right?)
I’ve got some tests that test GraphQL endpoints, so they
use CarlQlWeb.ConnCase
(and not ExUnit.Case directly).
I’ve been tearing things out and trying variations just to try to zero in on the problem. Currently my ConnCase includes the following setup:
setup [:checkout_database, :default_conn, :custom_req_headers, :append_mutation]
defp checkout_database(_context) do
TestUtils.reset_database() # <-- drop the mongo database and re-index collections
:ok
end
# other setup functions follow
Then, back in the test module (the one with failing tests), I am testing various GraphQL queries, so I seed some data so the queries have something to work with. I have a block like this:
setup do
seed_content()
:ok
end
defp seed_content do
# Upsert a handful of records
end
The module runs fine in isolation: all tests pass:
mix test test/my_app/api/my_resource_test.exs # <-- success! All pass!
But, when I run all the tests, things in this one resource test module fail:
mix test # <-- boo, a bunch of tests fail
So poking at the failures, I could see that data was bleeding over between tests. Because my setup helper seeds exactly 5 records into this collection, I put a line like this at the top of each test, just to make 100% sure that the collection didn’t get any other records added to it:
assert {:ok, [_,_,_,_,_]} = MyResource.get_many(%{})
And when I put those assertions at the beginning of each test in that module… mix test works: all tests pass. (And I’ve gotten in the habit of running mix test TWICE in a row, just to make sure that things are running consistently.)
If I comment out some of the assertions, test still pass, but I comment out others, and then tests fail again. And to make things really crazy, it’s not even consistent as to which test has or doesn’t have that assertion.
describe "some_query" do
test "after filter returns no results if you don't look far enough back", %{conn: conn} do
# All tests pass when this assertion is present. About a half dozen tests in this module FAIL
# when I comment out this assertion.
assert {:ok, [_,_,_,_,_]} = MyResource.get_many(%{}).
query = %{
query: "query { getMyResources(filter: { updatedAfter:{hours: 1}}) { someId } }"
}
conn = post(conn, "/api", query)
%{"data" => %{"getMyResources" => results}} = Jason.decode!(conn.resp_body)
assert results == []
end
end
I’m truly baffled. I think tests must be executing asynchronously still – that would at least explain why sometimes these work and sometimes they don’t. The commenting thing really is odd, however.
