Data persisting between tests

I’m using Mongo (no Ecto), and I’ve got tests in multiple modules that rely on fixtures: e.g. I insert a single record then assert a lookup, or I insert multiple records and then assert query results with filters applied etc. It’s the usual song and dance, pretty similar to the Ecto style tests. The Mongo repo is listed in the app’s application.ex as a child process. (not sure if that matters).

However, I’m having a crazy time with data persisting between tests. I wrote up a helper function that drops the mongo database and re-indexes it so I could (I thought) reset the database between tests.

Things I’ve tried:

  • resetting the database in the setup_all function
  • resetting the database one of the functions listed in the setup macro
  • resetting the database inline in the test
  • turning async off

But tests keep failing. If I isolate the test run to only run 1 module at a time, everything passes.

Can anyone shed light on this? Does anyone have troubleshooting tactics for this?

You need to turn async off and reset the database on every setup for all tests.

1 Like

Yeah, I’ve tried that – I’m still getting failures. :thinking:

Async tests in Elixir is a little annoying. I am having the same problem with Sandbox…I think there’s something wrong on it. I’ll take a look.

I’m wondering if I’m running into a problem in the used test case and the callback functions I have defined in a setup chain… e.g.

  setup [:checkout_database, :append_mutation, ... etc...]

  defp checkout_database(context) do

All the problems started when we added one more test module. ITS tests pass. Which made me dig into this and see the weird behavior.

This is entirely different. Async testing with Ecto and Postgres works by matching transactions to processes and rolling them back after each test case. Mongo doesn’t have transactions that work this way, so it isn’t possible to use with Mongo. This is entirely different from the sandbox allowances issue you have in the other thread.

Both modules are async false, and both modules have a setup that resets the database? Have you added some sanity checks after TestUtils.reset_database() to ensure that it actually clears everything out?

1 Like

Yes. I know. I meant both are boring to handle. Despite the fact they are different.

I am not facing the same problem. I have another one completely different, but also related to tests. Is it better now? Improving my English :slight_smile:

Oh my, yes. And things continue to get strange. Let me explain the setup here just in case any of these details are important.

My test_helper.exs contains only this:

ExUnit.start(exclude: [:skip], async: false)

(that should force all tests to be run synchronously, right?)

I’ve got some tests that test GraphQL endpoints, so they

use CarlQlWeb.ConnCase

(and not ExUnit.Case directly).

I’ve been tearing things out and trying variations just to try to zero in on the problem. Currently my ConnCase includes the following setup:

setup [:checkout_database, :default_conn, :custom_req_headers, :append_mutation]

  defp checkout_database(_context) do
    TestUtils.reset_database() # <-- drop the mongo database and re-index collections
  # other setup functions follow

Then, back in the test module (the one with failing tests), I am testing various GraphQL queries, so I seed some data so the queries have something to work with. I have a block like this:

setup do

defp seed_content do
   # Upsert a handful of records

The module runs fine in isolation: all tests pass:

mix test test/my_app/api/my_resource_test.exs # <-- success! All pass!

But, when I run all the tests, things in this one resource test module fail:

mix test # <-- boo, a bunch of tests fail

So poking at the failures, I could see that data was bleeding over between tests. Because my setup helper seeds exactly 5 records into this collection, I put a line like this at the top of each test, just to make 100% sure that the collection didn’t get any other records added to it:

assert {:ok, [_,_,_,_,_]} = MyResource.get_many(%{})

And when I put those assertions at the beginning of each test in that module… mix test works: all tests pass. (And I’ve gotten in the habit of running mix test TWICE in a row, just to make sure that things are running consistently.)

If I comment out some of the assertions, test still pass, but I comment out others, and then tests fail again. And to make things really crazy, it’s not even consistent as to which test has or doesn’t have that assertion.

describe "some_query" do
    test "after filter returns no results if you don't look far enough back", %{conn: conn} do
      # All tests pass when this assertion is present.  About a half dozen tests in this module FAIL
      # when I comment out this assertion.
      assert {:ok, [_,_,_,_,_]} = MyResource.get_many(%{}). 
      query = %{
        query: "query { getMyResources(filter: { updatedAfter:{hours: 1}}) { someId } }"

      conn = post(conn, "/api", query)
      %{"data" => %{"getMyResources" => results}} = Jason.decode!(conn.resp_body)
      assert results == []

I’m truly baffled. I think tests must be executing asynchronously still – that would at least explain why sometimes these work and sometimes they don’t. The commenting thing really is odd, however.

Did you take at look at what CarlQlWeb.ConnCase is doing? Maybe it’s forcing async in some way.

Also try to remove async: false from the ExUnit start function and put CarlQlWeb.ConnCase, async: false for all modules, even modules not related to mongo.

Yeah, I looked through that… it’s mostly the ConnCase stuff that Phoenix gives you, but I added a few fixture functions for fetching data records.

I added async: false to all modules… but the tests still fail intermittently. It doesn’t seem to matter if I use

ExUnit.start(exclude: [:skip], async: false)


ExUnit.start(exclude: [:skip])

Figuring this out has become an obsession at this point…

The only thing I can think of at this point is that this could have something to do with the fact that mongo is being dropped and indexed via mix tasks, e.g."mongo.drop")"mongo.index")

Do those happen in their own threads? And thus… they may complete at different times?

The saga continues… my mongo mix tasks had these in there:

{:ok, _} = Application.ensure_all_started(:my_app)
Mongo.start_link(Application.get_env(:mongo_wrapper, :mongo_db))

But commenting those lines out still doesn’t guarantee that the tests pass. I can get 3 or 4 test runs to pass, but then they fail intermittently still.

I’m beginning to suspect that this might have to do with how the task(s) is (are?) started… I wrote a module that sort of looks like Ecto’s Repo module. It’s got a start_link function:

def start_link(_opts \\ []) do
    Mongo.start_link(Application.get_env(:mongo_wrapper, :mongo_db))

And then that’s ref’d in our application’s supervisor (again, emulating an Ecto Repo):

  def start(_type, _args) do
    children = [
      # ... 
      {MongoWrapper.Repo, []},    
     # ...
    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)

I still feel like somehow, the tasks are happening asynchronously…

This is still going on. I tried sleeping the process after every dropping of the database, but that didn’t help.

Here’s something from the Mongo Docs that might be relevant:


Starting in MongoDB 4.2:

If you drop a database and create a new database with the same name, you must either:

  • Restart all mongos instances and all mongod shard members (including the secondary members);
  • Use the flushRouterConfig command on all mongos instances and all mongod shard members (including the secondary members) before reading or writing to that database.

FYI: flushRouterConfig has to be run against the admin database, so it’s not possible to leverage the same connection as the one that connects to a test database.

This raises an interesting point. Can you reproduce this kind of behaviour by running whatever the Mongo equivalent is of psql? You could have a script that writes a bunch of data, clears things out however you are in ecto, and then tries to read the data again.

Yeah, I can log into the mongo CLI and see that the test database has data in it after the test – that’s not unexpected, however: the test run does add data in. One workaround is to inject temporary collection names during collections so that each test runs in an isolated collection. I’m also looking into transactions, but this isn’t Mongo’s strong suit…

Sorry to be clear, what I’m trying to figure out is if your Elixir code is missing something important, or if the MongoDB is simply unable to do this properly. To determine this, I’m suggesting that you recreate some important subset of the data create / delete / create flow without involving Elixir at all. If the problem goes away, then we need to figure out what the Elixir code is doing differently. If the problem persists, then something is fundamentally flaws with Mongo, or at least this installation of it.

It’s definitely in Elixir somehow. In the mongo CLI, if I insert data and then drop the database, searching for that data turns up no results.

OK cool, so can you show the code for TestUtils.reset_database() ? Can you run the test suite with :debug level logging so that you can show us exactly what gets run at the mongo level?

I think I figured out the culprit – here’s the code from TestUtils.reset_database():

def reset_database do"mongo.drop")"mongo.index")

As soon as I copied the code out of those mix tasks and put them in my test module case, things worked (and it took a lot longer).

So this may be a stupid question, but when you call , does the execution of that code happen asynchronously in a separate process? (It would appear that the answer to that is YES). The docs for the Mix submodules are hard to navigate because it’s not clear how something like relates to the command-line counterpart of mix run, but maybe there’s a note about this behavior somewhere?

No, BUT you aren’t making any asserts that those tasks run successfully. By default tasks can only be one once and then after that they error unless you specifically re-enable them. Thus, the tasks ran the first test case, but after that just returned errors and wouldn’t run.

1 Like