Can `test_helper` in different `test_paths` run in sequence instead of in parallel?

Questions / Help
, ,
1

If you have multiple test_paths defined, is it possible to run them in sequence instead of in parallel?

Is it possible to setup entire test suits that should run in sequence?

Some background information:

I have two kind of tests.

  • Unit tests that tests a single unit in isolation using Mox to mock the modules dependency, these make up ~98% of the tests. All of these resides in the test directory.
  • Integration tests / End to end test. These test requires a environment where the rest of the system is up and running and is used to verify that the other team haven’t broken anything, these are mostly here for assurance. :slight_smile:
    These resides in the test_integration directory and are tagged with @tag :integration.

Right now we are unit testing the api endpoints by calling the router and letting it route a Plug.Test.conn request to the endpoint, due to this we can’t pass arguments directly to the endpoint plug. Instead we pass the mocked dependencies to the endpoint by changing the Application env and the endpoint plug fetches them with a Application.get_env call in a helper plug that adds a dependency struct to the connection.

The test_helper in the test directory creates Mox objects and changes the application env to use them by doing Application.put_env(:api, Twitter, TwitterMox) but all tests in the test_integration directory should use the “real” modules.

I solved this right now by having two test commands, one which excludes integration tests and one that only runs integration tests and having the test_helper use the code below but that means elixir overwrites the coverage files. If I could run the test_helpers in sequence the the helper would just need to change the application env to match the current testsuite.

# The `test/test_helper.exs` file.
alias Extra.Api.{Foo, Bar}
alias Extra.Task

require Logger

mock = fn (source) ->
  target = :"#{source}Mox"
  Mox.defmock(target, for: source)
  Application.put_env(:api, source, target)
#  Logger.info(fn -> "Added mox mock for #{source}."end)
end

case ExUnit.configuration |> Keyword.get(:exclude) |> Enum.member?(:integration) do
  true ->
    mock.(Foo)
    mock.(Bar)
    mock.(Task)

    ExUnit.start(
      capture_log: true,
    )

  false ->
    IO.puts "Skipping unit test since integration tests will be running."
end
2

See what we do in Ecto. We have different environments for the unit tests and for integration tests. This way they run in isolation.

It is worth also saying that the Mox API was designed to make it very easy to delegate to the actual service. If you use master, which has a stub! function, you can write:

YourMock
|> stub!(:foo, &ActualImpl.foo/2)
|> stub!(:bar, &ActualImpl.bar/3)

And that should make it work in your integration tests too.

2 Likes
3

Thanks, I did not realise that I could just change the test_paths configuration depending on env variables. Is obvious now that I see it. :slight_smile:

I haven’t tried (away from computer) but I don’t think the Mox example will work.

The Mox stub will only be reachable from the current process context, right? I if my ActualImpl uses a GenServer and then uses Application.get_env to get the default implementation (depending on configuration) it will receive a unconfigured Mox module. The root problem is that the integration tests will pass through multiple GenServer:s and Tasks since it is a test of the complete code path, cutting through the layers, from a Plug router, to the DB and back. And yes, I know these kind of test often gets a bit brittle, the unittests will be the test that actually discover bugs 9 times out of 10.

Of course, I could solve this by disabling async in the testcase and configure the env with the ActualImpl but that would slow down the tests. I could also solve it by always passing in the dependencies directly but that would not work in the Plug router testcase. On the other hand, that might be a indication that I should not test the api endpoint plug this way…

I think I will use the Ecto example and change where the coverage file gets placed. I won’t be able to run both test suits at one time but it is good enough.

4

We are adding shared tests (like Ecto’s sandbox) to the next version of Mox as well.

2 Likes
5

Awesome. I will spam mix hex.outdated --all until I see a Mox update. :thumbsup: