Organizing tests in a complex codebase

Questions / Help
1

Hi there, I am trying to come up with a good organization of tests for a complex codebase, and I could use some help. I have two goals:

  1. Have a clear separation between lightweight tests (which do not require starting the application) and heavyweight tests (which do).
  2. Let mix start the application only when necessary (e.g. if I run mix test apps/my_app/lightweight_tests, I want to avoid starting the application).

I tried the following:

  1. Introduce two root test directories by configuring mix.exs with test_paths: ["lightweight_tests", "heavyweight_tests"]
  2. Introduce an alias in mix.exs so mix test is automatically translated to mix test --no-start
  3. Configure heavyweight_tests/test_helper.exs to start the application using Application.ensure_all_started(:my_app)

With all this in place, I can run mix test apps/my_app/lightweight_tests/foo_test.exs, but unfortunately the application is being loaded anyway. It looks like the heavyweight_tests/test_helper.exs is being executed, even though no tests from that directory are being run.

Based on this, I have two questions:

  1. Is there any additional step I can take to only execute the relevant test_helper.exs script for the tests that I am running?
  2. Is my approach sound, or would you recommend doing things differently?
2

I’d suggest using test tags + some logic in test_helpers.exs to figure out if you need to start the application or not. Then you can alias including/excluding certain tests as mix tasks or manually do that on the mix test task. What you’re doing is not really creating two testsuits and I’m not even sure exunit supports that in this manner.

1 Like
3

This is exactly what I do.

I’ve successfully done this with my tagging machinery. ExUnit doesn’t support it natively; but it’s pretty easy to implement once your aliases are in place. An example from Matcha’s codebase:

Implementation

  1. I have my test directory break out my different “suites” into subfolders:

    test
├── benchmark
├── doctest
├── integration
├── unit
└── test_helper.exs

  2. Each test suite has a base module with a __using__ macro that wraps use ExUnit.Case with the correct tags for that suite via ExUnit’s @moduletag, so each test in a suite can simply do something like:

    defmodule Matcha.UnitTest do
  @moduledoc false

  use UnitTest
  # describe do; test do;
end

    Note each suite test file must use the ExUnit naming conventions, ex. test/unit/matcha_test.exs and a similarly named test/benchmark/matcha_test.exs; however the test modules need unique suffixes, ex Matcha.UnitTest and Matcha.BenchmarkTest, instead of the more traditional MatchaTest, as they would otherwise conflict when running all suites.

  3. My test helper, by default, excludes all suites (plus, any test tagged with @tag :skip):

    suites = [:benchmark, :doctest, :unit, :usage]
# The test suite to run is re-includeded via CLI flag
ExUnit.start(exclude: [:skip | suites])

    This complete exclusion of all tests gets overriden at the CLI level.

  4. In my mix.exs I declare some useful aliases to wrap those CLI flags:

    • The default set of suites to run (benchmarks are excluded by default), and a “run all defaults” alias: mix test.suites

    • Aliases for each individual suite (ie: mix test.benchmark; mix test.unit)

  5. In CI, I can invoke different suites/combination of suites via those aliases. For example:

Notes

The only real downside here is that as aliased, mix test excludes everything by default. This works for Matcha’s case since it’s a library and its DEVELOPMENT.md directs you to use mix test.suite instead when developing locally.

In an application context, you’d probably skip the mix test.suites alias and just provide the default suite flags in a mix test alias, so the standard developer workflow Just Works :tm:.

My private personal applications use that approach; though thier “heavyweight tests” that hit a test database live in a test/integration suite; and the CI runs those in a dedicated step alongside the unit test suite, attaching a postgres db and running migrations. This lets me just run lightweight stuff locally and quickly in precommit hooks; and heavy stuff only in CI off of my development machine.

So you’d substitute Matcha’s heavyweight test/benchmarks in the examples above with integration stuff instead. The integration suite is the only one that does ecto sandbox setup in the App.IntegrationTest.__using__ macro, instead of globally in the test helper.

To achieve your second goal, since mix test starts your app by default, I think you’d need to add the --no-start flag to your lightweight test aliases; then manually add your Application.ensure_all_started(:your_otp_app_name) in your heavyweight test module macros only? In practice I don’t worry about this because my app startup time is pretty negligable and those lightweight tests execute so fast in parallel; though since they are so fast the app startup does dominate the time the suite takes to run, and is an interesting optimization.

Conclusion

It’s kind of a lot of boilerplate to set up; but once in place works really well and transparently to segregate large and complicated battalions of tests into dedicated suites where you can customize what runs when, where appropriate.

I’m not sure I’ve done a great job of breaking this all down, let me know if I can clarify! I do hope this helps a little.

4 Likes
4

If you have separate ExUnit.CaseTemplates you can use their using/2 defmacro to set the @moduletag. No need to wrap what is already essentially __using__. using/2 can even receive the options passed to use CaseTemplate, …: ExUnit.CaseTemplate — ExUnit v1.16.0

3 Likes
5

That’s super slick, I did not know about ExUnit.CaseTemplates! Maybe I should refactor my custom module approach, it’d simplify things a hair.

6

Wow! Thanks for answering in so much detail. I’ll definitely play with these ideas and see how far I can get :slight_smile:

1 Like