Hi there, I am trying to come up with a good organization of tests for a complex codebase, and I could use some help. I have two goals:
- Have a clear separation between lightweight tests (which do not require starting the application) and heavyweight tests (which do).
- Let mix start the application only when necessary (e.g. if I run
mix test apps/my_app/lightweight_tests, I want to avoid starting the application).
I tried the following:
- Introduce two root test directories by configuring
test_paths: ["lightweight_tests", "heavyweight_tests"]
- Introduce an alias in
mix test is automatically translated to
mix test --no-start
heavyweight_tests/test_helper.exs to start the application using
With all this in place, I can run
mix test apps/my_app/lightweight_tests/foo_test.exs, but unfortunately the application is being loaded anyway. It looks like the
heavyweight_tests/test_helper.exs is being executed, even though no tests from that directory are being run.
Based on this, I have two questions:
- Is there any additional step I can take to only execute the relevant
test_helper.exs script for the tests that I am running?
- Is my approach sound, or would you recommend doing things differently?
I’d suggest using test tags + some logic in
test_helpers.exs to figure out if you need to start the application or not. Then you can alias including/excluding certain tests as mix tasks or manually do that on the
mix test task. What you’re doing is not really creating two testsuits and I’m not even sure exunit supports that in this manner.
This is exactly what I do.
I’ve successfully done this with my tagging machinery. ExUnit doesn’t support it natively; but it’s pretty easy to implement once your aliases are in place. An example from
I have my
test directory break out my different “suites” into subfolders:
Each test suite has a base module with a
__using__ macro that wraps
use ExUnit.Case with the correct tags for that suite via ExUnit’s
@moduletag, so each test in a suite can simply do something like:
defmodule Matcha.UnitTest do
# describe do; test do;
Note each suite test file must use the ExUnit naming conventions, ex. test/unit/matcha_test.exs and a similarly named test/benchmark/matcha_test.exs; however the test modules need unique suffixes, ex Matcha.UnitTest and Matcha.BenchmarkTest, instead of the more traditional
MatchaTest, as they would otherwise conflict when running all suites.
My test helper, by default, excludes all suites (plus, any test tagged with
suites = [:benchmark, :doctest, :unit, :usage]
# The test suite to run is re-includeded via CLI flag
ExUnit.start(exclude: [:skip | suites])
This complete exclusion of all tests gets overriden at the CLI level.
In my mix.exs I declare some useful aliases to wrap those CLI flags:
The default set of suites to run (benchmarks are excluded by default), and a “run all defaults” alias:
Aliases for each individual suite (ie:
In CI, I can invoke different suites/combination of suites via those aliases. For example:
The only real downside here is that as aliased,
mix test excludes everything by default. This works for
Matcha’s case since it’s a library and its
DEVELOPMENT.md directs you to use
mix test.suite instead when developing locally.
In an application context, you’d probably skip the
mix test.suites alias and just provide the default suite flags in a
mix test alias, so the standard developer workflow Just Works .
My private personal applications use that approach; though thier “heavyweight tests” that hit a test database live in a
test/integration suite; and the CI runs those in a dedicated step alongside the unit test suite, attaching a postgres db and running migrations. This lets me just run lightweight stuff locally and quickly in precommit hooks; and heavy stuff only in CI off of my development machine.
So you’d substitute
test/benchmarks in the examples above with integration stuff instead. The integration suite is the only one that does ecto sandbox setup in the
App.IntegrationTest.__using__ macro, instead of globally in the test helper.
To achieve your second goal, since
mix test starts your app by default, I think you’d need to add the
--no-start flag to your lightweight test aliases; then manually add your
Application.ensure_all_started(:your_otp_app_name) in your heavyweight test module macros only? In practice I don’t worry about this because my app startup time is pretty negligable and those lightweight tests execute so fast in parallel; though since they are so fast the app startup does dominate the time the suite takes to run, and is an interesting optimization.
It’s kind of a lot of boilerplate to set up; but once in place works really well and transparently to segregate large and complicated battalions of tests into dedicated suites where you can customize what runs when, where appropriate.
I’m not sure I’ve done a great job of breaking this all down, let me know if I can clarify! I do hope this helps a little.
If you have separate
ExUnit.CaseTemplates you can use their
using/2 defmacro to set the
@moduletag. No need to wrap what is already essentially
using/2 can even receive the options passed to
use CaseTemplate, …: ExUnit.CaseTemplate — ExUnit v1.14.2
That’s super slick, I did not know about
ExUnit.CaseTemplates! Maybe I should refactor my custom module approach, it’d simplify things a hair.
Wow! Thanks for answering in so much detail. I’ll definitely play with these ideas and see how far I can get