I’m writing some tests and I want to build a “common language” for writing test tags. I’m using this post to document my findings as I learn about the built-in test tags, while also seeing what custom tags the Elixir community uses in their tests.
@tag :skip or @tag skip: "some reason to skip" - Tests that should be skipped (unless run with --only skip). Support for this tag is built-in to ExUnit.
Custom Tags
Here are some of the custom tags I use or have encountered:
@tag :slow - Tests that take a long time to run.
@tag :external - Tests that call an external service or API
@tag :mock - Indicates that a test uses a mock to pass. (This tag can be paired with another test that uses @tag :external so there is an option to run a “live” version of the mocked test.)
@tag :fixme - A temporary placeholder tag that I use when I’m working on one specific test and only want it to run.
What test tags do you commonly use? Post them here so we can all speak the same language!
Other @tag tips, tricks, and hacks are welcome here too.
Other than what’s written in docs I’d like to point to the trick that allows to “generate” very similar tests from a matrix…
setup %{role: role, status: status} do
%{user: insert_user(role, status)}
end
for role <- [:admin, :guest],
{status, color} <- [{:online, "green"}, {:offline, "gray"}, {:away, "yello"}] do
@tag role: role, status: status, color: color
test "#{role} renders #{color} badge when #{status}", tags do
%{role: role, status: status, color: color} = tags
# render and assert on role, status, color
end
end
Though, arguably these tests are now become a bit less clear… as instead of
One that I don’t have yet but wanna built, benchee needs to retry things in some circumstances due to its nature. I’d like to move that from a function call to a tag.
Very cool. I started this thread just trying to get a “shared vocabulary”, but as I was writing it, I started discovering the depth of the @tag concept, how you could pass parameters to it, etc.
From what I can tell, there is definitely a lot of room for useful techniques to fully exploit this feature.
I’m using (mostly) @moduletag rather than individual @tag attributes; the tests are organized to facilitate that sort of general handling.
My use is pretty simplistic/naïve. Most of the Elixir testing I have works with Elixir applications which can be thought of a libraries and the testing reflect this assumption. I have three values into which my tests are categorized:
:unit
Unit tests. These tests are exercising implementations as expressed by module public functions; while these functions are public in the sense that they use def in their definitions, they are in practice considered internal or private API. These test run asynchronously and if they involve persistent data (i.e. a database) then they have test data either seeded or generated specifically so that they don’t depend on/interfere with other tests that might be running.
:integration
Integration tests. This testing is focused on two goals: testing the Public API of each Elixir project and testing end-to-end, business process/data flows. These tests are synchronous and are run with a set random seed (of 0) so that ordering is deterministic. It is expected that data created or manipulated in earlier tests are to be used in later tests; any test related data seeding or pre-test data generation either is non-existent or there to populate data for dependencies of the project being tested. We really want to see that data being created in one process can be used in other downstream processes.
:doctest
This is exactly what it sounds like, it just tests documentation examples. In most respects the examples are expected to behave like the :unit tests described above, except that only the “Public API” functions rather than internally facing functions are tested; this is because those are the only functions which have documented examples.
All three of these test modes are run during CI and need to pass prior to merging into the release branch (and yes, this project is release oriented, not CD oriented).
Naturally, “unit testing” and “integration testing” are terms which are a bit loaded and tend to mean different things depending on who you’re speaking with. In this case, don’t read too much into those terms… I’m not thinking of any specific formalism beyond what is described above. While there are many faults that can be found in our testing strategy, and someday more rigor may be required, but for now this is good enough.
Of course, you can… but with tags you don’t need to “escape” those values… and also we have them available in setup, as I tried to show in that synthetic example.
Occasionally and sparingly I’ll use tags to influence the current setup block. It’s a little hard to describe with prose so I’ll let an example describe it:
defmodule TagsExample do
use ExUnit.Case
def setup(%{} = ctx) do
user =
case ctx[:user_type] do
:adult -> %{name: "Bob", age: 30}
:child -> %{name: "Johnny", age: 10}
end
%{user: user}
end
@tag user_type: :adult
test "adults can drive", %{user: user} do
assert AgeChecker.can_drive?(user) == true
end
@tag user_type: :child
test "children cannot drive", %{user: user} do
assert AgeChecker.can_drive?(user) == fase
end
end
I am very cautious to not overuse this approach because it can make it quite difficult to read the tests. I do mostly use it to control the type of user created inside the setup blocks (e.g. @tag admin: true).
I’ve experimented with the same thing and was debating about whether or not posting about it as I don’t really know how I felt about my usage of it. Moving back to a simpler project, I haven’t introduced it there, but it’s useful.
I was primarily using it to DRY up a complex LiveView that had many tests calling the same “show” route. Of course, the test data needs to be created before the route can be visited so I have this set up (which is very similar to yours!):
setup context do
if context[:with_blueprint] do
blueprint = insert(:blueprint)
{:ok, lv, _html} = live(context.conn, ~p"/blanks/#{blueprint.blank}")
%{lv: lv, blueprint: blueprint}
else
:ok
end
end
Which lets me do:
@tag :with_blueprint
test "cannot remove default unsaved blueprint", %{lv: lv} do
refute element?(lv, "#remove-blueprint-0")
end
(element? is just a helper for lv |> element("#remove-blueprint-0) |> has_element?())
I also had additional values that would let me set up some related data dynamically.
Using module attributes you also don’t need to escape them Hence, I believe module attributes are better suited - usually you don’t need these in setup but if you do, go ahead of course - however that makes it a lot more complex imo.
Ah thanks, so there is! I have a whole bunch of functions like this and initially I was toying around with unhygenic macro-versions to avoid passing lv all the time. I backed-out of that idea and converted them all to functions which is likely how I ended up with this straggler. I’ll probably keep it to keep inline with my little single-word function thing I got going on!
I have a nice little test helper to seed the standard random functions from the Mix test seed so that random tests can be (exactly) reproduced, without needing to use a single fixed random seed.
I’m still a noob when it comes to anything beyond basic testing. I haven’t figured out how best to handle tests to “external” APIs. I can mock them which is fine, but I like to have real tests that check against the real, actual service. That’s how I have envisioned the idea behind the external tag. (I’ve seen it used elsewhere too.)
Not sure that supports my “Integration Test” use case. The non-random, sequential test execution is a feature, not a bug, for this case; this is why not only is the random seed fixed, it’s fixed to “0”: this means the tests run in the specific order in which they are defined in the test script.
My “Integration Tests” prove the supported business processes, not the functions. Let’s say I have two functions create_user and authenticate_user. I do want to test that I can create the user, but, more importantly in this context, that the user created by create_user can be understood and works when processing function authenticate_user: the data must flow from one function to the next for the integration test to serve its purpose. It would be nice for the data values processed by these functions to originate randomly at the start of the integration test, but I want to see the results of earlier, data producing functions working well with later, data consuming functions.
My “Unit Tests” prove the functions, not the business processes. These tests are run asynchronously using a completely random seed and are focused on individual function workings. In this case, my example authenticate_user function can be tested in isolation at any time since I can establish pre-conditions (including pre-requisite data) for that specific test. It’s nice for any data seeding here to be random, too.
So in all cases, I’d like to have property-based-testing-like source data, but the testing goals have different needs when it comes to ordering execution.
Just off the cuff, sounds like a fine use of tags.
What you describe is a real testing scenario and it’s the kind of thing you often want on demand, not all the time.
One thing I do is specifically exclude tags so that I have to specifically request the run. I do that in test_helper.exs with something along the lines of:
test_kind =
if ExUnit.configuration() |> Keyword.get(:include) |> Enum.member?(:integration) do
ExUnit.configure(seed: 0)
:integration_testing
else
ExUnit.configure(exclude: [:integration])
:unit_testing
end
That excludes my integration tests so that if I don’t specify a test category that I get the unit tests only. Something similar would be useful to exclude external tests, too: you have to explicitly ask for testing external resources.
That’s exactly why I started using an :external tag in my own tests.
‘End-to-end’ tests are nice, but they’re also flakey, so it’s nice to be able to run them separately and NOT have CI/CD pipelines fail because the test service is down.
I wouldn’t write a test that depends on another one running (and passing) first/before. I’d just repeat the ‘earlier’ test(s) (or the relevant code) in the ‘dependent’ test.
(Really, I’d extract the common ‘setup’ code into a function that both tests can then call – mainly so I know that both tests should be ‘relying’ on the exact same ‘code behavior’.)
