What test tags do you use in your Elixir projects?

I’m writing some tests and I want to build a “common language” for writing test tags. I’m using this post to document my findings as I learn about the built-in test tags, while also seeing what custom tags the Elixir community uses in their tests.

Built-In Tags

ExUnit has quite a few built-in tags:


  • @tag :skip or @tag skip: "some reason to skip" - Tests that should be skipped (unless run with --only skip). Support for this tag is built-in to ExUnit.

Custom Tags

Here are some of the custom tags I use or have encountered:

  • @tag :slow - Tests that take a long time to run.

  • @tag :external - Tests that call an external service or API

  • @tag :mock - Indicates that a test uses a mock to pass. (This tag can be paired with another test that uses @tag :external so there is an option to run a “live” version of the mocked test.)

  • @tag :fixme - A temporary placeholder tag that I use when I’m working on one specific test and only want it to run.

What test tags do you commonly use? Post them here so we can all speak the same language!

Other @tag tips, tricks, and hacks are welcome here too. :slight_smile:


Other than what’s written in docs I’d like to point to the trick that allows to “generate” very similar tests from a matrix…

setup %{role: role, status: status} do
  %{user: insert_user(role, status)}

for role <- [:admin, :guest],
    {status, color} <- [{:online, "green"}, {:offline, "gray"}, {:away, "yello"}] do
  @tag role: role, status: status, color: color
  test "#{role} renders #{color} badge when #{status}", tags do
    %{role: role, status: status, color: color} = tags
    # render and  assert on role, status, color

Though, arguably these tests are now become a bit less clear… as instead of

assert user.status == :online
assert user.role == :admin

now we would have something like

assert user.status == status
assert user.role == role

You don’t need tags for that though - you can use any custom module attribute or use escaping to embed the values in the test.

As for the original quesiton, benchee has a few, mostly dealing with different CI platforms/systems to run on:

  • needs_fast_function_repetition - as that is not triggered on Linux, they essentially they are skipped when running tests on Linux
  • performance - similar, but only run on Linux as the other CI’s are too slow
  • millisecond_resolution_clock - property of the system
  • another one that deals with whether we can run our memory measurement tests

For the interested: benchee/test/test_helper.exs at main · bencheeorg/benchee · GitHub

One that I don’t have yet but wanna built, benchee needs to retry things in some circumstances due to its nature. I’d like to move that from a function call to a tag.


Very cool. I started this thread just trying to get a “shared vocabulary”, but as I was writing it, I started discovering the depth of the @tag concept, how you could pass parameters to it, etc.

From what I can tell, there is definitely a lot of room for useful techniques to fully exploit this feature.

1 Like

I’m using (mostly) @moduletag rather than individual @tag attributes; the tests are organized to facilitate that sort of general handling.

My use is pretty simplistic/naïve. Most of the Elixir testing I have works with Elixir applications which can be thought of a libraries and the testing reflect this assumption. I have three values into which my tests are categorized:

  • :unit

    Unit tests. These tests are exercising implementations as expressed by module public functions; while these functions are public in the sense that they use def in their definitions, they are in practice considered internal or private API. These test run asynchronously and if they involve persistent data (i.e. a database) then they have test data either seeded or generated specifically so that they don’t depend on/interfere with other tests that might be running.

  • :integration

    Integration tests. This testing is focused on two goals: testing the Public API of each Elixir project and testing end-to-end, business process/data flows. These tests are synchronous and are run with a set random seed (of 0) so that ordering is deterministic. It is expected that data created or manipulated in earlier tests are to be used in later tests; any test related data seeding or pre-test data generation either is non-existent or there to populate data for dependencies of the project being tested. We really want to see that data being created in one process can be used in other downstream processes.

  • :doctest

    This is exactly what it sounds like, it just tests documentation examples. In most respects the examples are expected to behave like the :unit tests described above, except that only the “Public API” functions rather than internally facing functions are tested; this is because those are the only functions which have documented examples.

All three of these test modes are run during CI and need to pass prior to merging into the release branch (and yes, this project is release oriented, not CD oriented).

Naturally, “unit testing” and “integration testing” are terms which are a bit loaded and tend to mean different things depending on who you’re speaking with. In this case, don’t read too much into those terms… I’m not thinking of any specific formalism beyond what is described above. While there are many faults that can be found in our testing strategy, and someday more rigor may be required, but for now this is good enough.


Of course, you can… but with tags you don’t need to “escape” those values… and also we have them available in setup, as I tried to show in that synthetic example.

1 Like

Fun topic!

Occasionally and sparingly I’ll use tags to influence the current setup block. It’s a little hard to describe with prose so I’ll let an example describe it:

defmodule TagsExample do
  use ExUnit.Case

  def setup(%{} = ctx) do
    user =
      case ctx[:user_type] do
        :adult -> %{name: "Bob", age: 30}
        :child -> %{name: "Johnny", age: 10}

    %{user: user}

  @tag user_type: :adult
  test "adults can drive", %{user: user} do
    assert AgeChecker.can_drive?(user) == true

  @tag user_type: :child
  test "children cannot drive", %{user: user} do
    assert AgeChecker.can_drive?(user) == fase

I am very cautious to not overuse this approach because it can make it quite difficult to read the tests. I do mostly use it to control the type of user created inside the setup blocks (e.g. @tag admin: true).

In the past I’ve used the “scenario” approach mentioned in this blog post: Maintainable test setup with scenario pipelines - 9elements
But it irks me how you need to read it “backwards”. Maybe that could be reversed with a sprinkling of macros somehow (ala GitHub - mtrudel/machete: Literate test matchers for ExUnit).


I’ve experimented with the same thing and was debating about whether or not posting about it as I don’t really know how I felt about my usage of it. Moving back to a simpler project, I haven’t introduced it there, but it’s useful.

I was primarily using it to DRY up a complex LiveView that had many tests calling the same “show” route. Of course, the test data needs to be created before the route can be visited so I have this set up (which is very similar to yours!):

setup context do
  if context[:with_blueprint] do
    blueprint = insert(:blueprint)

    {:ok, lv, _html} = live(context.conn, ~p"/blanks/#{blueprint.blank}")

    %{lv: lv, blueprint: blueprint}

Which lets me do:

@tag :with_blueprint
test "cannot remove default unsaved blueprint", %{lv: lv} do
  refute element?(lv, "#remove-blueprint-0")

(element? is just a helper for lv |> element("#remove-blueprint-0) |> has_element?())

I also had additional values that would let me set up some related data dynamically.


Using module attributes you also don’t need to escape them :slight_smile: Hence, I believe module attributes are better suited - usually you don’t need these in setup but if you do, go ahead of course - however that makes it a lot more complex imo.


Hey, there is has_element?(view, selector, text_filter \\ nil) :slightly_smiling_face:

assert has_element?(view, "#remove-blueprint-0")
1 Like

Ah thanks, so there is! I have a whole bunch of functions like this and initially I was toying around with unhygenic macro-versions to avoid passing lv all the time. I backed-out of that idea and converted them all to functions which is likely how I ended up with this straggler. I’ll probably keep it to keep inline with my little single-word function thing I got going on!


I just added some tests with external: true tags; not quite sure what to do with CI/CD; probably ‘warn but don’t fail’ for those tests.

You might like ExUnit.Parameterized; very readable.


I have a nice little test helper to seed the standard random functions from the Mix test seed so that random tests can be (exactly) reproduced, without needing to use a single fixed random seed.

I’m still a noob when it comes to anything beyond basic testing. I haven’t figured out how best to handle tests to “external” APIs. I can mock them which is fine, but I like to have real tests that check against the real, actual service. That’s how I have envisioned the idea behind the external tag. (I’ve seen it used elsewhere too.)

1 Like

Not sure that supports my “Integration Test” use case. The non-random, sequential test execution is a feature, not a bug, for this case; this is why not only is the random seed fixed, it’s fixed to “0”: this means the tests run in the specific order in which they are defined in the test script.

My “Integration Tests” prove the supported business processes, not the functions. Let’s say I have two functions create_user and authenticate_user. I do want to test that I can create the user, but, more importantly in this context, that the user created by create_user can be understood and works when processing function authenticate_user: the data must flow from one function to the next for the integration test to serve its purpose. It would be nice for the data values processed by these functions to originate randomly at the start of the integration test, but I want to see the results of earlier, data producing functions working well with later, data consuming functions.

My “Unit Tests” prove the functions, not the business processes. These tests are run asynchronously using a completely random seed and are focused on individual function workings. In this case, my example authenticate_user function can be tested in isolation at any time since I can establish pre-conditions (including pre-requisite data) for that specific test. It’s nice for any data seeding here to be random, too.

So in all cases, I’d like to have property-based-testing-like source data, but the testing goals have different needs when it comes to ordering execution.

1 Like

Just off the cuff, sounds like a fine use of tags.

What you describe is a real testing scenario and it’s the kind of thing you often want on demand, not all the time.

One thing I do is specifically exclude tags so that I have to specifically request the run. I do that in test_helper.exs with something along the lines of:

test_kind =
  if ExUnit.configuration() |> Keyword.get(:include) |> Enum.member?(:integration) do
    ExUnit.configure(seed: 0)
    ExUnit.configure(exclude: [:integration])

That excludes my integration tests so that if I don’t specify a test category that I get the unit tests only. Something similar would be useful to exclude external tests, too: you have to explicitly ask for testing external resources.


That’s exactly why I started using an :external tag in my own tests.

‘End-to-end’ tests are nice, but they’re also flakey, so it’s nice to be able to run them separately and NOT have CI/CD pipelines fail because the test service is down.

I wouldn’t write a test that depends on another one running (and passing) first/before. I’d just repeat the ‘earlier’ test(s) (or the relevant code) in the ‘dependent’ test.

(Really, I’d extract the common ‘setup’ code into a function that both tests can then call – mainly so I know that both tests should be ‘relying’ on the exact same ‘code behavior’.)