Other than what’s written in docs I’d like to point to the trick that allows to “generate” very similar tests from a matrix..

setup %{role: role, status: status} do
  %{user: insert_user(role, status)}
end

for role <- [:admin, :guest],
    {status, color} <- [{:online, "green"}, {:offline, "gray"}, {:away, "yello"}] do
  @tag role: role, status: status, color: color
  test "#{role} renders #{color} badge when #{status}", tags do
    %{role: role, status: status, color: color} = tags
    # render and  assert on role, status, color
  end
end

Though, arguably these tests are now become a bit less clear.. as instead of

assert user.status == :online
assert user.role == :admin

now we would have something like

assert user.status == status
assert user.role == role

You don’t need tags for that though - you can use any custom module attribute or use escaping to embed the values in the test.

As for the original quesiton, benchee has a few, mostly dealing with different CI platforms/systems to run on:

• needs_fast_function_repetition - as that is not triggered on Linux, they essentially they are skipped when running tests on Linux
• performance - similar, but only run on Linux as the other CI’s are too slow
• millisecond_resolution_clock - property of the system
• another one that deals with whether we can run our memory measurement tests

For the interested: benchee/test/test_helper.exs at main · bencheeorg/benchee · GitHub

One that I don’t have yet but wanna built, benchee needs to retry things in some circumstances due to its nature. I’d like to move that from a function call to a tag.

Very cool. I started this thread just trying to get a “shared vocabulary”, but as I was writing it, I started discovering the depth of the @tag concept, how you could pass parameters to it, etc.

From what I can tell, there is definitely a lot of room for useful techniques to fully exploit this feature.

I’m using (mostly) @moduletag rather than individual @tag attributes; the tests are organized to facilitate that sort of general handling.

My use is pretty simplistic/naïve. Most of the Elixir testing I have works with Elixir applications which can be thought of a libraries and the testing reflect this assumption. I have three values into which my tests are categorized:

• :unit

  Unit tests. These tests are exercising implementations as expressed by module public functions; while these functions are public in the sense that they use def in their definitions, they are in practice considered internal or private API. These test run asynchronously and if they involve persistent data (i.e. a database) then they have test data either seeded or generated specifically so that they don’t depend on/interfere with other tests that might be running.

• :integration

  Integration tests. This testing is focused on two goals: testing the Public API of each Elixir project and testing end-to-end, business process/data flows. These tests are synchronous and are run with a set random seed (of 0) so that ordering is deterministic. It is expected that data created or manipulated in earlier tests are to be used in later tests; any test related data seeding or pre-test data generation either is non-existent or there to populate data for dependencies of the project being tested. We really want to see that data being created in one process can be used in other downstream processes.

• :doctest

  This is exactly what it sounds like, it just tests documentation examples. In most respects the examples are expected to behave like the :unit tests described above, except that only the “Public API” functions rather than internally facing functions are tested; this is because those are the only functions which have documented examples.

All three of these test modes are run during CI and need to pass prior to merging into the release branch (and yes, this project is release oriented, not CD oriented).

Naturally, “unit testing” and “integration testing” are terms which are a bit loaded and tend to mean different things depending on who you’re speaking with. In this case, don’t read too much into those terms… I’m not thinking of any specific formalism beyond what is described above. While there are many faults that can be found in our testing strategy, and someday more rigor may be required, but for now this is good enough.

Of course, you can.. but with tags you don’t need to “escape” those values.. and also we have them available in setup, as I tried to show in that synthetic example.

Fun topic!

Occasionally and sparingly I’ll use tags to influence the current setup block. It’s a little hard to describe with prose so I’ll let an example describe it:

defmodule TagsExample do
  use ExUnit.Case

  def setup(%{} = ctx) do
    user =
      case ctx[:user_type] do
        :adult -> %{name: "Bob", age: 30}
        :child -> %{name: "Johnny", age: 10}
      end

    %{user: user}
  end

  @tag user_type: :adult
  test "adults can drive", %{user: user} do
    assert AgeChecker.can_drive?(user) == true
  end

  @tag user_type: :child
  test "children cannot drive", %{user: user} do
    assert AgeChecker.can_drive?(user) == fase
  end
end

I am very cautious to not overuse this approach because it can make it quite difficult to read the tests. I do mostly use it to control the type of user created inside the setup blocks (e.g. @tag admin: true).

In the past I’ve used the “scenario” approach mentioned in this blog post: Maintainable test setup with scenario pipelines - 9elements
But it irks me how you need to read it “backwards”. Maybe that could be reversed with a sprinkling of macros somehow (ala GitHub - mtrudel/machete: Literate test matchers for ExUnit · GitHub).

I’ve experimented with the same thing and was debating about whether or not posting about it as I don’t really know how I felt about my usage of it. Moving back to a simpler project, I haven’t introduced it there, but it’s useful.

I was primarily using it to DRY up a complex LiveView that had many tests calling the same “show” route. Of course, the test data needs to be created before the route can be visited so I have this set up (which is very similar to yours!):

setup context do
  if context[:with_blueprint] do
    blueprint = insert(:blueprint)

    {:ok, lv, _html} = live(context.conn, ~p"/blanks/#{blueprint.blank}")

    %{lv: lv, blueprint: blueprint}
  else
    :ok
  end
end

Which lets me do:

@tag :with_blueprint
test "cannot remove default unsaved blueprint", %{lv: lv} do
  refute element?(lv, "#remove-blueprint-0")
end

(element? is just a helper for lv |> element("#remove-blueprint-0) |> has_element?())

I also had additional values that would let me set up some related data dynamically.

Using module attributes you also don’t need to escape them Hence, I believe module attributes are better suited - usually you don’t need these in setup but if you do, go ahead of course - however that makes it a lot more complex imo.

Hey, there is has_element?(view, selector, text_filter \\ nil)

assert has_element?(view, "#remove-blueprint-0")

Ah thanks, so there is! I have a whole bunch of functions like this and initially I was toying around with unhygenic macro-versions to avoid passing lv all the time. I backed-out of that idea and converted them all to functions which is likely how I ended up with this straggler. I’ll probably keep it to keep inline with my little single-word function thing I got going on!