Other than what’s written in docs I’d like to point to the trick that allows to “generate” very similar tests from a matrix..

setup %{role: role, status: status} do
  %{user: insert_user(role, status)}
end

for role <- [:admin, :guest],
    {status, color} <- [{:online, "green"}, {:offline, "gray"}, {:away, "yello"}] do
  @tag role: role, status: status, color: color
  test "#{role} renders #{color} badge when #{status}", tags do
    %{role: role, status: status, color: color} = tags
    # render and  assert on role, status, color
  end
end

Though, arguably these tests are now become a bit less clear.. as instead of

assert user.status == :online
assert user.role == :admin

now we would have something like

assert user.status == status
assert user.role == role

You don’t need tags for that though - you can use any custom module attribute or use escaping to embed the values in the test.

As for the original quesiton, benchee has a few, mostly dealing with different CI platforms/systems to run on:

• needs_fast_function_repetition - as that is not triggered on Linux, they essentially they are skipped when running tests on Linux
• performance - similar, but only run on Linux as the other CI’s are too slow
• millisecond_resolution_clock - property of the system
• another one that deals with whether we can run our memory measurement tests

For the interested: benchee/test/test_helper.exs at main · bencheeorg/benchee · GitHub

One that I don’t have yet but wanna built, benchee needs to retry things in some circumstances due to its nature. I’d like to move that from a function call to a tag.

I’m using (mostly) @moduletag rather than individual @tag attributes; the tests are organized to facilitate that sort of general handling.

My use is pretty simplistic/naïve. Most of the Elixir testing I have works with Elixir applications which can be thought of a libraries and the testing reflect this assumption. I have three values into which my tests are categorized:

• :unit

  Unit tests. These tests are exercising implementations as expressed by module public functions; while these functions are public in the sense that they use def in their definitions, they are in practice considered internal or private API. These test run asynchronously and if they involve persistent data (i.e. a database) then they have test data either seeded or generated specifically so that they don’t depend on/interfere with other tests that might be running.

• :integration

  Integration tests. This testing is focused on two goals: testing the Public API of each Elixir project and testing end-to-end, business process/data flows. These tests are synchronous and are run with a set random seed (of 0) so that ordering is deterministic. It is expected that data created or manipulated in earlier tests are to be used in later tests; any test related data seeding or pre-test data generation either is non-existent or there to populate data for dependencies of the project being tested. We really want to see that data being created in one process can be used in other downstream processes.

• :doctest

  This is exactly what it sounds like, it just tests documentation examples. In most respects the examples are expected to behave like the :unit tests described above, except that only the “Public API” functions rather than internally facing functions are tested; this is because those are the only functions which have documented examples.

All three of these test modes are run during CI and need to pass prior to merging into the release branch (and yes, this project is release oriented, not CD oriented).

Naturally, “unit testing” and “integration testing” are terms which are a bit loaded and tend to mean different things depending on who you’re speaking with. In this case, don’t read too much into those terms… I’m not thinking of any specific formalism beyond what is described above. While there are many faults that can be found in our testing strategy, and someday more rigor may be required, but for now this is good enough.