Each tests should own its setup

I strongly agree. If we want to DRY up duplication, we can do it like we do anywhere else, using functions.

test "does a thing" do
%{stuff: stuff} = setup_the_stuff(%{which: :stuff})
# test content here
end

With this approach, if I want to know what stuff was setup, I know to inspect the function(s) that the test called and/or the data they returned.

Another article that argues the same case:

Ah yes! Strong article as well with strong arguments, especially with unused data and mystery guests!

Nitpick for the first example, I think you can completely side-step problem with the magic number by using the product price, like the test said it would

  test "uses the product price as the order total", %{
    customer: customer,
    product: product
  } do
    assert {:ok, order} = Orders.create(customer, product)
    assert order.total == product.price
  end

wrt to large, shared setup blocks like the one in your example

setup do
  account = insert(:account)
  admin = insert(:user, role: :admin, account: account)
  employee = insert(:user, role: :employee, account: account)
  subscription = insert(:subscription, account: account)
  product = insert(:product)

  %{
    account: account,
    admin: admin,
    employee: employee,
    subscription: subscription,
    product: product
  }
end

Elixir already provides tools that make it possible to balance the of setup blocks

Setup functions can be changed

Instead of writing one large block you can build composable setup blocks. for example

setup [:with_account, :with_role, :with_subscription, :with_product]

Then define the setup blocks

def with_account(_context) do
  {:ok, account: insert(:account)}
end

def with_subscription(context) do
  {:ok, subscription: insert(:subscription, account: account)}
end

Using tags to modify setups per test

using @tag we can inject some values into the context ourselves, which we can use to modify shared setups to the needs of the test.

def with_product(context) do
  {:ok, product: insert(:product, Map.get(context, :product_data, %{}))
end

Using these features the first test could be rewritten like this, allowing you to keep the tests (Not that I recommend this in particular, because using the product price directly would be preferred)

  @tag product_data: %{price: 1000}
  test "uses the product price as the order total", %{
    customer: customer,
    product: product
  } do
    assert {:ok, order} = Orders.create(customer, product)
    assert order.total == 1000
  end

I do not disagree with the points presented, but I prefer the ability to compose and to extend when possible. Also dicipline your Agents early on with DRY or else you’ll find the same function 5 times in different places and with different names.

yes! I’ve been advocating for this for a long time and I’m happy that I managed to fully align my team around it early on

Good article but there’s an issue with the examples. Too much reliance on ExMachina. This can bite you pretty bad because inserts like this might not encode invariants which the business logic expects. Use your app’s API to prepare the data for your tests. If you can’t then you have a big problem.

This can bite you pretty bad because inserts like this might not encode invariants which the business logic expects.

Interesting, and it might be worth another blog post!

I agree that insert(:product) can potentially create a database-valid record that would be impossible to create through the real application.

There is a tradeoff to using your application’s business logic for test setup:

• You may need to set up an organisation and many related records.
• You may need to configure permissions.
• You may need to mock external systems such as third-party APIs.
• You may rerun side effects that make the tests slower and do not matter for the behavior being tested.
• Every time you change the API, you may need to update many tests.
• etc.

I haven’t encountered enough factory-related problems to justify paying those costs in every test.

For focused tests, I still prefer simple ExMachina-style factories whose defaults represent valid domain state, while keeping the values relevant to the behavior explicit in the test.

Though I would definitely use the application’s business logic when the workflow or the invariants it establishes are part of what the test needs to prove / if it makes setting up the test scenario significantly easier:

test "cancelling an approved holiday restores the employee's balance" do
  employee = insert(:employee, annual_holiday_days: 20)
  manager = insert(:manager)

  {:ok, request} =
    Holiday.request(employee, days: 5)

  {:ok, approved_request} =
    Holiday.approve(request, manager)

  assert Holiday.remaining_days(employee) == 15

  assert {:ok, _cancelled_request} =
           Holiday.cancel(approved_request, employee)

  assert Holiday.remaining_days(employee) == 20
end

I argued the general case for this in another thread so I won’t repeat it.

I’d say mocks are a one-time cost with Mox rather than a per-test one. Global stub in setup, expect only in the tests that assert on the interaction.

Side effects are real, and my answer is the same as in that thread. Skip the normal flow for the one step that’s too heavy, with a comment saying why.

I’d flip this around. The absence of it is the bigger cost. The schema grows an intermediate table, the real flow starts creating a membership, and insert(:user, company: company) stays green. Nothing failed, so nothing told you that your setup now describes a state your application no longer produces. Setup that goes through your context functions either breaks loudly when the flow changes or picks up the change on its own.

The “centralized setup layer built on functions” I recommended there is a library I maintain, I just didn’t name it at the time: seed_factory. It resolves dependent entities automatically, which is what makes going through business logic affordable.

Which brings me back to your original point. Rewriting ex_machina tests this way over the last three years, the setup often comes out shorter. I noticed that we mostly stopped moving setup into setup blocks. My read is that people extract setup mainly once it gets long, so shared setup is partly a symptom of expensive setup.

One day I’ll write the blog post about the road from ex_machina to seed_factory