ryanzidago

ryanzidago

Each tests should own its setup

Hi all,

In this article, I make the case for each test owning its setup.
Usually I forbid my AI agents to use the setup callbacks; I much prefer when tests are portable.

For example at work during code reviews, I often ask the AI agent to prove its finding by writing a failing test. Once the test is drafted, I ask it to publish it as a GitHub inline comment:

  • I use the test as a tool to illustrate the agent’s point
  • If the test assertions are valid, the author of the PR just needs to fix it
  • the setup does not get in the way because the test is self-sufficient to be understood, you can just read the comment and get it, instead of trying to find the corresponding setup.

Even with huge setups, I prefer no common setup callbacks and have each test manually defininig their setup.

Let me know what you think!

Most Liked

Awlexus

Awlexus

Nitpick for the first example, I think you can completely side-step problem with the magic number by using the product price, like the test said it would

  test "uses the product price as the order total", %{
    customer: customer,
    product: product
  } do
    assert {:ok, order} = Orders.create(customer, product)
    assert order.total == product.price
  end

wrt to large, shared setup blocks like the one in your example

setup do
  account = insert(:account)
  admin = insert(:user, role: :admin, account: account)
  employee = insert(:user, role: :employee, account: account)
  subscription = insert(:subscription, account: account)
  product = insert(:product)

  %{
    account: account,
    admin: admin,
    employee: employee,
    subscription: subscription,
    product: product
  }
end

Elixir already provides tools that make it possible to balance the of setup blocks

Setup functions can be changed

Instead of writing one large block you can build composable setup blocks. for example

setup [:with_account, :with_role, :with_subscription, :with_product]

Then define the setup blocks

def with_account(_context) do
  {:ok, account: insert(:account)}
end

def with_subscription(context) do
  {:ok, subscription: insert(:subscription, account: account)}
end

Using tags to modify setups per test

using @tag we can inject some values into the context ourselves, which we can use to modify shared setups to the needs of the test.

def with_product(context) do
  {:ok, product: insert(:product, Map.get(context, :product_data, %{}))
end

Using these features the first test could be rewritten like this, allowing you to keep the tests (Not that I recommend this in particular, because using the product price directly would be preferred)

  @tag product_data: %{price: 1000}
  test "uses the product price as the order total", %{
    customer: customer,
    product: product
  } do
    assert {:ok, order} = Orders.create(customer, product)
    assert order.total == 1000
  end

I do not disagree with the points presented, but I prefer the ability to compose and to extend when possible. Also dicipline your Agents early on with DRY or else you’ll find the same function 5 times in different places and with different names.

krasenyp

krasenyp

Good article but there’s an issue with the examples. Too much reliance on ExMachina. This can bite you pretty bad because inserts like this might not encode invariants which the business logic expects. Use your app’s API to prepare the data for your tests. If you can’t then you have a big problem.

nathanl

nathanl

I strongly agree. If we want to DRY up duplication, we can do it like we do anywhere else, using functions.

test "does a thing" do
%{stuff: stuff} = setup_the_stuff(%{which: :stuff})
# test content here
end

With this approach, if I want to know what stuff was setup, I know to inspect the function(s) that the test called and/or the data they returned.

Another article that argues the same case:

Last Post!

fuelen

fuelen

I argued the general case for this in another thread so I won’t repeat it.

I’d say mocks are a one-time cost with Mox rather than a per-test one. Global stub in setup, expect only in the tests that assert on the interaction.

Side effects are real, and my answer is the same as in that thread. Skip the normal flow for the one step that’s too heavy, with a comment saying why.

I’d flip this around. The absence of it is the bigger cost. The schema grows an intermediate table, the real flow starts creating a membership, and insert(:user, company: company) stays green. Nothing failed, so nothing told you that your setup now describes a state your application no longer produces. Setup that goes through your context functions either breaks loudly when the flow changes or picks up the change on its own.

The “centralized setup layer built on functions” I recommended there is a library I maintain, I just didn’t name it at the time: seed_factory. It resolves dependent entities automatically, which is what makes going through business logic affordable.

Which brings me back to your original point. Rewriting ex_machina tests this way over the last three years, the setup often comes out shorter. I noticed that we mostly stopped moving setup into setup blocks. My read is that people extract setup mainly once it gets long, so shared setup is partly a symptom of expensive setup.

One day I’ll write the blog post about the road from ex_machina to seed_factory :slight_smile:

Where Next?

Popular in Blog Posts Top

marcelo
Long story short, over the past years we’ve been scaling our operation quite a lot at Unbabel. Most of our codebase is Python and some of...
New
alvises
Just published the first part of series of articles aimed to explain the architecture behind a kv-store engine written in Elixir and impl...
New
pillaiindu
Our very handsome and humble José Valim (@josevalim) just published Kubernetes and the Erlang VM: orchestration on the large and the sma...
New
paulanthonywilson
I had a bit of a mini-adventure following Sobelow’s advice on adding a CSP to a Phoenix App. If you want to follow along, or want to add ...
New
wmnnd
Here’s the story how one of the world’s first production deployments of LiveView came to be - and how trying to improve it almost caused ...
New
ryotsu
How does a torrent client download huge files from the internet with just a .torrent file? In this post we’ll explore the BitTorrent prot...
New
gaggle
This post explores different ways to do test automation in Elixir, focusing on how to handle dependency injection — covering patterns, li...
New

Other popular topics Top

New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54996 245
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 40082 209
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New

We're in Beta

About us Mission Statement