Best practices for testing behaviour implementations concurrently

Questions / Help
, , ,
1

I have seen some good examples for best practices in sharing test suites:
Test a module for a given behaviour
Test a behaviour that uses super
How to test a behaviour
reusing a test suite

I am still curious on what’s the best practice for swapping behaviour implementations through config while allowing tests to run concurrently. Primarily from a library perspective. I have managed to come right using Mox (option 2 below), but I can’t shake the feeling that there are better ways to do it. I would love thoughts and input.

Example

Let’s say I have an adapter module

defmodule MyApp.Adapter do

  @callback action_1() :: :ok | :error
  @callback action_2() :: :ok | :error
  @callback action_3() :: :ok | :error

  def action_1, do: impl.action_1

  def action_2, do: impl.action_2

  def action_3, do: impl.action_3

  def impl, do: Application.get_env(:my_app, :adapter, AdapterImpl)
end

for completeness it can be used by something like this

defmodule MyApp.AdapterCaller do
alias MyApp.Adapter

  def call_adapter("action_1"), do: Adapter.action_1
  def call_adapter("action_2"), do: Adapter.action_2
  def call_adapter("action_3"), do: Adapter.action_3
end

Lets say 3 adapters implement the behaviour namely AdapterOne, AdapterTwo, AdapterThree

I have referenced quite a few code bases and gone through ExUnit and Mox docs, but haven’t felt “Yeah this is the way to lock in”.

It is pretty simple to setup generic tests that can be used for all implementations (especially following the above links), but using the application config means that I can’t run them in parallel for no reason other than setup and I want that concurrency.

In trying to figure out which way I want to go, I have explored the following ways to manage it:

1. Set the config during setup

setup do
Application.put_env(:my_app, :adapter, AdapterOne)
end

In terms of being able to test, this works fine, but because of the shared state of the config, I can’t run them in parallel because it would swap adapters mid test if I’m unlucky

2. Use Mox and create a Mock Adapter, Set the Env and Mock returns.

in test_helper.exs

Mox.defmock(MockAdapter, for: MyApp.Adapter)
Application.put_env(:my_app, :adapter, MockAdapter)

and in the test

Mox.expect(MockAdapter, :action_1, fn -> AdapterOne.action_1 end)

This is my preferred approach because of the parameter checking and other assertions I get for free, but I’m not married to it.
It solves the shared configuration issue, and it looks like I can avoid conflicts with its allowances and some setup stuff I will play with after I posted that.
Is this generally what is recommended?

3. Put Env in TestCases

I avoid this because it’s basically the same as 1. but catches you when you least expect it

4. Change code to allow dependency injection.

If I change my adapter implementation to

defmodule MyApp.AdapterCaller do
alias MyApp.Adapter

  def call_adapter("action_1", adapter \\AdapterOne), do: adapter.action_1
  def call_adapter("action_2", adapter \\AdapterOne), do: adapter.action_1
  def call_adapter("action_3", adapter \\AdapterOne), do: adapter.action_1
end

It becomes simple to test concurrently because I’m passing the adapter I’m testing. There are occasions where this is my goto approach for general design and without defaults, but it’s incredibly situational.

  • If the behaviour is nested deep in your code base, you have a cascading expansion of contract changes but also any caller needs to manage an instance that should be a config.
  • Designing code to be testable is good, but changing the design of code in order to test is bad.

This approach is in my pocket but only appropriate <40% of the time.

5. Do unnecessary things with a mock instance

Definitely don’t do this, but listing as it’s a thought that crossed my mind.
if you have a mock instance created somewhere

Application.put_env(:my_app, :adapter, MockAdapter)

You can create a set of parameters and configs that ensures the correct instance is returned based on the test.
While possible this quickly becomes an unmaintanable mess and should go no further than a thought experiment.

6. Mix test setup and params.

An option I think is valid but I have not explored it properly, but I think you can setup suites and run sets of tests in isolation.
When do people usually take this approach? Is it more for performance than isolating collisions?

7. Partitions coming in 1.18

Having read through it, this seems like another way you could manage setting environments without conflict
Changelog

Would love to know thoughts or better approaches!

1 Like
2

Why test through MyApp.Adapter in the first place? Test AdapterOne, AdapterTwo, AdapterThree concurrently through a shared set of testcases. No need to deal with the complexity of global state to begin with.

2 Likes
3

In the example that’s a fair point, I might have over simplified in what I wrote.
In many cases that is preferred yes.

To expand a bit.

  • For unit tests, to your point if you have tested each adapter independently, when you test the Adapter itself you can take any implementation, because you’re primarily focused on responding to what the behaviour can return
  • For integration tests I don’t think it’s as straight forward at a certain point of complexity. In one of the packages I’m working on it is the difference between Redis and Postgres and to test an important scenario there are many interactions with the adapter, and while the responses will match the contract, the expected behaviours might not. I’m not sure how you would test them in isolation while having confidence that the scenarios are going to be correct.

I hope that makes sense?

4

I’d ask if you really need to go through MyApp.Adapter to get to that behaviour difference.

But also you added the Application.get_env making the adapter selection be based on global state. You could choose a different path and base the adapter selection on something you can control per (async) integration tests, which is basically your 4. point.

Personally I feel like many discussions like these exist because there’s code in the tested codebases, which make dependency injection impossible just to then need more complexity to be able to still do flexible dependency injection for tests. I prefer removing the parts in the code preventing dependency injection. Reasonable dependency injection roots (where selections are being made) are either application start callbacks for application wide selection or endpoints for webrequest local selection. Having those selections be scattered throughout a codebase is not a great idea in my mind.

Sometimes you have external dependencies (as in cannot be run locally, not just being external to elixir) where you then need some mocking to be able to flexibly not just select a module, but also how it acts. Those are where Mox and similar tools imo become actually useful, rather than also using it for everything remotely related to dependency injection.

3 Likes
5

IMO your point 4 is the best you can do. As @LostKobrakai said, make sure not to “infect” too many modules / functions in your app with the ability to supply an implementation though – that dependency-injection surface should be minimal, and it does not make sense to allow it in too many places anyway (f.ex. why would you be able to use the real implementation of a 3rd party API in one context / business-domain function but then immediately use a mock downstream of that function, right?).

1 Like
6

Thanks for the response. I had a bit of a think on this.

I strongly agree with the Mocks side of things. The only things you should Mock are as you say things you can’t really control for. My most common example being s3.

I had a look at my application and library code thinking about this. Application wise, I’ve been fairly consistent with that approach, even with testing behaviours there’s always something else that needs to be setup closer to the execution of different behaviours.

I realised for library code, I struggled to let go of mental models of configs from other languages (Ruby - you can do whatever you want, C# - You have a config interface, you can make a virtual singleton for dynamically).

The reason why I was torn between mocks and DI was how I had framed it for myself, and created this topic because I felt something was wrong.

Essentially my resistance to pure DI was the config can have more behaviours set than one, and a growing set of config parameters passed from entry point to the root, felt off.
To avoid function signature bloat, the usual approach is to create a wrapper input struct. Usually it puts a bad taste in my mouth because it can lead to optional signature bloat and hidden dependencies.

That said an input struct makes sense for the correct model, and it would simplify everything.
I can either do a config_behaviours struct, that can be passed from the entry point and it would be constant as the application grows, and constructed from the entrypoint like you mentioned.
I also have the agency to refine the input if I feel like I’m passing bloat around.

So thanks for the response, it helped a lot in nudging me the right direction.

TL;DR DI is definitely the correct route after thinking about it.

7

Since you settled on DI, there’s an approach which, for some reason, is not the blessed way but in my opinion is very good.

You define a protocol with your top-level functions. Then, you implement the protocol for a struct which holds the dependencies. The struct data defaults to environment configuration. They can be mocked, real, faked, etc. The struct can be created via a new function, and its data can be overridden by the function’s arguments.with

The advantage is that you can have multiple implementers of your business logic protocol. You gain flexibility but also unfamiliarity with the wider community.

1 Like
8

To me protocols are mostly a persona non-grata simply because they are a bit… implicit. You have a variable, you call a function on it and you are not sure what gets called.

I realize it’s a flimsy and slippery slope but to me having the ability to dispatch explicitly somewhere (including in config) is a touch more valuable.

I don’t feel strongly about it, mind you, it was a choice made almost at the beginning of my Elixir journey so it’s likely not a very up-to-date opinion either.

1 Like