Strategies to manipulate test spies and stubs?

[Fl4m3Ph03n1x]

Background

Recently I have thrown again into the see of testing my functional code with Elixir. However I am finding it rather difficult to create tests for basic manager / collaborators communication because I find Elixir’s toolset lacking when it comes to spies and stubs. Pure functions only whose dependencies get injected, no mocks here, no modules.

Objective

Several approaches have been suggested to me when testing stubs and spies. The objective of this discussion is to have a (somewhat) comprehensive lists of strategies used with their pros and cons.

You may not 100% agree with the definitions I am about to suggest, but for the technical purposes of this discussion let’s simply go with them (you may disagree with me in another discussion )

Spies

A spy’s main objective is to tell how information about a function, such as:

1. Was a function invoked?
2. How many times it was invoked
3. With what parameters it was invoked

Stubs

A stub’s main purpose is to direct the workflow of your program. This basically boils down to:

1. returning an output you define at test level

You may force a function to raise an error or return a given value.

Common approaches

Following I discuss some common approaches to testing.

test process sends messages to self()

How to test side effects on other processes?

Background I have a module that makes a async request to another service. This async request is made via Task.start and I literally don’t care if the request fails or not, I only want to make it. My objective here, is to test whether or not I am making the request with the correct parameters. Code The function send takes in a params string and an options keyword list. This keywords list allows me to inject the function that actually makes the HTTP GET request opts[:request_fn].(). def send(p…

pros

• check a given function was called
• check with which arguments it was called
• does not use global state

cons

• cannot tell you a function was invoked precisely X times
• cannot change return value based on how many times a function was invoked

usage of :counters (requires erlang 21.2)

Test function was called exactly X times

If you are using OTP 21+ then you can write it as: @tag :wip test "calls given function X times" do counter = :counters.new(1, []) deps = [ lookup_fn: fn _key -> :counters.add(counter, 1, 1) {:ok, 1} end ] MyApp.do_work(deps) assert 2 == :counters.get(counter, 1) end

pros

• allows you to know exactly how many times a given function was invoked
• allows you to, based on the number of times a function was called, return different results
• does not use global state

cons

• doesn’t allow you to know with which parameters a function was called with

More?

There are more solutions out there. @peerreynders suggested the use of named ETS tables to store state in tests and @hauleth the use of Agents. I look forward to having them join this conversation (if they want to ofc).

What other strategies do you know?

[hauleth]

@Fl4m3Ph03n1x

cannot change return value based on how many times a function was invoked

As I said earlier, such behaviour is in general discouraged in functional programming as this mean that given function is not “pure”, aka it’s output do not depend solely on the input (aka result value do not depends only on arguments). You should avoid impure functions whenever possible.

@Fl4m3Ph03n1x

named ETS tables

You name all ETS tables, but you do not need to use “named ETS” tables, and when you use “unnamed tables” then nothing prevents you from generating tables with the same name:

iex(1)> a = :ets.new(:foo, []) 
#Reference<0.3962131891.4068868100.257019>
iex(2)> b = :ets.new(:foo, [])
#Reference<0.3962131891.4068868100.257044>
iex(3)> :ets.insert(a, {:a, 1})
true
iex(4)> :ets.insert(b, {:b, 1})
true
iex(5)> :ets.match(a, :'$1')
[[a: 1]]
iex(6)> :ets.match(b, :'$1')
[[b: 1]]

And you can use them in exactly the same way as :counters, which is via tid() instead of atomic name.

[Fl4m3Ph03n1x]

@hauleth

As I said earlier, such behaviour is in general discouraged in functional programming as this mean that given function is not “pure”, aka it’s output do not depend solely on the input (aka result value do not depends only on arguments). You should avoid impure functions whenever possible.

True. Let’s imagine you have a function that does a GET request to some service:

def my_request(val) do
  data = val + 1
  my_lib.get("some_website/val")
end

This would be not only impure, but also highly coupled to the library and untestable. The preferred way to do it would be:

def my_request(val, deps \\ [get_fn: my_lib.get]) do
  data = val + 1
  deps[:get_fn].("some_website/val")
end

Now you can test it and the code is loosely coupled because you inject the dependency but it is still impure.

Am I doomed to never know if my system using this function works properly?
No.

You don’t need to know what the dependency does in your code. All you need to do is that this is a call to an external service and that you are performing it. The basis of this ideology comes from Sandi Metz Magic Tricks which I totally recommend.

All you need to do (in this case) is to make sure the function was invoked with the right parameters. More complex cases may invoke the same function several times in different stages of processing, and this is where my needs fit. Once again, I don;t care what is happening, I just care I am invoking the call to an external party with the right parameters.

What about the pure functional way?

There is even a better way to do this. A way that is pure. In one word - Monads. But people here seem to have an aversion to them, so I am not going to bore the only person who had the good will of replying to my post

[LostKobrakai]

@Fl4m3Ph03n1x

Now you can test it and the code is loosely coupled because you inject the dependency but it is still impure.

But you also introducted the idea of injected dependencies, which need to be set somewhere. And that somewhere is most often some global entity, unless you can inject from within the test itself. If you indeed can inject from the test then you should be just fine with e.g. mox Mox — Mox v0.5.0 even if multiple processes (concurrent testing) are involved.

Edit: The difficult part here is how to let :get_fn be aware of where to send data like “I was called with args”, so that the test can check those. If you can directly inject e.g. the pid of the test (and each test has it’s own pid) then the rest is just implementation of sending the correct data back.

[hauleth]

It depends on the my_lib and there is few possible approaches to that:

Mock my_lib

You can use for example mockery in form of:

def my_request(val) do
  mockable(MyLib).get("some_website/val")
end

And then in your test you can use:

test "calls MyLib.get/1" do
  mock MyLib, [get: 1], fn _data -> System.unique_integer() end

  Subject.my_request(10)

  assert_called MyLib, get: 1
end

Or if you want sequence of numbers

test "calls MyLib.get/1" do
  counter = :counters.new(1, [])

  mock MyLib, [get: 1], fn _data ->
    :ok = :counters.add(counter, 1, 1)
    :counters.get(counter, 1)
  end

  Subject.my_request(10)

  assert_called MyLib, get: 1
end

Or if you are using Erlang <21.2

test "calls MyLib.get/1" do
  pid = start_supervised!({Agent, 0})

  mock MyLib, [get: 1], fn _data ->
    Agent.get_and_update(pid, &{&1 + 1, &1 + 1})
  end

  Subject.my_request(10)

  assert_called MyLib, get: 1
end

Mock HTTP client

Assume that my_lib use Tesla library for requests then you can use:

setup do
  Application.put_env(:tesla, MyLib.HTTPClient, adapter: Tesla.Mock
end

test "API does request to `http://example.com`" do
  mock fn ->
    %{method: :get, url: "http://example.com/hello"} ->
        %Tesla.Env{status: 200, body: "#{System.unique_integer()}"}
  end

  Subject.my_request(10)
end

Mock target service

If target URL is configurable then you can use solution like Bypass to create fake target of the request instead of mocking MyLib or HTTP client. I used this solution for testing S3 backends in my projects.

---

So as you can see, there is plenty of possible solutions, and Elixir by default do not force one approach over another, which IMHO is good thing.

[Fl4m3Ph03n1x]

@LostKobrakai

unless you can inject from within the test itself.

That’s the whole point! Your unit tests should be injecting the dependencies, not some global entity. Food for thought regarding this theme:

The Clean Code Talks - Don't Look For Things!

As for mox, the fact that it forces me to have a contract (even when I don’t need one) really throws me off. Mox is fine when you do Mocks, but when you do functional injection like me, it becomes more of a hindrance than help.

@hauleth

You can use for example mockery

I have checked mockery, but the fact that it actually ties my production code with a testing lib really throws me off. my_request is a real function, its module should not have a dependency on a testing framework. It should not even be aware that it is being tested.

If I use mockery, I say that the modules which dependencies I am testing now have a tight coupling to the testing framework (mockery). I am not a fan of that (so picky, right?)

It is also still a viable solution for those that don’t mind the coupling.

Mocking the target service (mocking as a verb, careful! ) is definitely useful, but I see that more into the real of integrated tests (which I defend have a place in test suites as well).

I really liked you Agents solution to replace the counters though!
What are the possible benefits / drawbacks of using it when compared to counters in you opinion?

[LostKobrakai]

@Fl4m3Ph03n1x

That’s the whole point! Your unit tests should be injecting the dependencies, not some global entity.

Then I’m wondering what you’re looking for because the the mock libraries out there are mostly needed because people cannot directly inject dependencies from their tests. For your case I’d go with what you proposed in your other topic:

test "calls given function 10 times" do
      my_pid = self()

      deps = [
        lookup_fn: fn key ->
          send(my_pid, {:dependency, :lookup, [key])
          {:ok, 1}
        end
      ]

      MyApp.do_work(deps)

      # called 10 times
      for _ <- 1..10, do: assert_receive {:dependency, :lookup, _args}
      # ensure it's not called again
      refute_receive {:dependency, :lookup, _args}
end

You could create some helper methods around common assertion patterns, but otherwise I’m really not getting what you’re looking for?

[hauleth]

@Fl4m3Ph03n1x

I have checked mockery, but the fact that it actually ties my production code with a testing lib really throws me off.

Then treat it as DI container (which this project practically is) rather than “testing lib”.

[Fl4m3Ph03n1x]

@LostKobrakai

You could create some helper methods around common assertion patterns, but otherwise I’m really not getting what you’re looking for?

In this specific discussion, I am trying to figure out the most common patterns used by the community to unit test their code. I want to know how you achieve all the necessary verifications a spy or a stub require. Do you also use counters? Do you use ETS tables? How and why?

On a larger scale, I am wondering if there is a library out there like Sinon JS, which already has all this functionalities and that doesn’t force you into a coding style (like Mox does with contracts) or outright forces dependencies on your production code (like mockery).

@hauleth

Then treat it as DI container (which this project practically is) rather than “testing lib”.

A fair point.

[hauleth]

@Fl4m3Ph03n1x

I really liked you Agents solution to replace the counters though!
What are the possible benefits / drawbacks of using it when compared to counters in you opinion?

TBH almost none. Agent could be slightly slower in highly concurrent test case, but if you would see the difference between these two, then you probably should rewrite that test anyway.

[LostKobrakai]

@Fl4m3Ph03n1x

In this specific discussion, I am trying to figure out the most common patterns used by the community to unit test their code. I want to know how you achieve all the necessary verifications a spy or a stub require.

I’m not sure any one person can answer “for the community”. I personally use Mox when I need to exchange behaviour for testing. I’m of the opinion that if two implementations (prod and test are already two) should provide functionality they need a clear contract on what they accept and return.

@Fl4m3Ph03n1x

On a larger scale, I am wondering if there is a library out there like Sinon JS, which already has all this functionalities …

I know of none. There are two tricky things in elixir, which make “changing functionality for tests” hard. No easy way to alter code from the outside at runtime and concurrent tests. The libraries I know of are created on the notion that one of those is the problem. But neither is really a problem if you directly inject the to-be-changed dependency, where you could save on some of the complexities inherent to those existing libraries and the problems they solve.

[axelson]

@Fl4m3Ph03n1x

In this specific discussion, I am trying to figure out the most common patterns used by the community to unit test their code.

we use Mox extensively for our unit testing (although perhaps you disagree that we are unit testing). I tend to agree with @LostKobrakai that making the multiple implementations of a function/module explicit via a behaviour benefits the code by defining an explicit contract.

Also in case you’re not aware with Mox it is very easy for the mocked module to return different values when it is called a second or third time:

test "email sending is handled" do
  MyApp.Test.MockEmailSender
  |> Mox.expect(:send_email, fn _email_text -> {:ok, :sent} end)
  |> Mox.expect(:send_email, fn _email_text -> {:error, :remote_server_offline} end)
  
   # When this calls `MockEmailSender.send_email/1` it will return `{:ok, :sent}`
  MyApp.notify_user("hi")

  # When this calls `MockEmailSender.send_email/1` it will return `{:error, :remote_server_offline}`
  MyApp.notify_user("bye")
end

Also that test above works perfectly with async: true even if you’re running code with Tasks (as long as you’re using Elixir 1.8+).

Sometimes I also stub a Mock to run the actual code. That looks like:

Mox.stub(MyApp.Test.MockEmailSender, :send_email, &MyApp.EmailSender.send_email/1)`

One last thought, I’ve found that using a Mock with an explicit contract lines up very well with the “functional core, imperative shell” school of thought.

[Fl4m3Ph03n1x]

@axelson

we use Mox extensively for our unit testing (although perhaps you disagree that we are unit testing). I tend to agree with @LostKobrakai that making the multiple implementations of a function/module explicit via a behaviour benefits the code by defining an explicit contract.

What an interesting point of view. The issue I have with Mox is that if I use it I will end up having a ton of contracts (in the literal sense that they are just interfaces) when in reality I only have 1 implementation of the contract and I wouldn’t need a contract. Not only that, I also have to Mock the entire thing when I could just be passing the 1 function my code actually needs. What makes it worst, is that to use Mox effectively I would, in theory, need to define a contract for every collaborator my code talks to. If this doesn’t look like insanity, then I don’t know…

To quote the article that introduced Mox to the world:

@http://blog.plataformatec.com.br/2015/10/mocks-and-explicit-contracts/

In fact, passing the dependency as argument is much simpler and should be preferred over relying on configuration files and Application.get_env/3

Which is exactly what I do in my example - I simply pass the dependency as an argument directly as suggested in the article.

@axelson

Also in case you’re not aware with Mox it is very easy for the mocked module to return different values when it is called a second or third time:

Ah, I didn’t get that from the docs. Thanks for pointing it out!

@axelson

One last thought, I’ve found that using a Mock with an explicit contract lines up very well with the “functional core, imperative shell” school of thought.

A school I quite familiar with. However I never managed to implement it, because the way I see it, it requires you to build not 1 app, but many little apps that talk with each other (dare I say components? PragDave is rolling his eyes now ). The problem with this is that even though there are some attempts at making these components work (like PragDave’s components library) they are not well accepted and have some nasty fallbacks as well.

So in the end I don’t see a way to use that paradigm because I can’t find a way to consistently dismantle my big apps into smaller apps and make them communicate in a way that doesn’t require a config file with over 1000 lines that is impossible to maintain and that doesn’t pollute my code with unnecessary GenServers and processes just for the sake of making components.

---

Additional rant on Mox

I get that with Mox, you only create a mock for the boundaries of your system. But then you are not really testing the internals of the system, you are just testing what it spits out into the world - integrated tests. Your system may have 100 modules with their own APIs but unless you are willing to create a contract for every single one of them then you can’t test them using Mox.

Now I defend integrated tests are needed, but I rather have a ton of unit tests for each module then a ton of integrated tests via Mox. And this is where this tool falls short for me - I don’t think it fits how I test things.