Test function was called exactly X times

Background

I have some code that invokes a given function a certain number of times. I pass this function in the parameters so it is easy to stub.

However I need to know that under some conditions the function was called exactly X times. ExUnit.Case has an extremely limited support for this however, so I went ahead with the “send yourself a given message trick

Code

    @tag :wip
    test "calls given function X times" do
      my_pid = self()

      deps = [
        lookup_fn: fn _key ->
          send(my_pid, :lookup)
          {:ok, 1}
        end
      ]

      MyApp.do_work(deps)
     # how to test I got the :lookup message exactly twice?
    end

Research

The first easy solution is using receive multiple times:

receive do
  msg -> assert msg == :lookup
end

receive do
  msg -> assert msg == :lookup
end

receive do
  msg -> assert msg == :lookup
end

# etc...

For example, if I wanted to check :lookup was called 10 times, I would have 10 receives.
This solution is very poor for 2 reasons:

  1. A ton of code repetition
  2. It doesn’t check that :lookup was called exactly X times, it checks it was received at least X times.

I searched for some libraries but couldn’t find anything. The closes thing I found was the SO question using macros:

Which I tried to adapt into an assert_receive_at_least (but failed miserably):

defmacro assert_receive_at_least(pattern, times, timeout \\ 500) do
    quote do

      defp loop(_pattern, current_times, total_times, _timeout) when current_times == total_times do
        {:ok, :received}
      end
      defp loop(pattern, current_times, total_times, timeout) do
        receive do
          msg -> assert pattern == msg
        after timeout -> {:error, :timeout}
        end
      end

      defp run(pattern, times, timeout) do
        loop(pattern, 0, times, timeout)
      end

      run(unquote(pattern), unquote(times), unquote(timeout))
    end
  end

Questions

  1. How do I check if a given message was received X times?
  2. How do I check if a given message was received at least X times?
  3. Are there any libraries that add decent stub support ?
x = 10
for _ <- 1..x, do: assert_received :lookup
refute_received :lookup
5 Likes

I would encourage you to think about the problem differently if possible - are there existing, concrete side effects you can observe directly after the function has been called N times? Perhaps several new DB rows get created, a counter gets incremented, etc.

If you’ve read Mocks and Explicit Contracts and still want to continue with this path knowing those trade-offs and accepting them, Mox is a great choice. May require some refactoring to establish your contracts as a behaviour.

1 Like

Could you elaborate a little bit more on your solution?
Also, I assume that is the solution to question 1. Would you be kind enough to suggest something about the second question?

This is precisely the things I don’t want to test. I want to test that the messages of my system were sent to its collaborators. I don’t wanna know what they did. This is a hallmark principle of good testing and good design I won’t abandon so easily.

I have read that article. In fact that article is the reason why I am injecting the dependencies via functional injection. First, because not every code benefits from the added noise of a contract, and second because José Valim himself suggests in the article that functional dependency injection is better.

But even if I were to use Mox, it would still fall (very) short of the basic features I need to test, such as meaningful stub and spies.

There is a fairly subtle important distinction between “was this function called N times”, which is a blatant implementation detail, and “was this message delivered N times”, which is still an implementation detail, but one that is likely to survive longer within your business domain than a function name. You can test the latter without any external libraries, depending on how your code is structured. Presumably the pid or registered name of your collaborator appears somewhere within your function arguments, and you can point that to another listener in various ways that would allow you to assert on the messages received.

Can you mention some of the shortcomings you found when trying the library? Have you opened any GitHub issues on the project to have them considered? It supports defining stubs and mocks based on anonymous or captured functions, and it would be fairly trivial to pass from that back into a “real” behavioral module to observe that the functions were called but still test the real behavior too.

1 Like

Maybe https://github.com/appunite/mockery#checking-if-function-was-called would be better for your needs. It works well if you don’t need to check function calls between different processes (disclaimer: I am the author of this library, I can be biased)

2 Likes

It’s already covered.

  • for _ <- 1…x, do: assert_received :lookup this will use assert_receive x times. If x receives do not happen, a timeout will fail the test.
  • refute_received :lookup this ensures that that there are no further receives beyond x for the specified timeout.

So combined the test ensures that exactly x messages were received within the specified timeouts.

3 Likes

If you are using OTP 21+ then you can write it as:

    @tag :wip
    test "calls given function X times" do
      counter = :counters.new(1, [])

      deps = [
        lookup_fn: fn _key ->
          :counters.add(counter, 1, 1)

          {:ok, 1}
        end
      ]

      MyApp.do_work(deps)
      assert 2 == :counters.get(counter, 1)
    end
1 Like

The function name is a detail, you are correct. I don’t care which specific function my code runs (if its name is A or B or whatever) but I do care that it runs the functions I inject on it a given number of times. This is the basics of testing (usually you test if a function was called once) and coulnd’t be farther from an implementation detail.

Allow me to showcase what a real test library is :smiley:

https://sinonjs.org/releases/v7.2.4/

spies, stubs, fakes, mocks, timers, etc.
Admittedly they go out of the way with their own assertions (something which ExUnit could benefit from) but that’s another discussion.

A fair point, but one that I feel would be a waste of my time. I find it that Jose Valim would rather do other things then listening for a cry baby asking for features. Not only that, I need a solution I can use now, not a solution I can use in 4 months when the Elixir team finally has time to update Mox.

Ultimately, even though you are 100% logically correct, I still don’t engage in that course of action because I have a strong believe my efforts would be fruitless.

From the documentation I see a couple of profound limitations:

  1. It still forces you to use contracts and works only via mocks. As I mentioned before, not everything benefits from a redundant interface.
  2. It really doesn’t allow the stub to return a different result on each call.

If I am incorrect, please feel free to en-light me :smiley:

Checking it, thanks for popping in !

Using a global state ETS table on my tests truly is something I frown upon. My main issues with this approach are:

  1. I need a unique counter for each test and I need to remember which counters are already taken
  2. It couples my tests to global state
  3. If I am not careful I may very well loose the ability to run my tests concurrently, as shown in the tutorial about ETS tables.

I will give it to you that your solution does technically work and that it may end up being less complex than the process approach. So it is something to have in mind. Thanks for sharing!

1 Like

No, you do not need to, as you are passing counter, which is unnamed and identified by reference.

Like most mocking libraries, fakes, etc. It just mean that sometimes it is just hidden from you. And if you want, you can always use Agent instead of :counters.

Not exactly, just use unnamed ETS tables, and you will be good to go.

1 Like

So I don’t have to keep track? Quite interesting.

A fair argument. Truth is as long as I can run my tests with async: true I don’t really care.

Like the one :counters create?

You have overall good arguments and have convinced me.

Do you have any links or documentation to :counters.new ?
I check the official erlang docs but was unable to find anything.

Well, probably the erlang docs:

http://erlang.org/doc/man/counters.html#new-2

First hit on google for me :wink:

1 Like

So this is not part of the ETS API, right?
That explains why I coulnd’t find it, I was checking the ETS docs.

The module was :counter, not :ets.

Also :counter is part of the :erts application, while :ets is in :stdlib, which itself does depend on :erts IIRC. So :counter can not use :ets.

Also due to the constraints laid out in the module documentations prelude, I’d guess its using :atomic. Which again was mentioned already in this thread.

2 Likes

Correct me if I’m wrong but I think the approach will only work if MyApp.do_work(deps) executes purely inside the test process - if there are any other processes spawned there could be a potential race condition:

There is no guarantee that by the time we hit assert x == :counters.get(counter, 1)

  • all of the the x calls have already completed
  • there won’t be any extraneous calls for a given time period.

@LostKobrakai’s assert_received/refuted_received approach works for sequential and concurrent code within the given timeouts. So I would view the message based approach as more resilient and less aware of the implementation.


This is the basics of testing (usually you test if a function was called once) and couldn’t be farther from an implementation detail.

Given the sparse details it’s kind of difficult to judge. You seem to be testing that a lookup was performed X times. Lookups can be cached, which could break the test.

So should I be a classicist or a mockist?:

I really like the fact that while writing the test you focus on the result of the behavior, not how it’s done.

From a classicist POV counting function invocations doesn’t seem to be focusing on the result (unless in some strange way it is).

1 Like

Yep, there is no such guarantee. But the same goes for all asynchronous tests, but assert_received and refute_received works on messages that already are there, so they have exactly the same problem, what you would need to use is assert_receive (yeah, this is confusing), but even then the problem will be there. In other words, in truly concurrent environment (like BEAM) being 100% sure that asynchronous requests will not be “magically” repeated is impossible, in NodeJS it is possible as there is no “true concurrency” as main loop is single threaded.

1 Like

I had tunnel vision on assert_receive and refute_receive - even though I typed assert_received and refute_received.

You can skip all the mocks etc and just call the function and then use the trace features built into Erlang to ensure that it was called the correct number of times and with the correct params. Really so much easier

1 Like

Just an observation.

I personally never got into mocking. I only crossed paths with it about 5 years ago when using Mocha/Sinon/Chai. Given the nature of JavaScript (or Ruby for that matter) I can see how one could go down that road.

My exposure to microtesting goes back to the early days JUnit.

Mocking only became a thing some time later and when I looked into it in Java-land, a lot of it relied on reflection - my reaction:

  • Eeewh! That’s cheating.

Well, not entirely. It’s justified if you are mocking 3rd party dependencies but even then there was always the choice to just decouple that behind a façade.

That’s cheating.

That reaction was driven by the notion that mocking your own code removes the incentive to constantly assess and re-evaluate your boundaries. With classical testing there is the constant pressure to adjust your boundaries to be able to test what you want to test.

To me personally the primary benefit of Test Driven Development (in the classical sense) was that constant challenging of those boundaries (and the portals passing through them) - not the test first religion.

Of course even that can go off the rails as evidenced by the whole test induced design damage discussion.

One thing that isn’t clear from this discussion is:

  • why is it so important that the lookup is called 10 times that it needs to be tested at this level?
  • what is responsible for performing that lookup in the real system?

For example, if the lookup goes to another process anyway, wouldn’t it make more sense to test it with a fake process which keeps track of the invocation details that are then sent to the testing process?


Somewhat related in the JavaScript space: Please don’t mock me

4 Likes