Mox and Protocols?

Questions / Help
1

Hi,

I’m using Mox and I defined a Shortlist behaviour like this:

defmodule Shortlist do
  @type result :: {:ok, Item.t} | {:error, reason}
  @callback save(Item.t) :: result
end

As it is now, if I want to implement it with an Agent, that gives me 2 options:

  • the implementation would rely on a named Agent process so it won’t require the pid of the Agent in the function signature. Of course, I’ll have to do a bit of test setup to start/stop the process around each test and I won’t be able to run them async
  • or, if I change the function signature, I can receive an extra pid param, allowing tests to run against distinct agents in order to run concurrently

Yet, that is narrowing the possible implementations to only process based implementations. Instead, if I want to rely on an ETS table, or Ecto.Repo, or a remote http service, or…, I don’t believe this behaviour currently allows that.
I think that changing it to a Protocol would give me total freedom… Example:

defprotocol Shortlist do
  def save(shortlist, item)
end

defmodule InMemoryShortlist do
  defstruct [:agent_pid]
  
  defimpl Shortlist do
    def save(in_memory_shortlist, item) do
      # send message  with item in its payload to in_memory_shortlist.agent_pid
    end
  end
end

Great, but this comes at the cost of losing Mox to write expectations. Sure, I can make each test provide its own implementation of the protocol (hopefully some will be reused) and use assert_received. That will be quite a ceremony though.

Have I overlooked something?

2 Likes
2

If you’re just worried about testing the Agent: You could have functions accepting an optional pid, which are not part of the behaviour.

3

I’m worried that my high level code (in the often called functional core) will end up knowing too much about how a shortlist is implemented if I modify the behaviour signature to introduce a PID. Like I said, that would also limit extensibility.

4

Yes, protocols in this case would be functionally equivalent to Mox. They are both an interface wth a dispatch mechanism behind the scenes.

5

Glad you agree @josevalim !

However, when many tests need a variant of the same mock, defining yet another implementation of the protocol is not going to scale conveniently. Example:

defprotocol Shortlist do
  def put_in(shortlist, item)
end

test "item not already shortlisted" do
  defmodule AgentShortlist do
    defstruct [:agent_pid]

    defimpl Shortlist do
      def put_in(shortlist, item) do
        # send item to the agent and succeed
      end
    end
  end
  Shortlist.put_in(%AgentShortlist{...}, %Thing{...})
end

test "item already shortlisted" do
  defmodule AlreadyShortlistedAgentShortlist do
    defstruct [:agent_pid]

    defimpl Shortlist do
      def put_in(shortlist, item) do
        # send item to the agent and fail because it's already shortlisted
      end
    end
  end
  Shortlist.put_in(%AlreadyShortlistedAgentShortlist{...}, %Thing{...})
end

# and on and on...

With Mox I can create a mock on the fly within each test, ideally It would accept mocking protocols too?

6

Yes, precisely. But protocols do not know how to invoke Mox unless you create a data structure that wraps mox and implement a protocol for it that does the dispatching.

7

@josevalim I read this a couple of times and tried to illustrate your suggestion but failed. Protocol + Mox means I would have to define the Protocol and a behaviour? I’m confused.

test "already shortlisted item" do
  shortlist = %AgentBasedShortlist{...} # how do I make this one pretend that the item is already shortlisted?
  item = %Thing{...}
  assert {:error, :already_shortlisted} == Shortlist.put_in(shortlist, item)
end

defprotocol Shortlist do
  def put_in(shortlist, item)
end

defmodule AgentBasedShorlist do
  defstruct [:agent_pid] # a mox wrapper in there?

  defimpl Shortlist do
    def put_in(shortlist, item) do
      # ???
    end
  end
end
8

Yes, unfortunately. Mox does not know anything about protocols.

9

Alright, thanks for your help.

10

so it won’t require the pid of the Agent in the function signature.

This approach achieves “hiding” through a “hidden, implied, global name”.

In functional programming the preferred approach to “information hiding” is an opaque datatype. There even is a spec for that: @opaque.

The idea being that the client is handling something where the internals of which are entirely unknown - e.g. it wouldn’t know if it is handling a data structure or a PID. But that also has an impact on the shape of the API - you would need:

  • {:ok, new_shortlist} = Shortlist.save(old_shortlist, item)
  • {:error, reason} = Shortlist.save(old_shortlist, item)

new_shortlist seems redundant for a PID based implementation but is a necessary concession for the possibility of a data structure based implementation.

For an example of this approach see

Splitting APIs, Servers, and Implementations in Elixir

Now I may be entirely out of line but from the discussion it isn’t entirely clear to me for what kind of testing you require these mocks for. I’m not saying that mocks aren’t valuable but sometimes they can get out of hand. From that perspective I recommend some of Sandi Metz’s material like:

to ensure that all tests (and mocks) contribute more value than they cost.

3 Likes
11

Hi @peerreynders,

Thanks for your input, let’s recap.

I consider the business logic high level and database/files/processes/… low level details, as per the dependency inversion principle. Therefore, assuming a Logic module and a Storage module, Logic should not depend on Storage but Storage should depend on Logic.

This can be done with a Behaviour declaring the minimal contract (functions and args) required by Logic to operate.

It turns out that a stateful implementation of Storage will require an extra argument (a pid for example) to talk to the right storage. Doing so, Logic would receive and pass around a concrete state handle (the pid for example) thereby coupling itself to “implementation details”. That would also limit extensibility only to process based storages.

Protocols allow to hide the concrete state handle (filename, repo, pid, ets table, …) inside a custom struct. Then Logic just receives and pass around such a struct. It may or may not be stateful, it may or may not be a process, it doesn’t care, and yet it works.

This is a extensible decoupled solution: awesome! The only annoyance is that I can’t use Mox anymore to create custom mocks on the fly. To achieve the same would require lots of ceremony, I think, by defining many implementations of the protocol.

Back to your question:

Now I may be entirely out of line but from the discussion it isn’t entirely clear to me for what kind of testing you require these mocks for.

As I’m TDDing this code, I am writing collaboration tests for Logic. They ensure that Logic uses Storage appropriately: correct function call, correct function arguments. To do so, I need to create storage mocks and expect specific function calls with specific values on these mocks.

Does this answer your question?

2 Likes
12

I’m not sure this is an inescapable conclusion.

Example:

defmodule TupleStorage do

  def init(),
    do: {:value, nil} # the opaque "handle"

  def put({:value, _}, value),
    do: {:value, value}

  def peek({:value, value}),
    do: value

end

defmodule AgentStorage do
  def init() do
    {:ok, pid} = Agent.start(fn -> nil end)
    pid             # the opaque "handle"
  end

  def put(pid, value) do
    Agent.update(pid, fn _ -> value end)
    pid
  end

  def peek(pid),
    do: Agent.get(pid, fn value -> value end)

end

defmodule Demo do
  def demo(storage, handle) do
    handle = apply(storage, :put, [handle, :something])
    apply(storage, :peek, [handle])
  end
end

handle = TupleStorage.init()
IO.inspect(Demo.demo(TupleStorage, handle))

handle = AgentStorage.init()
IO.inspect(Demo.demo(AgentStorage, handle))

$ elixir demo.exs
:something
:something
$
  • TupleStorage stores the state in an immutable data structure.
  • AgentStorage stores the state inside a mutable agent (process).
  • Demo.demo can use either by using the common contract. It knows which function names and arguments to use while at the same time simply using the specified handle while at the same time not knowing or caring what handle actually is.
  • Behaviours simply impose these contract constraints during compile time.

So I argue that it’s the shape of your API that is revealing the “implementation details” by accepting a handle but not returning (a potentially new) one - not the data type of handle (which the “client” ultimately doesn’t care about thanks to Elixir being a dynamically typed langauge).

Which leads to the question if there is a way to still use behaviours (rather than protocols) by rethinking the shape of the API.

1 Like
13

Hi @peerreynders,

It seems that you solved my problem and identified the false dichotomy I felt I was making initially.
Also, you are right about the wrong type of the return value of Shortlist.put_in. Thank you.

Meanwhile, I kept experimenting with mocking with protocols (after all, the idea was suggested by @josevalim in his famous blog post “Mocks and explicit contracts”. I did find some degree of happiness with the following solution. I defined an extra property in the struct which implement the Storage protocol in tests. This enables individual tests to specify the desired return value of the protocol functions.

Example:

# PRODUCTION CODE

defprotocol Shortlist do
  def put_in(shortlist, item)
end

defmodule Logic do 
  def submit_item(item, config) do
    # ...
    Shortlist.put_in(config.shortlist, item)
    # ...
  end
end

# TESTS

defmodule Shortlist.Mock do
  defstruct [:put_in_result] # canned result specified by each test

  defimpl Shortlist do
    def put_in(shortlist_mock, item) do
      # mocking is about expectations, so this line enables message-based-expectations, see the test
      send self(), {:put_in, shortlist_mock, item}
      shortlist_mock.put_in_result
    end
  end
end

test "item not already listed" do
  config = %{
    shortlist: %Shortlist.Mock{
      # test specific canned result
      put_in_result: {:ok, %Shortlist.Mock{}
    }
  }

  an_item = ...
  Logic.submit(an_item, config)

  assert_received {:put_in, ^config.shortlist, ^an_item}
end

This solution is:

  • :+1: polymorphic
  • :+1: mockable
  • :+1: safely testable with async: true
  • :+1: it’s fast, there isn’t even a Registry process like I believe there is under the hood in Mox
  • :+1: I don’t have to wire modules (behaviour implementation and client) with config.exs, this feels very good, no need for Application.get_env(:app, :behaviour_name) scattered in the codebase, I can just pass a config (Map|Struct) argument around
  • :+1: unlike @peerreynders’s solution, every implementation details is stuffed inside a single argument by leveraging structs/protocols

This is for the “degree of happiness” mentioned above.

Now I’m not totally happy with this solution. It feels more verbose / require a bit of ceremony:

  • I don’t have an assistant like Mox:
    • :neutral_face: I don’t have a fancy equivalent of Mox.expect(module, fun_name, fn ... end) but I can make a private helper inside the test module to make this concise, explicit and readable
    • :neutral_face: I don’t have automatic verifications a-la verify_on_exit! so I have to manually verify with assert_received
  • :neutral_face: I have to define a test protocol implementation for each protocol to mock, instead of the simpler defmock MockModule, for: Behaviour, this is not as bad as it may sound though, see code above.
  • :-1::-1: I now have a mix of behaviours and protocols. My tests end up with a bit of Protocol mocking and a bit of Behaviour mocking with Mox. For example: ItemValidator is just an algorithm so I should make it a behaviour and use Mox, while Shortlist is a protocol. I thought I could make ItemValidator a protocol too, leading me to no behaviours, ultimately leading to ditching Mox. The issue I found immediately is that It forced me to create a Validator struct with no properties, because it’s just an algorithm. That’s a signal that I’m certainly pushing too hard in this direction.

This last sentence reveals my overall impression with this entire thing. I am really tempted to surrender and follow @peerreynders’s approach. Though, in theory Protocols are right-er in this context. Ditching them in favor of the more convenient behaviours doesn’t feel right either. Convenience doesn’t feel like the right reason to use behaviours instead of protocols, and if that’s the main reason then maybe a bit more of tooling a-la-Mox would change the deal?

Thoughts @peerreynders & @josevalim ?

14

I believe you should pick protocols or behaviours depending on which one is the best solution to the problem at hand, without taking tests into account, since any mechanism you pick will support writing proper tests anyway.

If you have a protocol, then you don’t need Mox. In the same way that if you need to depend and invoke some third-party code, but you can express this dependency as an anonymous function, then you don’t need Mox either.

To be fair, I use an approach similar to yours, which is to define custom modules and/or protocols, in 99% of the cases. I would only use Mox if I need to define mocks that act globally.

3 Likes
15

Convenient is great when it happens but should never beat Correct. Correct may be more work right now but “Not-correct” has a habit of creating significant problems down the road.

16

Indeed, wise words.