ExUnit testing with registry?

Questions / Help
1

Most of the processes in our system wait for a request, do some processing, possibly a state update, and then send out more requests - most of the time not to the requestor but to another entity. The system is more like a middle man between two entities, terminating protocols on both sides.

The processes are registered with :gproc, so I had the glorious idea that in unit tests we can simply register the test case itself as the recipient for a given name or property and wait if we receive the expected messages once we stimulate our process or not. On timeout, we fail instead. It looks like this:

test "foo" do
  {:ok, pid} = bar.start
  Process.monitor(pid)

  :gproc.reg({:n, :l, :actual_recipient})

  send(pid, :example_message)

  result = receive do
    :example_reply -> :pass
    after 10 -> :fail
  end

  assert result == :pass

end

This was all fine and dandy… but not when you run tests in parallel. Several might register for the same name in :gproc and so this approach to testing works with parallel testing set to off.

How do you test such processes?

Can maybe each ExUnit.Case be run in its own node to remove the problem (with async: false) while being able to run suites in parallel?

1 Like
2

Would it be helpful to move the :gproc lookup outside of the process and instead pass a PID on startup? It would make the process or supervisor that starts your process responsible for the lookup, but would allow for parallel tests on one node.

test "foo" do
  {:ok, pid} = bar.start(self)
  Process.monitor(pid)

  send(pid, :example_message)

  result = receive do
    :example_reply -> :pass
    after 10 -> :fail
  end

  assert result == :pass
end

Also, there’s an awesome ExUnit macro for testing messages in a case just like this, so we could simplify the test case to:

test "foo" do
  {:ok, pid} = bar.start(self)
  Process.monitor(pid) # not sure if this is necessary

  send(pid, :example_message)

  assert_receive :example_reply, 10
end
2 Likes
3

The thing I have done is have a start option to give the name to register the process under. I can default to the main production name, but then in my test, I give it a unique name. That way you can have multiple instances of the process in memory at the same time because they won’t fight over the registered name.

4 Likes
4

Ah! Thank you both. These are excellent suggestions. :slight_smile:

I will put a combination of these two suggestions into practice.

3 Likes
5

Wait a moment. I wasn’t thinking straight last night! :sweat_smile:

This actually solves a problem, but not the one I was trying to solve.

Doing it like this avoids name conflicts caused by the process under test, but what I wanted to do was register under the name of another process and there might be multiple such registrations.

Part of the problem is that during the execution of the test scenario the process might send messages to several other recipients. Giving it the names of all of its friends seems impractical, especially since some of them are computed by functions… Also, if these were handed over, readability would suffer.

I often rely on this pattern:

A process implementing module A wants to call a process implementing module B. So I provide a public function in module B to get the name of B (especially if there are several instances distinct by a numeric id) and also one to get its pid and one to send it messages.

Callee B:

defmodule B do
  def name(id), do: {:n, :l, {:someWorkedId, id}}
  def pid(id), do: :gproc.whereis_name(name(pid))
  def send(id, event), do: :gproc_send(name(id), event)
  ...
end

defmodule A do

def foo(event) do
   ... some processing ... 

  B.send(42, another_event)
end

end

This way it is always clear who I am calling to (I use more telling names :wink:) and keeps the code nice and clean. Problems arise on testing, though.

Either there is a way to rewire existing processes for testing only without restructuring them for testing purposes only, or I guess I need to let individual test suites run on different nodes so they don’t pollute the common “namespace.”

I mean, this problem must arise all the time when working with a registry, is there a pattern for it?

1 Like
6

So in your example, A is notifying B. In the test do you want A to notify some TestB instead of or as well as B?

1 Like
7

“Instead of” would be the desired case.

1 Like
8

Yeah, in that case something along the lines of what @gregvaughn suggested would be the way I’d consider first. In other words, I would try to start an “instance” of A such that it works with the current test process. Meaning some special and unique id would need to be provided to A.

Another alternative is to make each test process register itself as the B (which is presumably what you’re trying to do right now). In this case, you somehow need to ensure (perhaps through config) that real B is not started (so there’s no clash), and make all such tests sequential, so they don’t clash.

Does that make sense?

1 Like
9

Kind of not, no. :sweat_smile:

What the process under test registers under does not solve my problem.

I need to solve that I need to register as its “interface partners” but at the same time have no two testcases running in parallel that register the same ID. The real B is not the problem. The problem is that two testcases can pose as B at the same time in different parallel-running tests.

In the end, pure sequential testing avoids all these problems. It’s just slow in the long run. Especially since we expect to have 1,000s of these tests if we go production with this code.

1 Like
10

I’m assuming every test starts its own instance of A, is that right? If that’s the case, then the test could pass some unique id to its own A which would make sure that this A talks only to this test process, and there are no name collisions.

If A has to be a singleton, then that approach will not work. In such case, you can consider somehow passing the information about who is to be notified when issuing the original request to A. Since there is some request which lands to A and makes it notify B, then under test you could add some optional info to the request which would make sure that A notifies only a particular test process.

Another option is to make A notify whoever is interested (so multiple dynamic subscribers). You could use gproc properties for that. Basically, instead of :n in the gproc triplet, use :p. This allows many processes to register the same property, and they can all be notified as a group.

3 Likes
11

This is basically a global variable problem. The registered name is a global var that works fine in production as designed, but causes a problem in tests because you need to modify it, and that requires a mutex^H sequential tests.

You touched on the solution I’ve used, but quickly dismissed it. What I did is pass in the registered names of the collaborators during start_link. I hesitate to link to this code (because it’s 2 years old and I’m not sure if it’ll still run in the latest elixir), but it may give you a better idea:

Once I did this, I noticed that I could keep pushing that configuration further and further up the stack. I ultimately pushed it into config files which could basically specify the shape of the supervision tree. Maybe that’s too far to be practical in most cases – it was a learning exercise for me. However, I like the encapsulation it gave as well as reuse via configurability.

Ultimately, I think there’s 3 possible outcomes here. Suffer slow sequential tests, make testing more complex by distributing it, or change the design of your code to specify collaborators and global names in a centralized way that can be overridden in tests. I’d love to hear another approach though.

Last minute edit: Oh wait! Sasa has a 4th approach: make all your point-to-point communications become pub-sub.

3 Likes
12

Hmmm. Still does not solve my problem.

What I don’t want: To inject the names of collaborators into the startup of a process.

Why?

  1. It leads into injection hell. For no other reason than testing it injects names into a process. This makes for ugly tests like in (driven to the extreme) GoogleMock, where one has to inject everything just for the sake of testing. For context: Each external class is “injected” when using GoogleMock, leading to injection signatures only required for testing the system. The flexibility usually has absolutely no practical use as in almost all cases it is only used for injecting mocks.
  2. It becomes quickly unwieldy when testing a service that has several interface partners. That is kind of implied by 1), and even if elixir encourages cutting code into smaller processes, testing a process with five partners should not become unwieldy.
  3. It is not only about names and :gproc. Any external dependency shares the same problem, and the problem becomes much worse when referencing a DB table in Mnesia. Because then we would need to inject names for alternate DB tables and so forth.

I personally find, and I fully qualify this as my opinion, unit testing rather useless. It is like repeating my own assumptions out to myself.

This does not mean I reject testing, but that I find scenario testing much more useful. But in order to test a process fully considering scenarios, I have to recreate its environment. And so far (and this could be clearly a limit in my understanding), I find support for that in ExUnit a bit rudimentary.

ExUnit is very clever, though. Each test, and each executor of a test, are spawned in their own processes, clearly showcasing the power of lightweight processes and message passing in elixir. Reviewing that code in the 1.3.4 code base I found that his marvelous piece of code spawns all testcases in separate clean little processes. Tests are loaded from a server. Test results are sent back to a server. It is a beautiful networked application.

What I am looking for is the final step: To run each of these tests in a slave node. There are code examples out there for loading and running code in a slave node very easily. Each slave node would exemplify the perfect test environment: no clashes with other resources in the environment and the ability to set up your own DB. In fact, one could even start a gproc in distributed mode over there but not connect it to any other node. This way all the addressing facilities could be used freely, also the DB, without affecting any other test. For the price of instantiating a node and the price of code load and starting Mnesia one could run all tests in complete isolation from each other without having to change the actual code in a single way.

In my opinion, this would leverage the power of elixir the most - to have a scenario-based framework where I can start one service/process/cluster of processes in isolation and test it with my own fixture without interfering with any other test. The added cost would be worth it, I think. In other words, I want to offload these details onto the framework or a wrapper, not on my actual code. The code is not the problem, the testing of the code is. An actual scenario-testing tool would simply start the code in isolation (our current tool does) and only run into problems with operating system shared resources (like IP ports). And the same I want to do from ExUnit so that I can test to a finer granularity than our outside test system does.

I still have the same problem. But now I have a hunch as how to solve it. In principle, I could even modify ExUnit to do the job instead of me… or put some library code in the setup. I have to understand some details about the communication between ExUnit.Runner and ExUnit.Server first, though.

13

You might find the Phoenix Pubsub tests interesting as an example - they spawn slaves nodes to test in distributed mode.

14

Hello, dom.

Thank you very much. :slight_smile:

I guess I should not be surprised to find stavro as contributor on this one since he also published a slave demo in GitHub. His code and this code gave me the following idea:

Do you think it can work?

Anyways, thank you all for the replies. I know I have been tedious and very specific in my request.

15

@Oliver did you ever find a solution for this?

16

Hi, @anthonator.

My elixir skills were insufficient at the time plus my project got cancelled. So, unfortunately no.

My new project so far has not run into the problem as our testing requirements are not as strict: It’s a test system.

I’m not likely to revisit the problem soon. (I also don’t mind injecting part of the code so much now come to think of it… :sweat_smile:)

1 Like