Test processes using handle_continue

Is the SUT your process?

  • If yes, and these are unit tests, then my answer is - do not start the server at all. Just call functions and provide your own state directly. In that way you do not have the problem with asynchronous testing.
  • If that process isn’t SUT then mock it by starting different process with the same name
  • If these tests are integration tests then sleep in the test setup and then perform all actions.

Nothing prevents you from instead of using GenServer.call(my_server_pid, :message) use MyServer.handle_call(:message, make_ref(), state).

I can’t remember where I saw it now, but I’m sure I saw a post about using the :erlang.trace function to set up tracing on the SUT from the test. i.e. setup tracing for YourModule.handle_continue function and when it’s called tear down the tracing and continue with the assertions.

I can’t remember any more details than that, and I haven’t tried it myself but it seems possible to know the function has been called without having to add an extra message just for testing.

But what do you want to test then? Showing it might help understanding how to do it? If you want to test that a call made after the full initialisation (meaning init + :continue) results in either A or B according to the state set after the :continue executes, then doing a call with a big enough timeout should be (for practical purposes) deterministic (say 10secs timeout). If you want to test that side effects to the process state happen, then probably chaining manual calls to the handle methods should be enough, etc

1 Like

So I had a play around with this:

A simple server:

defmodule Tracetest.Server do
  use GenServer

  def start_link(arg) do
    GenServer.start_link(__MODULE__, [arg])
  end

  @impl true
  def init([arg]) do
    {:ok, :some_state, {:continue, arg}}
  end

  @impl true
  def handle_continue(_, state) do
    {:noreply, state}
  end
end

The test:

 test "can know when handle_continue finished" do
    :erlang.trace(:new, true, [:call, :return_to])
    :erlang.trace_pattern({Tracetest.Server, :handle_continue, 2}, true, [:local])

    {:ok, pid} = Tracetest.Server.start_link(:foo)

    assert_receive {:trace, ^pid, :call,
                    {Tracetest.Server, :handle_continue, [:foo, :some_state]}}

    assert_receive {:trace, ^pid, :return_to, {:gen_server, :try_dispatch, 4}}

    assert true == true
  end

I setup the same tracing calls in the iex console, started the server there, and ran flush to see what trace messages got received. There were only two so it was straightforward to match on them. I am sure your real code is a lot more complicated than this but perhaps just waiting till the method had been called rather than a return would be enough to avoid the race condition.

2 Likes

The last few topics that you have posted suggest to me that you need a new boundary - for example between the worker process and the http client library.

Essentially that puts the http client library in the horrible outside world and the new module becomes the way your application wants to interact with “the service” - i.e. it becomes a contract that is specified by your application.

Once that boundary is established a test double can be slipped in that the worker process uses to complete its flow to produce results (e.g. a response message) that are directly observable by the test case process.

Depending on the necessary sophistication of the test double “test first” may need to take a back seat for a while.

Especially for you I have prepared small library that I want to publish on Hex later:

-module(gen_local).

%% API exports
-export([start/2,
         call/2,
         cast/2,
         send/2]).

-record(state, {state, module}).

%%====================================================================
%% API functions
%%====================================================================

start(Module, Arg) ->
    case Module:init(Arg) of
        {ok, State} -> {ok, #state{state = State, module = Module}};
        {ok, State, {continue, Msg}} ->
            handle_continue(Msg, #state{state = State, module = Module});
        {ok, State, _Timeout} -> {ok, #state{state = State, module = Module}};
        ignore -> ignore;
        {stop, Reason} -> {stopped, Reason}
    end.

call(#state{module = Module, state = State} = S, Msg) ->
    Tag = make_ref(),
    case Module:handle_call(Msg, {self(), Tag}, State) of
        {reply, Reply, NewState} -> {ok, Reply, S#state{state = NewState}};
        {reply, Reply, NewState, {continue, Cont}} ->
            case handle_continue(Cont, S#state{state = NewState}) of
                {ok, NewNewState} -> {ok, Reply, NewNewState};
                Other -> Other
            end;
        {reply, Reply, NewState, _Timeout} -> {ok, Reply, S#state{state = NewState}};
        {noreply, NewState} -> async_reply(Tag, S#state{state = NewState});
        {noreply, NewState, {continue, Cont}} ->
            case handle_continue(Cont, S#state{state = NewState}) of
                {ok, NewNewState} -> async_reply(Tag, S#state{state = NewNewState});
                Other -> Other
            end;
        {noreply, NewState, _Timeout} -> async_reply(Tag, S#state{state = NewState});
        {stop, Reason, NewState} -> {stopped, Reason, NewState};
        {stop, Reason, Reply, NewState} -> {stopped, Reason, Reply, NewState}
    end.

cast(S, Msg) ->
    handle_reply(fake_call(S, handle_cast, Msg), S).

send(S, Msg) ->
    handle_reply(fake_call(S, handle_info, Msg), S).

%%====================================================================
%% Internal functions
%%====================================================================

fake_call(#state{state = State, module = Module}, Callback, Msg) ->
    Module:Callback(Msg, State).

handle_continue(Msg, S) ->
    handle_reply(fake_call(S, handle_continue, Msg), S).

async_reply(Tag, State) ->
    receive
        {Tag, Reply} -> {ok, Reply, State}
    end.

handle_reply({noreply, NewState}, S) ->
    {ok, S#state{state = NewState}};
handle_reply({noreply, NewState, {continue, Msg}}, S) ->
    handle_continue(Msg, S#state{state = NewState});
handle_reply({noreply, NewState, _Timeout}, S) ->
    {ok, S#state{state = NewState}};
handle_reply({stop, Reason, NewState}, _S) ->
    {stopped, Reason, NewState}.

This provide interface similar to the gen_server but functions are run synchronously instead of asynchronously. This should make testing of gen_server a little bit easier (however there still is no timeout, so you cannot test that).

EDIT:

Published on GitHub for now.

1 Like

When working with a lot of asynchronous calls (handle_continue, handle_info, :timer.send_interval, etc) I often make use of a helper to automate testing for results repeatedly. This avoids using fixed Process.sleep/1 calls where the sleep period is statically defined and either too long (which is pointlessly slow) or too short (which causes flickering tests):

def with_backoff(opts \\ [], fun) do
  total = Keyword.get(opts, :total, 50)
  sleep = Keyword.get(opts, :sleep, 10)

  with_backoff(fun, 0, total, sleep)
end

def with_backoff(fun, count, total, sleep) do
  fun.()
rescue
  exception in [ExUnit.AssertionError] ->
    if count < total do
      Process.sleep(sleep)

      with_backoff(fun, count + 1, total, sleep)
    else
      reraise(exception, __STACKTRACE__)
    end
end

I typically use this from higher level integration tests, where it isn’t desirable to call handle_ functions directly.

1 Like

In your production system how do you know that the workers are available to run? Also what type of failure do you get in your tests when the workers aren’t finished initializing? If you’re just doing a handle_call to the workers then I would expect it to work just fine (unless the initialization takes more than 5 seconds which would cause you to exceed the default handle_call timeout)

I feel there is some clarification needed here. The Worker process is a GenServer, that does, some work.
As all GenServers, it’s code is divided into 3 parts:

  1. The public API that clients can call
  2. GenServer calls to the process itself
  3. handle_x calls which do the real work

In the tests I am performing right now, I am testing the public API. Say I have the following code:


defmodule Clint.Worker do
  use GenServer

  defmodule State do
    defstruct conn_pid: nil,
      active_streams: [],
      worker_id: nil,
      opts: %{},
      deps: %{}
  end

  ###############
  # Public API  #
  ###############

  @spec start_link(map) :: Supervisor.on_start
  def start_link(args) do
    deps =
      args
      |> Map.get(:deps, %{})
      |> build_deps()

    worker_id = Map.fetch!(args, :worker_id)
    opts = Map.fetch!(args, :opts)

    Logger.debug("Starting worker #{inspect worker_id }")

    init_state = %State{worker_id: worker_id, opts: opts, deps: deps}
    pname = :bananas
    GenServer.start_link(__MODULE__, init_state, name: pname)
  end

  @spec request(atom, integer, charlist, map) :: {:ok, :received}
  def request(group_name, worker_id, url, injected_deps \\ %{}) do
    deps = build_deps(injected_deps)
    GenServer.cast(:bananas, {:fire, url})
    {:ok, :received}
  end

  @spec build_deps(map) :: map
  defp build_deps(injected_deps), do:
    Map.merge(@default_deps, injected_deps, fn _, a, b -> Map.merge(a, b) end)

  #############
  # Callbacks #
  #############

  @impl GenServer
  def init(state) do
    Process.flag(:trap_exit, true)
    {:ok, state, {:continue, :establish_conn}}
  end

  @impl GenServer
  def handle_continue(:establish_conn, state) do
    with  {:ok, conn_pid} <- Logic.establish_connection do
      new_state = %{state | conn_pid: conn_pid}
      {:noreply, new_state}
    else
      {:error, reason} -> {:stop, reason, state}
    end
  end

  @impl GenServer
  def handle_cast({:fire, url}, state) do
    stream_ref = state.deps.http.get.(state.conn_pid, url)

    new_state = %{state | active_streams: [stream_ref | state.active_streams]}
    {:noreply, new_state}
  end
end

Here the public API (what clients will call) are the start_link and request functions. I am testing the Public API of the worker, the face it shows to the world.

This fits nicely into the “Test the interface, not the implementation” rule of testing, but it is insufficient. For example, using this methodology I can’t test any handle_info calls, unless I create a worker and then send him the exact messages that trigger handle_info (which according to some community members, I should do. I am now trying out this strategy).

Is this Unit testing? Is this integration?
I argue that the Worker is my unit, so I can argue this is unit testing. Some of you will say “You are mixing your state tests with GenServer OTP and therefore this is integration testing”. I wouldn’t say you are wrong, but I will definitely say this is a very thin line, at least for me.


@chrismcg I have never used trace for testing before, and I fail to see how I could apply it here. I need to invest more time into this idea, as it looks really cool!

I am however not sure why I need to wait for the

:erlang.trace(:new, true, [:call, :return_to])

call.


I usually (try to) make my questions and topics as small and isolated as possible, to make the load on the people reading smaller. Sometimes, this comes at the cost of clarity, which I believe is the case here.

My worker does not depend on any HTTP client. It depends on an HTTP contract, which can be implemented by any client I wish. The boundary is well defined, the way I see it :stuck_out_tongue:

Perhaps, the trouble here is in making it clear it is a boundary. Perhaps you believe I am testing too much the details of my worker which leads you to think I have no boundaries defined. You are one of the most well educated people in testing I have seen and I find it curious how your opinions differ from mine in so many areas. All I can say to defend myself is that I am following a traditional London style TDD, while injecting the dependencies directly without creating Mock modules, because I believe functional injection is the best way of passing dependencies.

Test first is not always the solution, true. This is actually the refactor of a project that I made and that I consider is a complete disaster. No better time to fix it than now :smiley:

In fact, one of the many problem is that this project has not tests at all :rofl:


Oh my God. Thank you so much. I don;t want to sound … ungrateful, but I am not well versed in erlang yet, so I have trouble understanding what this actually accomplishes. I can only hope this comes with great documentation for dummies like me :smiley:


@sorentwo So you let the tests fail repeatedly until you hit a timeout that is slow enough? I understand you provide a maximum number of tries for the test to run, correct (50 times) ?

It simply provides module that you can use instead of GenServer.call/2 to make “calls” synchronous, ex. if you have:

{:ok, pid} = GenServer.start_link(Clint.Worker, args)

GenServer.cast(pid, {:fire, url})

You can replace it with:

{:ok, pidish} = :gen_local.start_link(Clint.Worker, args)

{:ok, pidish} = :gen_local.cast(pidish, {:fire, url}

To run whole thing synchronously. Unfortunately there is no support for named processes as this would make state handling pretty hard and slightly “automagical”. Of course this approach do not suite all gen_server use cases as for example it will not work with active gen_tcp/gen_udp connections (naturally).

1 Like

Back at the top you said that

The BEAM provides this through it’s built in tracing facilities. The code I prototyped showed how to get a message in your test process when handle_continue is finished.

This line says “I want to trace function calls and their returns in all new processes (and ports) created from now, please send me a message for each one”. The trace_pattern call in the next line says “Actually I only want a message if it’s a call to or return from handle_continue in a specific module” (:local is needed to make the return tracing work).

When that is setup the SUT process is started. BEAM will then send messages to the test process for calls and returns that match what’s been asked for. Those messages can be waited for and once the :return_to message has been received you know that handle_continue has finished.

If you just waited on the :call message there would still be a race condition as your Logic.establish_connection could still be running when the test process started running the asserts.

HTH. I haven’t actually used this technique though I can think of times that I might have now!

1 Like

I have played with this a bit more and come up with a version that only gets a message when handle_continue is returned from:

  test "can know when handle_continue finished" do
    :erlang.trace(:new_processes, true, [:call])

    :erlang.trace_pattern(
      # interested in this module, function and arity
      {Tracetest.Server, :handle_continue, 2},
      # this match pattern doesn't care what the arguments are
      # {:message, false} means don't send the :call message
      # {:return_trace} means do send a :return_from message
      [{:_, [], [{:message, false}, {:return_trace}]}],
      # needed for local calls within modules to work
      [:local]
    )

    {:ok, pid} = Tracetest.Server.start_link(:foo)

    # this is triggered when finished by the :return_trace in the match spec above
    assert_receive {:trace, ^pid, :return_from, {Tracetest.Server, :handle_continue, 2},
                    {:noreply, :some_state}}

    assert true == true
  end

I just want to mention one big caveat with using tracing in that context: It’ll quite severely couple the test to the implementation.

4 Likes

It’s knows the name of the module, the :handle_continue function, and it’s arity. In my example it knows the return args as well but you don’t need those if you just care about whether the function returned or not. I personally would not rate that as “severely” coupled in this case because :handle_continue is part of OTP not some internal function someone could rename during a refactoring.

Why do you rate it severe?

1 Like

Take an implementation before handle_continue became a thing using Process.send_after. You couldn’t change this to use handle_continue without breaking the test. Or maybe using gen_statem at some point makes more sense, which has different callbacks, you’d also need to change the test.

1 Like

Sure but I’d be fine with that personally in this particular case which I think of as sort of a last resort. It’s a tradeoff between adding something to a public API just for the tests and knowing a small amount about the internal implementation.

I don’t think I’d use this technique a lot, I’d much prefer to structure the code not to need it or wait for something publicly observable like a registry entry. I do remember a couple of times where I’d have been happy to use this instead of changing the code though.

If you did make changes like you said then the fix shouldn’t be too hard to work out. The code to do this could easily be extracted to a “start_and_wait” function if it was used in multiple tests so there would only be one place to change.

If I was reviewing some code and I saw something like this I would definitely raise a flag though, especially if it was tied to application specific implementation details.

1 Like

Maybe I simply prefer Inside-Out testing in order to control test maintenance costs (giving up some defect localization).

Once one becomes diligent with testing it’s important to find ways to balance the value of testing with the burden it imposes - test obsessed can go too far.

Kent Beck’s (2008) opinion just recently surfaced here again.

There are times where “should I even be testing this”, “should I be testing this particular aspect” or “should I be testing it in this manner” are valid questions.

1 Like

I have been dealing with this same issue recently of wanting to wait for handle_contiue to complete before I start testing my server.

I found a the simplest solution was to just put in a call to the server since the message won’t be processed until after the continue has completed. I didn’t want to wrote code in a handle_call just for testing so I used the erlang sys module for this since it provides convince debug functions for working with processes. I found calling :sys.get_state(server_pid) often did the trick. This way I don’t have to use some arbitrary sleep duration to try and guess how long it will take.

5 Likes

That’s a good point. I would think nothing of adding a :stop/:shutdown message to the process even if production doesn’t use it. That way the test case could

  • start the process
  • send the :stop message
  • wait for the process to terminate (either via :EXIT or :DOWN)
  • then assert the observable actions
1 Like

That’s right. Usually it passes on the second or third attempt locally, but if the system is noisy or it is running in CI there is some breathing room.

This is essentially how assertions work in web testing frameworks like Capybara/Hound.

1 Like