Test processes using handle_continue

That’s right. Usually it passes on the second or third attempt locally, but if the system is noisy or it is running in CI there is some breathing room.

This is essentially how assertions work in web testing frameworks like Capybara/Hound.

That’s a good point. I would think nothing of adding a :stop/:shutdown message to the process even if production doesn’t use it. That way the test case could

• start the process
• send the :stop message
• wait for the process to terminate (either via :EXIT or :DOWN)
• then assert the observable actions

I have been dealing with this same issue recently of wanting to wait for handle_contiue to complete before I start testing my server.

I found a the simplest solution was to just put in a call to the server since the message won’t be processed until after the continue has completed. I didn’t want to wrote code in a handle_call just for testing so I used the erlang sys module for this since it provides convince debug functions for working with processes. I found calling :sys.get_state(server_pid) often did the trick. This way I don’t have to use some arbitrary sleep duration to try and guess how long it will take.

Maybe I simply prefer Inside-Out testing in order to control test maintenance costs (giving up some defect localization).

Once one becomes diligent with testing it’s important to find ways to balance the value of testing with the burden it imposes - test obsessed can go too far.

Kent Beck’s (2008) opinion just recently surfaced here again.

There are times where “should I even be testing this”, “should I be testing this particular aspect” or “should I be testing it in this manner” are valid questions.

Sure but I’d be fine with that personally in this particular case which I think of as sort of a last resort. It’s a tradeoff between adding something to a public API just for the tests and knowing a small amount about the internal implementation.

I don’t think I’d use this technique a lot, I’d much prefer to structure the code not to need it or wait for something publicly observable like a registry entry. I do remember a couple of times where I’d have been happy to use this instead of changing the code though.

If you did make changes like you said then the fix shouldn’t be too hard to work out. The code to do this could easily be extracted to a “start_and_wait” function if it was used in multiple tests so there would only be one place to change.

If I was reviewing some code and I saw something like this I would definitely raise a flag though, especially if it was tied to application specific implementation details.

Take an implementation before handle_continue became a thing using Process.send_after. You couldn’t change this to use handle_continue without breaking the test. Or maybe using gen_statem at some point makes more sense, which has different callbacks, you’d also need to change the test.

It’s knows the name of the module, the :handle_continue function, and it’s arity. In my example it knows the return args as well but you don’t need those if you just care about whether the function returned or not. I personally would not rate that as “severely” coupled in this case because :handle_continue is part of OTP not some internal function someone could rename during a refactoring.

Why do you rate it severe?

I just want to mention one big caveat with using tracing in that context: It’ll quite severely couple the test to the implementation.

I have played with this a bit more and come up with a version that only gets a message when handle_continue is returned from:

  test "can know when handle_continue finished" do
    :erlang.trace(:new_processes, true, [:call])

    :erlang.trace_pattern(
      # interested in this module, function and arity
      {Tracetest.Server, :handle_continue, 2},
      # this match pattern doesn't care what the arguments are
      # {:message, false} means don't send the :call message
      # {:return_trace} means do send a :return_from message
      [{:_, [], [{:message, false}, {:return_trace}]}],
      # needed for local calls within modules to work
      [:local]
    )

    {:ok, pid} = Tracetest.Server.start_link(:foo)

    # this is triggered when finished by the :return_trace in the match spec above
    assert_receive {:trace, ^pid, :return_from, {Tracetest.Server, :handle_continue, 2},
                    {:noreply, :some_state}}

    assert true == true
  end

Back at the top you said that

The BEAM provides this through it’s built in tracing facilities. The code I prototyped showed how to get a message in your test process when handle_continue is finished.

This line says “I want to trace function calls and their returns in all new processes (and ports) created from now, please send me a message for each one”. The trace_pattern call in the next line says “Actually I only want a message if it’s a call to or return from handle_continue in a specific module” (:local is needed to make the return tracing work).

When that is setup the SUT process is started. BEAM will then send messages to the test process for calls and returns that match what’s been asked for. Those messages can be waited for and once the :return_to message has been received you know that handle_continue has finished.

If you just waited on the :call message there would still be a race condition as your Logic.establish_connection could still be running when the test process started running the asserts.

HTH. I haven’t actually used this technique though I can think of times that I might have now!