In my test suite I occasionally find myself needing to work with timeouts if some processing is done asynchronously. In this case I often resort to
:timer.sleep(100)
assert condition
Is there a way to instead specify an assertion that takes a condition and a timeout with the following logic: If the condition holds before the timeout is up then the test passes. Otherwise it fails. If that assertion would not simply implement the above but check regularly, say every ms, then this would allow for
a more robust test suite as the timeout can be put to a much larger value than the above sleep workaround
for a much faster test suite, since the full sleep is usally superfluous.
Implementing this functionality with a macro should be (I guess) straight forward. If so, are there by any chance already any libraries that offer this type of assertion or if not, is the above an anti-pattern that should be avoided?
It is definitely an anti-pattern as you will always 100ms even if the condition you are asserting on would be true after 1ms.
Ideally you want to assert_receive and assert on messages or perform a blocking call. For example, if you are talking to a server that performs async actions, you could have a sync call that simply returns ok. That in itself will guarantee all async messages have been processed.
Agreed, the sleep workaround is an anti pattern. I was thinking more in the lines of the following
defmacro await_assert(assertion, timeout) do
tmp_module_name = "AwaitedAssertion" <> String.replace(to_string(:rand.uniform()), ".", "")
quote do
defmodule :"#{unquote(tmp_module_name)}" do
def test_assertion(t) when t <= 0 do
assert false, "Operation took too long."
end
def test_assertion(t) do
try do
unquote(assertion)
:ok
rescue
_ ->
:timer.sleep(1)
test_assertion(t - 1)
end
end
end
apply(:"#{unquote(tmp_module_name)}", :test_assertion, [unquote(timeout)])
end
end
In that way I would evaluate the condition every millisecond until a timeout is reached after which I fail the assertion. Would you still call this an anti-pattern? An example for a use case would be an asynchronous action that triggers the insertion of a record being inserted into an ets table which I would like to verify.
assert_receive/3 has already been pointed out to you - have a look at it and it’s friends.
“Busy wait” is the anti-pattern - it doesn’t matter if the wait is 100 or 1 ms.
In an efficient system events worth observing need to be broadcast - not “watched”. Interested parties should be able to register (and unregister) their “interest” so that they can be notified when the event occurs. That notification (event message) can then be used with assert_receive/3.
The fact that you need to structure a test in a “busy wait” manner may suggest that you have a design issue.
What is that ETS table used for? Typically they exist to support some sort of request served by the process. If the same process sends the async insert followed by a synchronous request then the request should be served after the insert has been processed - so the result of the request should be consistent with the insert being successful.
I still have to disagree here. assert_receive/3 is great for many use cases but not suitable for everything. Take as example a request that triggers the insertion of data into a cache. I want to assert now that after the request, data eventually is served from the cache. In this case I would not want to add a broadcast mechanism to the cache that informs interested parties about insertions and invalidations a) because it is expensive and unnecessary (except for testing) and b) because that would be a very different test.
This is no different than logging. Write your ETS access functions in such a way that for testing they compile to send a digest message to a named process for testing. When compiling for production that code can then be safely omitted.
That is a nice idea. Thanks. Macros still amaze me. Here is a little code I’ve been playing around with to implement the suggestion:
defmodule TestMessage.Registry do
@moduledoc false
@registry TestMessage.Registry
@default_key :test_message
def name, do: @registry
def default_key, do: @default_key
def init do
{:ok, _} = Registry.start_link(keys: :duplicate, name: @registry, partitions: System.schedulers_online)
end
def subscribe(key \\ @default_key) do
{:ok, _} = Registry.register(@registry, key, [])
end
end
Initialize the test message registry in the test_helper.exs as
TestMessage.Registry.init()
And register for test messages via the subscribe function within test cases. To send test messages use the following module that implements @peerreynders sugestion of checking that the code is compiled for testing
defmodule TestMessage.Sender do
@moduledoc false
alias TestMessage.Sender
alias TestMessage.Registry, as: TestMessageRegistry
defmacro send_test_message(message) do
quote do
Sender.send_test_message(TestMessageRegistry.default_key(), unquote(message))
end
end
defmacro send_test_message(key, message) do
if Mix.env == :test do
quote bind_quoted: [key: key, message: message] do
Registry.dispatch(TestMessageRegistry.name(), key, fn entries ->
for {pid, _} <- entries, do: send(pid, message)
end)
end
end
end
end
Just another idea, following @josevalim rationally and with no external dependencies. Code isn’t readable since Erlang trace API is complicated. But if you abstract the complication away in test helper functions, it can become manageable…
pattern = {Module, :function, 0} # it can be any {mod, fun, arity}
:erlang.trace_pattern(pattern, [{:_, [], [{:return_trace}]}])
:erlang.trace(:all, true, [:call])
# do async work that ends up calling Module.function()
assert_receive {:trace, _pid, :return_from, ^pattern, {:noreply, _state}} # block until Module.function is called
# write the rest of your assertions here, the ones that need `Module.function` to be returned...
