So I have a ton of async code and I generally don’t run into this error. Do you have a lot of GenServers? It could be a code smell that you’re not following the guidelines to “Use processes only to model runtime properties, such as mutable state, concurrency and failures, never for code organization” (GenServer — Elixir v1.20.2). Spawning a task will respect the $callers/$ancestors hierarchy, and find the correct task to use, so you could use that.

If in an extreme case you really do need to communicate through a system that needs to have different views of the universe, it’s possible. For example, in one system, I sharded my PubSub topics to be painted with a test_id, (by prepending the test_id) and had a registry for semi-transient gen_statems (which are modeling real world entities that need to be checked in with a periodic timeout) that would only present the test’s view of the universe to whomever needed to access it by having the key be the same test_id. I had to override the gen_statem start_link so that it paints the correct test_id.

Stateful systems can get convoluted and need care, Having gone through that IMO think the way that Ecto’s Sandbox works makes you stop and think for a second, hey, am I doing this in a way that is well-architected? And if you are, the sandbox problem will go away.

We have some spiritually-similar concerns in a few areas in our stack, and I’m not fully content with how we’ve tackled them so far, but it has proven somewhat effective at alleviating our previous test suite flakiness due to the performance differences between our developer hardware and our CI platform. Like you, I didn’t want to make application code changes solely for the benefit of more reliable testing.

Where we’re lucky enough that the dynamic/async work is being done under a DynamicSupervisor, we use a pattern like so, and call it during ExUnit’s setup and on_exit both:

  def wait_for_zero_workers do
    case DynamicSupervisor.which_children(The.Supervisor.Module) do
      [] ->
        :ok

      [_h | _t] ->
        Process.sleep(250)
        wait_for_zero_workers()
    end
  end

Since the on_exit is another process from the test body, this one doesn’t help with the test case yanking the plug on the Ecto Sandbox. We’d have to call this func at the end of the test case body instead. I think we did do that in a few places.

In other place, we rolled up our sleeves and did some honestly-pretty-gross tracing shenanigans, but it cleared up our test race conditions around that area pretty much immediately, and didn’t require app code changes.

  @fsm_event_mfa {Our.Async.StateMachine, :handle_event, 4}
  @recv_timeout 1_000

  defp wait_for_exit do
    pid = wait_for_init()
    assert_receive {:trace, ^pid, :out_exited, 0}, @recv_timeout
  end

  defp wait_for_init do
    assert_receive {:trace, pid, :call, {Our.Async.StateMachine, :handle_event, [:enter, :init, :init, _]}},
                   @recv_timeout

    pid
  end

  defp watch_fsm_progress(_ctx) do
    # Ensure module is preloaded before adding trace
    true = Code.ensure_loaded?(elem(@fsm_event_mfa, 0))

    :erlang.trace_pattern(
      # Watch executions of this MFA
      @fsm_event_mfa,
      [
        {
          # Assign first argument to $1
          # Assign second and third arguments to $2, both must be same value
          [:"$1", :"$2", :"$2", :_],
          # First argument must be :enter, second/third must be :init
          [{:and, {:"=:=", :"$1", :enter}, {:"=:=", :"$2", :init}}],
          # Default behavior - receive a {:trace, pid, :call, mfa} message
          []
        }
      ]
    )

    # Receive OTP messages on function calls and process exits made by new processes
    flags = [:call, :exiting]
    :erlang.trace(:new_processes, true, flags)

    # Clean up added traces at end of test case
    on_exit(fn ->
      :erlang.trace(:new_processes, false, flags)
      :erlang.trace_pattern(@fsm_event_mfa, false)
    end)

    :ok
  end

Then most test case bodies end with wait_for_exit().

For context, these are basically dynamically-spawned gen_statem state machines (not living under a DynamicSupervisor and I don’t recall why) that send and receive messages over an external message bus. For various reasons they are allowed to do more internal work after their final networked reply, so we want to let them fully execute to completion during test suites. That final work might include DB access, so the testing story was a little non-deterministic before we added this. (If you squint a little, parts of that might sound a little like Broadway or perhaps conduit_amqp, and that’s not far off conceptually, except we’re not able to leverage those examples directly due to our historical choice of bus.)

This approach has got a learning curve for our newer Elixir devs, for sure, and I was hip-deep in the Erlang docs myself when I wrote most of it. I definitely view this as a probably-too-clever band-aid that papered over a likely code smell, rather than an optimal design to strive for, but the technique might transfer to others who are interested in this topic.

Or we’re just crazy in good company