Deterministic simulation testing for OTP apps?

I’ve been fixing race condition / crash bugs in an open source org, for a few months now and almost all of them had the same problem. They only happen with a very specific order of events (process crashes mid-message, a reply arrives late, a timer fires at the wrong moment) so the normal ExUnit tests never catch them. We find them in prod, then I spend a day trying to reproduce it and 20 minutes fixing it.

So I started reading about how FoundationDB and TigerBeetle deal with this. They do “deterministic simulation testing”: run the whole system inside a simulator that controls time, message order, randomness and crashes, run it thousands of times with different seeds, and when something breaks you get a seed that replays the exact same failure every time.

I couldn’t find anything like this for Elixir and I’m thinking of building it. Very rough idea of what a test could look like:

simtest "cache stays consistent when workers crash", runs: 1_000 do
  start_supervised!(MyApp.CacheSupervisor)

  Sim.spawn_clients(10, fn -> MyApp.Cache.put(random_key(), random_value()) end)

  Sim.inject(:crash, MyApp.Cache.Worker, probability: 0.05)
  Sim.inject(:delay_messages, max: :timer.seconds(2))
  Sim.advance(:timer.minutes(10))

  assert MyApp.Cache.consistent?()
end

and when it fails you’d get the seed plus the smallest set of events that still breaks it, something like:

failed on run 412 (seed 8812734019)
shrunk to:
  1. worker 2 receives {:put, "a", 1}
  2. worker 2 crashes before replying
  3. supervisor restarts it, client retries the put

My plan for how it works: in test env, rewrite modules at compile time so send/receive, timers, monotonic_time and :rand go through a fake scheduler instead of the real one. Concuerror does something kind of similar for Erlang but it’s aimed at exhaustively exploring interleavings, not seeded fault injection with virtual time, and using it from Elixir is painful (at least it was for me).

Other stuff I looked at: gen_server_virtual_time and Klotho (both nice but only handle time), and StreamData (seeds, but for data not scheduling). If something already does all of this please tell me before I sink weeks into it lol.

Things I’m not sure about:

• is compile time rewriting going to be too fragile? the alternative is making people use a special GenServer-ish behaviour, which feels worse to me but maybe it’s more realistic
• I’m planning single node first and network partitions / multi node later. does that order make sense?
• what would you need to see before trusting something like this? my idea is to run it against some well known libs and see if it finds real bugs

Also curious if you’ve had bugs like this in prod and how you ended up reproducing them.

If there’s a reason this can’t work on the BEAM I’d rather hear it now .