volcov
LetItCrash - A testing library for crash recovery and OTP supervision behavior
Hello,
How is everyone?
I hope you’re well =)
I worked on a project with some legacy code that existed before Oban’s arrival. This legacy code functioned as a “job” that was created alongside dynamic supervisors. Understanding the outcome of a supervisor failure and recovery has always been a challenge.
So I thought about creating a set of tools to help me write tests that would simulate these scenarios.
It’s a narrow path between knowing “if I’m testing the Supervisor” (which isn’t the goal) and whether I’m testing the behavior of the result after the Supervisor has done its job.
And from this thought, this lib was born. I still have many ideas for what to add to it. There are a world of scenarios to cover (Workers, DB connections, State), but version 0.1.0 has the general idea I want to facilitate.
I brought it to the forum to see if the idea makes sense =)
I hope it can help more people who are also facing this challenge.
Hex: let_it_crash | Hex
GitHub:
Most Liked
volcov
First of all, I apologize for the delay in responding to your comment, I was buried by a big project and I didn’t come back here.
Thank you so much for taking the time to share your thoughts! I completely understand your perspective, and honestly, I questioned myself many times about whether this library was really necessary or if I was just “testing OTP” unnecessarily.
However, after using let_it_crash actively in our production codebase, I found it incredibly valuable for a real-world problem we were facing. Let me share our experience:
Our Use Case
We had a scoring system with this supervision tree:
ScoreSupervisor
├── DynamicSupervisor (manages calculation workers)
└── ScoreCoordinator (coordinates the flow)
The Problem: Our system would occasionally get stuck during the normalization phase in production, but we couldn’t figure out why.
What let_it_crash Revealed
When we wrote tests simulating crashes during active processing:
test crash during active processing" do
ScoreCoordinator.start_score_calculations() # Start with active workers
LetItCrash.crash!(*ScoreCoordinator*)
assert LetItCrash.recovered?(ScoreCoordinator) # ❌ FAILED!
end
We discovered a real bug: Our supervisor used :one_for_all strategy, which:
- Killed active workers unnecessarily when coordinator crashed
- Created orphaned workers trying to communicate with dead processes
- Led to failed recovery and the system getting stuck
What We Were Actually Testing
You’re absolutely right that we shouldn’t test if OTP works (we know it does!). But we were testing:
1. Our Configuration Choices**
-
Is
:one_for_allor:rest_for_onebetter for our specific architecture? -
Does our coordinator’s
trap_exitwork correctly with shutdown signals?
2. Our Application Logic
-
Does our cleanup code in
terminate/2properly handle active workers? -
Is our
shutting_down?flag preventing new work correctly? -
Are monitor references being cleaned up?
3. Interactions Between Components
-
What happens to workers when coordinator crashes mid-calculation?
-
Does state remain consistent after recovery?
-
Can the system continue processing after recovery?
4. Edge Cases Specific to Our Domain
-
Crash with zero entities
-
Crash with exactly
@max_concurrent_workersentities -
Crash during normalization phase
-
Multiple rapid crashes
Conclusion
I initially had the samequestion you expressed. But when our production system had a mysterious bug that only happened under specific failure conditions, let_it_crash gave us a systematic way to:
-
Reproduce the issue in tests
-
Understand the root cause
-
Validate the fix
-
Ensure it doesn’t regress
I don’t know if I managed to express exactly what this journey was like, let me know if I left more questions than answers haha
Thanks again for the discussion ![]()
Eiji
While usually there is no rule that’s always right and there are almost always edge cases we as a community still try to make conventions clear and simple for everyone, so in some specific cases we know what to expect. It’s not rather about how to life, but something like a standard explaining quality of life.|
Therefore it would be amazing especially for a new developers if you could write the code that is compatible with our naming convention. Here is a direct link to the section with a trailing bang character in the function name: Trailing bang (foo!) | Naming conventions @ Elixir hex documentation
Eiji
I think that the crash! naming may be very confusing …
Unlike
crash/1, this function uses:killinstead of:shutdown, which ensures the process is terminated even if it hasProcess.flag(:trap_exit, true). This is useful for testing processes that trap exits, such as GenServers that need to perform cleanup on normal exits.
I would change it to something like:
@spec crash(type :: :shutdown | :kill, pid())
def crash(type \\ :shutdown, piid)
Popular in Announcing
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









