volcov

volcov

LetItCrash - A testing library for crash recovery and OTP supervision behavior

Hello,
How is everyone?
I hope you’re well =)

I worked on a project with some legacy code that existed before Oban’s arrival. This legacy code functioned as a “job” that was created alongside dynamic supervisors. Understanding the outcome of a supervisor failure and recovery has always been a challenge.

So I thought about creating a set of tools to help me write tests that would simulate these scenarios.

It’s a narrow path between knowing “if I’m testing the Supervisor” (which isn’t the goal) and whether I’m testing the behavior of the result after the Supervisor has done its job.

And from this thought, this lib was born. I still have many ideas for what to add to it. There are a world of scenarios to cover (Workers, DB connections, State), but version 0.1.0 has the general idea I want to facilitate.

I brought it to the forum to see if the idea makes sense =)
I hope it can help more people who are also facing this challenge.

Hex: let_it_crash | Hex
GitHub:

https://github.com/volcov/let_it_crash

Most Liked

volcov

volcov

First of all, I apologize for the delay in responding to your comment, I was buried by a big project and I didn’t come back here.

Thank you so much for taking the time to share your thoughts! I completely understand your perspective, and honestly, I questioned myself many times about whether this library was really necessary or if I was just “testing OTP” unnecessarily.

However, after using let_it_crash actively in our production codebase, I found it incredibly valuable for a real-world problem we were facing. Let me share our experience:

Our Use Case

We had a scoring system with this supervision tree:


ScoreSupervisor

├── DynamicSupervisor (manages calculation workers)

└── ScoreCoordinator (coordinates the flow)

The Problem: Our system would occasionally get stuck during the normalization phase in production, but we couldn’t figure out why.

What let_it_crash Revealed

When we wrote tests simulating crashes during active processing:


test crash during active processing" do

ScoreCoordinator.start_score_calculations()  # Start with active workers

LetItCrash.crash!(*ScoreCoordinator*)

  assert LetItCrash.recovered?(ScoreCoordinator)  # ❌ FAILED!

end

We discovered a real bug: Our supervisor used :one_for_all strategy, which:

  1. Killed active workers unnecessarily when coordinator crashed
  2. Created orphaned workers trying to communicate with dead processes
  3. Led to failed recovery and the system getting stuck

What We Were Actually Testing

You’re absolutely right that we shouldn’t test if OTP works (we know it does!). But we were testing:

1. Our Configuration Choices**

  • Is :one_for_all or :rest_for_one better for our specific architecture?

  • Does our coordinator’s trap_exit work correctly with shutdown signals?

2. Our Application Logic

  • Does our cleanup code in terminate/2 properly handle active workers?

  • Is our shutting_down? flag preventing new work correctly?

  • Are monitor references being cleaned up?

3. Interactions Between Components

  • What happens to workers when coordinator crashes mid-calculation?

  • Does state remain consistent after recovery?

  • Can the system continue processing after recovery?

4. Edge Cases Specific to Our Domain

  • Crash with zero entities

  • Crash with exactly @max_concurrent_workers entities

  • Crash during normalization phase

  • Multiple rapid crashes

Conclusion

I initially had the samequestion you expressed. But when our production system had a mysterious bug that only happened under specific failure conditions, let_it_crash gave us a systematic way to:

  1. Reproduce the issue in tests

  2. Understand the root cause

  3. Validate the fix

  4. Ensure it doesn’t regress

I don’t know if I managed to express exactly what this journey was like, let me know if I left more questions than answers haha

Thanks again for the discussion :purple_heart:

Eiji

Eiji

While usually there is no rule that’s always right and there are almost always edge cases we as a community still try to make conventions clear and simple for everyone, so in some specific cases we know what to expect. It’s not rather about how to life, but something like a standard explaining quality of life.|

Therefore it would be amazing especially for a new developers if you could write the code that is compatible with our naming convention. Here is a direct link to the section with a trailing bang character in the function name: Trailing bang (foo!) | Naming conventions @ Elixir hex documentation

Eiji

Eiji

I think that the crash! naming may be very confusing …

Unlike crash/1, this function uses :kill instead of :shutdown, which ensures the process is terminated even if it has Process.flag(:trap_exit, true). This is useful for testing processes that trap exits, such as GenServers that need to perform cleanup on normal exits.

I would change it to something like:

@spec crash(type :: :shutdown | :kill, pid())
def crash(type \\ :shutdown, piid)

Where Next?

Popular in Announcing Top

mischov
import Meeseeks.CSS html = HTTPoison.get!("https://news.ycombinator.com/").body for story <- Meeseeks.all(html, css("tr.athing")) do...
New
jakub-zawislak
Hi everyone, I’m coming from the Symfony (PHP) framework. I like Phoenix, but it has a one thing that was build much better in the Symfo...
New
tmbb
I’ve been working on two packages (not on hex.pm yet) to build admin interfaces for phoenix apps: bureaucrat - which contains a bunch ...
New
oltarasenko
Dear Elixir community, After a year of development, bug fixes, and improvements, we are proudly ready to share the release of Crawly 0.1...
New
nikokozak
Hello all, I’ve been working on Svonix - a library for quickly integrating Svelte components into Phoenix views. It’s a much-needed succ...
New
kip
Image is an image processing library for Elixir. It is based upon the fabulous vix library that provides a libvips wrapper for Elixir. I...
622 18934 194
New
RobertDober
Earmark is a pure-Elixir Markdown converter. It is intended to be used as a library (just call Earmark.as_html), but can also be used as...
239 12645 134
New
Flo0807
Hello everyone! I am excited to share our heart project Backpex with you. After building several Phoenix applications, we realized that...
New
markmark206
simple_feature_flags is a tiny package that lets you turn features on or off based on which environment (e.g. localhost, staging, product...
New
marcuslankenau
I feel kind of stuck with the absence of a proper xml library for Elixir. Currently I use SweetXML which was ok for me more or less to pa...
New

Other popular topics Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement