Rich_Morin

Rich_Morin

I’ve seen a couple of references to the idea of “let it crash” and how it can recover from incorrect software. For example, in The do’s and don’ts of error handling, Joe Armstrong said:

Assume software is incorrect and will fail at run time
then do something about it at run-time

However, I think the real story may be a bit more nuanced than this. Let’s say that I run the same software, with the same internal state, on the same input data. I’m pretty sure that it will malfunction again in just the same manner. Fortunately, there are reasons why this might not be the case.

If a process has been building up “bad state” over some period of time, restarting it will clear things up, allowing the new copy to run (at least for a while…). This is a lot like power-cycling a computer, network router, etc… In addition, if the problem is related to a timing issue (e.g., a race condition), restarting the process(es) involved will typically resolve the issue.

Even in the absence of these issues, killing a faulty process and starting up a different version could allow things to keep going. Joe speaks about this possibility in his dissertation:

If an error is detected when trying to achieve a goal, we make an
attempt to correct the error. If we cannot correct the error we immediately abort the current task and start performing a simpler task.

All of this makes me wonder how many production systems have “fallback” tasks of this sort. I know that I’ve never written any, but others may well have done so. Can anyone cite (and describe) some examples?

-r

Showing Posts 1 to 2

LostKobrakai

LostKobrakai

One example that comes to my mind is with caching, where you try to keep a cache up to date, but if you fail you might just serve stale content you already have access to. In this case “updating an out of date cached value” is more complex than “using the out of date cached value”.

I’d also argue that circuit breakers are an implementation of that, even though the “simpler task” is telling the user that whatever isn’t executed is considered unavailable for the time being. Not that useful to the enduser, as functionality is not available in both the erroring case as well as in the being told things don’t work case, but the latter is saner for the system to handle and by preventing access it might allow downstream system to recover over time.

You might find a lot more of that in topics around distributed systems – where failure is much more obviously inevitable – than in pieces around coding for a single machine. Also commonly distributed setups involve some degree of not being able to control all pieces of software running, so actual faulty implementations are more of an consideration.

Rich_Morin

Rich_Morin OP

Thanks! Here are some references I found for caching and the “circuit breaker” pattern, along with some implementations in Elixir:

  • How to write a caching server in Elixir

  • Cachex

  • CircuitBreaker (Martin Fowler)

    The basic idea behind the circuit breaker is very simple. You wrap a protected function call in a circuit breaker object, which monitors for failures. Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all. Usually you’ll also want some kind of monitor alert if the circuit breaker trips.

  • Circuit Breaker Pattern in Elixir (Allan MacGregor)

  • Hex package search

— All posts loaded —

Where Next? Top

Trending in Discussions Top

AstonJ
As the title says, please share what you’ve been up to with Elixir. Whether that’s been learning it, looking into it, making stuff with i...
2977 94592 917
New
cblavier
Hey there, It’s been more than a year since we started using LiveView as our main UI library and building a whole library of UI componen...
New
caslu
I want to open this thread for you all to discuss and help those who really like Ash but are still hesitant to use it in a real project. ...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
heathen
Quite interesting article Google brought me. Didn’t find any mentions about it here. What do you think in general? Would you use togethe...
New
marciol
It would be helpful to have a list of companies worldwide that hire engineers without prior experience in Elixir. Often, it can be quite ...
New
durvia
Anyone running long-lived stateful processes on BEAM? We’re building an AI agent runtime and would love to compare notes. We’re a small ...
New

Other Trending Topics Top

marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
webofbits
Aludel - LLM Evaluation Workbench Aludel is an embeddable Phoenix LiveView dashboard for evaluating and comparing LLM prompts across mult...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews