Elixir in Anger and RCAs (root cause analysis) - How do you deal with them?

There rarely is only one root cause indeed. But to try and tame the chaos we can at least assign weights to the several different reasons that we found out (or only suspect of).

From then on it becomes a pretty simple game of “always kill the biggest contributor to problems first, then find the next one which is now the biggest, then kill that one too until we’re done”.

Are those assessments of weight distributions correct? Hell no, they are usually grossly wrong, but I found managers / CEOs / CTOs like it when you at least took the initiative to make some analysis and at least attempted to assign risk percentages to the several culprits you’d like to kill off.

Same way as big systems die of a thousand paper cuts, they are saved by stitching one cut at a time. The good news is that one “cut” (i.e. a subtle bug) can fix much more than one place in the code.

Program bugs are like dogs: they like to lie around in the company of other dogs. Don’t allow them. Eliminating subtle failure modes one by one can quadratically reduce the amount of visible bugs since most bugs really love to combine themselves with other bugs.

I found that being meticulous and just patiently untangling the fishing net that represents all the problems in a project really does pay off.

Bad news is, commercial teams rarely have the patience for this. All other things being equal, somebody younger inevitably comes around and gives the genius idea to rewrite the whole thing! And people being people, they will always prefer to pay the debt that’s due in the future and not the debt that’s due now. Because yeah, they’ll make a mess out of the rewrite as well.

(And don’t get me wrong, I more often advocate for rewrites than for repairs but I do recognize when a rewrite has a bigger cost than a repair. Many people can’t and don’t.)

To materialize part of that abstract philosophy when it comes to Elixir, my current work team taught me something valuable: make your code assert-like e.g.

:ok = ExternalApi.do_stuff()

That way, if it does not return :ok then your error capturing system (Rollbar, AppSignal, HoneyBadger etc.) will record the exact return value and next time around you’ll know how to gracefully handle the [apparently expected] error.

These assertive statements serve as invariants that clearly signal “the program is not expected to work if this statement does not return what we expect here”.

And that can and will give you the very first low-hanging fruit which to pick when it comes to bugs or logical errors in your code.

I mean yeah, we should all take a page from Ada and just have pre-conditions, post-conditions and invariants but failing that, we can at least emulate them closely enough.

Or do what Rust does: prefer slow and painful process of gradually eliminating compiler errors but when you are done then a huge chunk of all possible bugs are mathematically proven to be impossible in your program (lest a hardware freak event occurs e.g. bit flip in non-ECC RAM).

This is more about “Systems Thinking” than it is about Elixir. That being said, those with solid BEAM experience tend to have solid experience with complex systems. The message passing architecture allows for non-determinism which (as much as I hate the analogy) brings to mind biological systems and their passing of molecules between “actors”.

Compare/contrast with a doctor trying to treat a patient. The treatment has varying levels of abstraction. A treatment might make the patient comfortable (treating symptoms) as they expect the body to heal itself given some time. It might go deeper and recognize some vitamin deficiency and recommend supplements. It might go even deeper and recommend a change to diet and exercise. Or even deeper to societal concerns in which a diet that used to be fine no longer is because the available food has different nutrition than it used to. Even deeper down the system is a consideration of the economic motivations of food manufacturers.

So, is the “root cause” the vitamin deficiency, or a lack of some law limiting/requiring some food manufacture technique? It’s complicated and you can choose to address the problem at multiple levels.

Wow, these opinions are so great, this is truly the kind of insight I think has incredible value for everyone reading.

@dimitarvp Over the years, have you found a way to “guess” if a re-write is going to be more costly than a repair? I loved your analogy with “killing the bigger fish first”. Isn’t a re-write of a complex system always going to be more costly than a partial fix? Do let me know!

@gregvaughn Thank you for joining the conversation!
Yes, the original post is more towards Systems in general, but I believe that BEAM is a pretty complex system and that specific battle stories with it are surely to be valuable. I liked your analogy with the doctor. Have you, in your long experience, ever faced a situation like that (with a system) that you could share? Have you, for example, tried to fix an issue only to realize that there was something at a deeper level that could be addressed? As engineers, we often have to make decisions about what to fix and when. How do you decide?

+1 for Strong typing system! Opaque types certainly help !

This is a bit tangential but, on I’ve avoided doing anything on frontend (read : JS libraries) because it is messy. Even a small project is “hairy”. With Typescript, there is some relief. With opaque types in Elm → It was pleasure working on frontend.
So mathematical guarantees are awesome.

Right on. Customer impacting production problems are rarely caused by one disaster, but more often as several minor problems, each is not so bad in isolation, somehow happen at the same time.

This will be a bit of a spread out comment but I feel the separate points are connected.

There really is no clear-cut way but I can recommend one thing that almost always netted me an accurate assessment:

The amount of institutional and business knowledge that’s encoded only inside the code and nowhere else.

If your “legacy” system has that all over then no, please don’t rewrite it. You’ll make an even bigger mess and you’ll reintroduce bugs that have been long fixed in the old system even if the fixes are obscure and hard to find in the code (because again, critical business knowledge is encoded there).

Related to that, here’s something that most working programmers ABSOLUTELY HATE hearing:

The programming language makes very little difference.

We the programmers are petty like that and start dividing ourselves into almost-religious camps over BS syntactic preferences. That’s a distraction sitting in front of what’s important: what does that language / runtime / std library give you that is an edge over the competition? In Erlang/Elixir’s case that’s the BEAM VM due to its guarantees. In Haskell / Rust / OCaml that’s super strict typing that mathematically proves certain bugs are not going to happen in your program after it successfully compiles. In Ada it’s that the preconditions / postconditions / invariants of your class that will never be invalid so you can code in peace knowing that e.g. an account balance will never be a negative number or a person’s name never starts with a number etc.

RE: rewrites, I like rewriting stuff but let’s be clear, that’s a tinkerer’s point of view and priority; they don’t always align with business priorities.

But there’s something else. In the “Refactoring” book by Martin Fowler he makes a very important statement:

“Refactoring code you are new to helps you understand it.”

However, that’s refactoring, NOT rewriting.

In conclusion, I found that if I just start gradually refactoring an old code base to make it easier to iterate on it later, I very often find invaluable insights in the old code and a complete rewrite becomes unnecessary.

I would add that making sure you have the right fix for a bug is even more important than fixing the right bug.

I just encountered a situation where, a bug in some data processing logic caused some data to be missing in a particular API response where the FE was expecting it. Some dev had the idea that they knew another place where they could get that same data. Since they were assigned the bug (after all, the “problem” was in a bit of UX that was malfunctioning) they went ahead and added some fallback logic to look at data B if data A was missing. But actually, they were not correct that those datas were interchangeable, so their fix actually introduced another, even more subtle bug. This apparently happened a few times before the real source of the bug in the BE data processing logic was identified and fixed. But by then, a significant amount of tech debt had been added to an already rough bit of react code (why do there seem to be so many react devs that think unit testing components is optional?).

Bug reports should always be reviewed by more than one person and go through discovery etc phases before a fix is applied.

Well put! I’d like to add that while you may be able to write something that perfectly fits your present needs, chances are that the legacy code did too at some point. Code doesn’t become “legacy” on its own, and if the conditions that turned it into a ball of mud are still there, then a rewrite won’t stay perfect for long.

In a sense this ties in where the thread started: before diving into anything as drastic as a rewrite, it’s often a good idea to explore the reasons the codebase went off the rails. It’ll help you regardless of what you decide on in the end.

Since this thread is now mentioning legacy code, I must now make another question: What is legacy code?.

Code being old does not mean it is legacy if it can be maintained without issues and evolve without issues. So what do you guys consider a legacy code?

And how often does legacy code lead into subtle problems that then need RCA’s to be explained?

PS: My opinions on Legacy code are influenced by this book (I have not finished yet):

IMO legacy code is hand-me-down. The original authors are gone; the current maintainers are reluctant to make extensive refactoring. It has nothing to do with the technology.