JEG2

JEG2

Author of Designing Elixir Systems with OTP

Why is this hard?

I’m closing out, for now, my series on questions at the heart of development with an analysis of when we need more abstraction. For example, taking @germsvel’s advice about architecting more accurate models can reduce future bugs.

Most Liked

dimitarvp

dimitarvp

Problem with many business people is that they don’t hear “he estimated the deadline”. They hear “he committed to the deadline”.

adamu

adamu

Thanks for sharing. I’ve added your site to my RSS reader :grin:

Personally I’m not really convinced estimates in software are possible at all. I think we might need to simply take everyone along for the ride so they can see first-hand how the design is unfolding.

waseigo

waseigo

I worked for many years in companies developing mechanical and electronics products, and I can tell you that there is nothing special about software in that regard. The most important difference between software and other artifacts is its mutability and (relatively) much lower cost of engineering changes, especially in the early iterations, which carries over much better in the later iterations too!

Top-notch companies with people engaged in multi-disciplinary product development work also know by now to prototype early, and delay decisions until the gremlins have gotten worked out from the concept, the system architecture, and its design/implementation in hardware, whether that’s problems with turbine-blade resonances, casting process tolerances, RF interference, warping of parts during plastic injection molding, or durability of components due to vibrations. In those cases, once you “hit hardware”, even in prototype form, you are subject to supply-chain lead-times and loops that are far more expensive than the cost of the prototype’s production.

Having said that, there are certainly still many non-software businesses that develop in what many deride as “waterfall”–and even then, it’s practically never waterfall. It’s most likely Stage Gate / phase-gate, and even then, it’s untenable for such a business to launch a new product every 5-7 years as they once were used to. So, most non-software companies have already for years now adopted some ideas of agility (not mentioning capital-A Agility, that’s a whole different story, with rituals and certificates and other nonsense), or introduced design processes and rules of thumb / internal guidelines for the design of key components.

Everything that has uncertainty has / must have a design process behind it, that reduces the uncertainty to an ALARP degree for the product to work as expected. From the “cartridge concept” of how some mechanical components (turbochargers, rotary valves are two I’m aware of) are designed to suddenly (after decades of design stagnation) prioritize repairability/maintainability, to figuring out the trade-offs behind the bounding box of the product, the size of the FPGA, the available space for the battery, etc., e.g. in this one of which I was in charge of.

The “software exceptionalism” is misplaced, sorry.

Edit: estimation is similarly an issue in non-software businesses, too. Generally, pointy-haired bosses who don’t understand or don’t want to understand something that goes against what they’re being pressured to enforce, are what ruins things. I once was asked to develop the shaft size of new turbochargers. Eh, we have design rules for that, how hard can it be? Original estimate (by my decidedly non-PHB supervisor) was 3 weeks. Within these 3 weeks it became obvious that the design rules were useless, as they targeted shafts of single-stage turbocharging systems. My investigation for updating the design guideline with new rules for higher pressures and 2-stage turbocharging systems took 13 months. Good thing that it wasn’t on the critical path, and thanks to parametric 3D CAD we could delay decisions about the shaft’s interfaces and converge as late as possible. 3 months into the investigation the formulas for the first 2-stage system’s parameters were available (after millions of Monte-Carlo simulations), so no harm was done. But 3 weeks and 3 months is still a crazy difference. Making bold estimations without understanding what you’re really estimating is hare-brained, regardless of software or non-software work. And most people in managerial roles are far removed from the actual system, to know what they’re really asking for, or what they can expect, or what slack they should allow for.

Last Post!

waseigo

waseigo

Another aspect that makes software engineering hard is that our mental model of the system (the domain model) can never exactly match the actual behaviour of the system when it runs.

This is actually a very good point! Non-software systems are subject to the laws of physics; yes, ultimately, the code you write ultimately compiles down to code that transistors work on, which are also subject to physics.

In mechanical engineering systems, for example: sure, we use simpler models and even CFD/FEA to build abstractions with predictive power by modeling laws of thermodynamics, etc. But a key difference to software is that we are closer to “bare metal”. We have far less choice of the system architecture, because our abstractions are lower-level, and the interfaces are tangible, not conceptual.

Any conceptual understanding we might have will differ from what is executed - not to mention different people might have different mental models of the system themselves.

That is also a big difference. The way I think of this is: between requirements and design, let alone implementation, in software there is a much bigger “cloud of uncertainty” compared to other, non-software artifacts, exactly because of the abstraction level of the systems being developed.

It’s the job of engineers and managers to ensure that we do everything we can to make sure everyone is on the same page conceptually, and we have to constantly strive to keep the implementation (code) and its executed state (processes, side-effects) as close to the conceptual model as possible.

IMO, there is less of a difference there, as anyone who has worked on tangible products from widgets to houses can attest. This is about defining goals, eliciting functional and non-functional requirements (including company-internal and legal ones), aligning them against the project plan, its viability/ROI and the related business plan, etc.–not very different to developing anything, software or not.

Anything that involves humans involves subjective opinions, and the more subjective the subject-matter, the more you have to navigate the process politically.

The upside of working on tangible products is that senior stakeholders can beat their chest all they want, but the laws of physics don’t care. Whereas, in software some PHB will come up with “requirements” stemming from chasing fads like “can we use GenAI for that?”, “of course we’ll deploy on Azure”, “of course we’ll use microservices, I read about this in the McKinsey Quarterly!”, “the project must use Agile” etc., and this throws a wrench in things…

Blessed are those who don’t have to work under the whims of idiots chasing the shiny new thing :slight_smile:

Where Next?

Popular in Blog Posts Top

marcelo
Long story short, over the past years we’ve been scaling our operation quite a lot at Unbabel. Most of our codebase is Python and some of...
New
AstonJ
Update: How to use the blogs section You can post in one of the Official Blog Posts threads (like this one), or, via Devtalk and a new t...
New
sheharyarn
In this article, I talk about the Elixir library Delta we at Slab just open-sourced, its various features including support for Operation...
New
mudasobwa
Just blogged on the general approach to creating easily extendable applications with plugin support.
New
brainlid
OTP 26 was released and the Elixir 1.14.4 builds have been updated! Erlang OTP 26 changed how map keys are sorted, or not sorted actually...
New
psantos
Just wrote a new blog post about how to deploy a Phoenix app using Kamal 2. Check it out and let me know what do you think: https://blog...
New
lawik
Building on other people’s work I bashed things together and suddenly I can know when someone is speaking using Elixir and Membrane.
New

Other popular topics Top

vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 42715 114
New

We're in Beta

About us Mission Statement