JEG2

JEG2

Author of Designing Elixir Systems with OTP

Why is this hard?

I’m closing out, for now, my series on questions at the heart of development with an analysis of when we need more abstraction. For example, taking @germsvel’s advice about architecting more accurate models can reduce future bugs.

Most Liked

dimitarvp

dimitarvp

Problem with many business people is that they don’t hear “he estimated the deadline”. They hear “he committed to the deadline”.

adamu

adamu

Thanks for sharing. I’ve added your site to my RSS reader :grin:

Personally I’m not really convinced estimates in software are possible at all. I think we might need to simply take everyone along for the ride so they can see first-hand how the design is unfolding.

waseigo

waseigo

I worked for many years in companies developing mechanical and electronics products, and I can tell you that there is nothing special about software in that regard. The most important difference between software and other artifacts is its mutability and (relatively) much lower cost of engineering changes, especially in the early iterations, which carries over much better in the later iterations too!

Top-notch companies with people engaged in multi-disciplinary product development work also know by now to prototype early, and delay decisions until the gremlins have gotten worked out from the concept, the system architecture, and its design/implementation in hardware, whether that’s problems with turbine-blade resonances, casting process tolerances, RF interference, warping of parts during plastic injection molding, or durability of components due to vibrations. In those cases, once you “hit hardware”, even in prototype form, you are subject to supply-chain lead-times and loops that are far more expensive than the cost of the prototype’s production.

Having said that, there are certainly still many non-software businesses that develop in what many deride as “waterfall”–and even then, it’s practically never waterfall. It’s most likely Stage Gate / phase-gate, and even then, it’s untenable for such a business to launch a new product every 5-7 years as they once were used to. So, most non-software companies have already for years now adopted some ideas of agility (not mentioning capital-A Agility, that’s a whole different story, with rituals and certificates and other nonsense), or introduced design processes and rules of thumb / internal guidelines for the design of key components.

Everything that has uncertainty has / must have a design process behind it, that reduces the uncertainty to an ALARP degree for the product to work as expected. From the “cartridge concept” of how some mechanical components (turbochargers, rotary valves are two I’m aware of) are designed to suddenly (after decades of design stagnation) prioritize repairability/maintainability, to figuring out the trade-offs behind the bounding box of the product, the size of the FPGA, the available space for the battery, etc., e.g. in this one of which I was in charge of.

The “software exceptionalism” is misplaced, sorry.

Edit: estimation is similarly an issue in non-software businesses, too. Generally, pointy-haired bosses who don’t understand or don’t want to understand something that goes against what they’re being pressured to enforce, are what ruins things. I once was asked to develop the shaft size of new turbochargers. Eh, we have design rules for that, how hard can it be? Original estimate (by my decidedly non-PHB supervisor) was 3 weeks. Within these 3 weeks it became obvious that the design rules were useless, as they targeted shafts of single-stage turbocharging systems. My investigation for updating the design guideline with new rules for higher pressures and 2-stage turbocharging systems took 13 months. Good thing that it wasn’t on the critical path, and thanks to parametric 3D CAD we could delay decisions about the shaft’s interfaces and converge as late as possible. 3 months into the investigation the formulas for the first 2-stage system’s parameters were available (after millions of Monte-Carlo simulations), so no harm was done. But 3 weeks and 3 months is still a crazy difference. Making bold estimations without understanding what you’re really estimating is hare-brained, regardless of software or non-software work. And most people in managerial roles are far removed from the actual system, to know what they’re really asking for, or what they can expect, or what slack they should allow for.

Last Post!

waseigo

waseigo

Another aspect that makes software engineering hard is that our mental model of the system (the domain model) can never exactly match the actual behaviour of the system when it runs.

This is actually a very good point! Non-software systems are subject to the laws of physics; yes, ultimately, the code you write ultimately compiles down to code that transistors work on, which are also subject to physics.

In mechanical engineering systems, for example: sure, we use simpler models and even CFD/FEA to build abstractions with predictive power by modeling laws of thermodynamics, etc. But a key difference to software is that we are closer to “bare metal”. We have far less choice of the system architecture, because our abstractions are lower-level, and the interfaces are tangible, not conceptual.

Any conceptual understanding we might have will differ from what is executed - not to mention different people might have different mental models of the system themselves.

That is also a big difference. The way I think of this is: between requirements and design, let alone implementation, in software there is a much bigger “cloud of uncertainty” compared to other, non-software artifacts, exactly because of the abstraction level of the systems being developed.

It’s the job of engineers and managers to ensure that we do everything we can to make sure everyone is on the same page conceptually, and we have to constantly strive to keep the implementation (code) and its executed state (processes, side-effects) as close to the conceptual model as possible.

IMO, there is less of a difference there, as anyone who has worked on tangible products from widgets to houses can attest. This is about defining goals, eliciting functional and non-functional requirements (including company-internal and legal ones), aligning them against the project plan, its viability/ROI and the related business plan, etc.–not very different to developing anything, software or not.

Anything that involves humans involves subjective opinions, and the more subjective the subject-matter, the more you have to navigate the process politically.

The upside of working on tangible products is that senior stakeholders can beat their chest all they want, but the laws of physics don’t care. Whereas, in software some PHB will come up with “requirements” stemming from chasing fads like “can we use GenAI for that?”, “of course we’ll deploy on Azure”, “of course we’ll use microservices, I read about this in the McKinsey Quarterly!”, “the project must use Agile” etc., and this throws a wrench in things…

Blessed are those who don’t have to work under the whims of idiots chasing the shiny new thing :slight_smile:

Where Next?

Popular in Blog Posts Top

ErlangSolutions
An infographic that compares Erlang, Elixir, and Go’s strengths in the respect to the programming languages’ concurrency, reliability, sc...
New
pillaiindu
Our very handsome and humble José Valim (@josevalim) just published Kubernetes and the Erlang VM: orchestration on the large and the sma...
New
tmartin8080
Sharing some articles I’ve written recently while working in Elixir. Some are how-to guides and others for reference. Will update this ...
New
RudManusachi
Hi there! Recently I was playing around with extracting and updating data in the DB and for fun challenged myself to try to implement a ...
New
mssantosdev
This post is a guide on how Norba and I are building LiveMatch, a real-time app for soccer to follow multiple games in one place. https:...
New
victorbjorklund
I’m showing you how you can customise the phx.new generator to give you a new Phoenix project the way YOU want it. In this post I show yo...
New
rocket4ce
A comprehensive guide for deploying Phoenix 1.8 applications using Coolify on Hetzner servers. Covers server setup, Coolify configuration...
New

Other popular topics Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44265 214
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement