Becoming a better engineer: how to learn how complex systems work together?

Hello all, my first real software engineer job was in Elixir and I have since, largely through the help of this community, grown as an engineer in a very significant way. Firstly, thank you.

As to my question, I was talking with someone I respect in the industry and they told me something along the lines that the skill of building CRUD apps is not where the highest value is - this will be automated in the future - but the value lies in being able to enter a complex system and making changes. In short, understanding complex systems is a skill I should focus on.

How do I grow this skillset? Any input is appreciated as always.

Side note: for the curious geohot goes on an entire rant about ‘software engineers’ here: I found it interesting as well. (Sorry I don’t have a timestamp).

1 Like

This strikes me as somewhat of a strange question. Probably there was more to what your mentor said, but “Understanding complex systems” is not a software thing, it is a cognition thing. All disciplines involve systems of one kind of one kind or another, and excelling in any of them requires understanding them, and a bit more than that, I think. Certainly, if the only system you’re comfortable with is the (current) stack you’re using to build your CRUD app, then there is a lot of room for growth. But even if you understand just that very well, that puts you beyond a lot of web developers, in my experience, for whom “it works” is the end of the show, and honestly, beyond the reach of any AI-generated web apps in the indefinite future, IMO. I think software involves a lot more art than cannot itself be automated than most people seem to think. And maybe that gets us closer to the truth. It’s not so much that you have to know the intimate details of OS design to be an excellent web developer, or even “software engineer,” but the more really well-designed engineered systems (including some humble web apps) you are exposed to and understand, the better engineer you become yourself.


Yeah, that’s what “they” were saying was just a few years away when I picked up Rails 10 years ago too :stuck_out_tongue:


Don’t get stressed about automation of programming just yet. Even if some organizations can actually do it they are too afraid to open-source it and will only use it to their advantage. The natural human greed will prevent such techniques being public for quite some decades in the future still.

Your question is very generic as has been said. Some of the Elixir books actually walk you through building rather complex apps (don’t have the exact titles in my head, sorry) and that likely will help you a lot.

But beyond that, you need to practice! If you ask a bit more concrete question I am sure the forum will help you.

1 Like

Being “able to enter into a complex systems and make changes” can mean many things; it could mean just being able to understand a spaghetti code mess well enough to make changes. That’s a useful skill to be sure, but I think having a broader view is better here.

Complexity is an enemy of software engineers; we need to be able to deal with it to make changes to existing complex systems, to create new systems with minimum of accidental complexity, and to reduce accidental complexity in already existing systems. A few thoughts on how to develop this as a skill:

  • In daily work, taking on challenging tasks dealing with multiple moving parts is good practice, especially when you don’t know those parts well yet. If you can take the time to refactor and improve the existing code and reduce its complexity, all the better.
  • Try to take on larger responsibilities that require you to have a better understanding of systems you’re dealing with and to make architectural decisions.
  • Improving understanding of fundamentals helps to better grasp the pros and cons of different choices that go into building systems. One book I recommend for this is Designing Data-Intensive Applications

Here is my practical advise:

  • Learn touch typing and vim, and work on linux server command line and config
  • Coding a lot, any lang at 50K lines of code
  • Read one language spec, I read java one, so after that you will not fear any new lang
  • Read Riak pagers and dig into Riak Core source code, and after that you kind of comfortable with any complex projects
  • Use Wireshark

Hi @tiotolstoy

CRUD apps, if done well, are not a simple thing. One could spend a lifetime exploring the ins and outs of making a “simple” application work well, and besides, it takes a lot of skills to make a system simple.

Building software that is easy to maintain and evolve is in itself all but trivial, and even simple CRUD apps involve so many skills, often provided by separate specialists: back-end, front-end, infrastructure, reliability engineering, UI/UX, performance, …

Complex systems are not a goal: the goal is to design the simplest system that does the job well. Some systems are inevitably complex for valid reasons, like the subject matter being actually complex, but these are a small subset of the real systems. Most complex systems are “accidentally complex”, as they could be simpler, but they ended up being complex for historical reasons.

In any case, each complex system is complex in its own way, so it’s not possible to “learn complexity”. One can mature experience on the job on a particular system, but the acquired knowledge is hardly valuable in the job market.

A skill that one can learn, and that turns out to be useful in any job, is to “break down” complexity. It is the ability, for example, to divide a big problem into smaller ones. Or to model a problem in a way that only focuses on what really matters. Or the ability to recognize trade offs, and decide on which complexity brings more value than it costs, and which doesn’t.

My suggestion is to stop worrying about the immediate career implications of simple vs. complex systems: CRUD applications are not going to be automated in the foreseeable future, and anyway, as the tooling improves, so will your possibilities as an engineer. Focus instead on breaking down complexity when you inevitably meet it. Learn to model problems, and understand trade-offs. The skill is to make things as simple as they can.


If your immediate goal is to enter a complex system and make changes, you should stop worrying about understanding the complex system for a while. In stead, you should try to isolate out the part you want to change (zoom in), pretending other parts don’t exist.
Once you have zoomed in and contributed in several small areas, you can zoom back out and try to understand the big picture. I find this approach better suited for me. If I tryed too hard to figure out the big picture from the beginning, I would be overwhelmed by the amount of details and lost my confidence.


The only reason we write complex code is because the real life is complex and our job is modeling it. Understand how “entities” (the vague definition is used on purpose, because it could be literally anything) interact with each other on the real world and translate it to code on the proper abstraction level is the heart and soul of a good software engineer.

Our job is mostly turn real world systems into software systems.

For instance, if you are modeling a calculator app you have some entities:

  • Digits ex: 1,2,3,4
  • Operators ex: +, - , * , /
  • Aggregators ex: ( , [ , ) , ]

And you can think how they interact with each other:

  • Digits are the core values of your system, they are the inputs and the output it self
  • Operators have semantic value on your domain, it means that are business rules (Math rules) that this operators applies, operators only make sense when used with digits 2 + 2
  • Each operator have a relation with other operators, like * and / must be solved first and + and - last
  • Aggregators applies business rules, that change the the default priority of operators

With this you can start modeling this calculator system in a software system:

  • If you use a OO language, create a class for your Digit entitiy, which could hold information about the digit, beyond the digit value it self, or a struct could represent it in Elixir.
  • Each operator could be defined as functions that expect digits as entries sum(digit1, digit2) inside their own modules
  • Aggregators must be analyzed before hand on the calulation, and you will find out that use a classic data structure called stack to do it will help you a lot.

This is a very simple example that I could talk a lot more… but the point is just to show that for modeling a complex system you have to understand the domain/business it is related to and try the best you can implement this relations as code.

I’ll probally be hated on this, but I think an Object Oriented language can teach you a lot about this, it is the most popular programming paradigm for a reason. (:

That said, to “enter a complex system and make changes” you have to understand what the system is about and how it models the reality. Knowing how and when entities interact with each other.

Entities are usually represented as a well defined document inside a project, in elixir we call them modules on OO classes, but the fact is that stuffs relates to each other.

A usefull technique is trying to identify:

  • What is the input?
  • What is the output?

The output must be related to the input, so as you read the code, you will identify how it transform or use the input to generate the output.

For example:

Input: Client Identifier
Output: Client Name

Predict: Reading this code you probally will find a database call using the identifier and retrive it’s name

Use this techinique of thinking about inputs and outputs, and predict an expected behaviour has helped me a lot when trying to understand new systems. Because I knew exact what I was looking for and more important why I was looking for it.

With this I can ask the right questions even when I predicted wrongly.

Real world example:

I was trying to understand a function that updates the user table, but the input of the function was not the user id, instead it was the user document.

I knew that another table has this information, linking document numbers with users ids.

But I did not found this on the code.

After this I could ask for help with the exact question: “How this functions knows which user must be updated?”

The answer was: “The user id is a hash of the document”

With this I could learn and understand the system easily, because I knew what the function should do and predicted (wrongly) how.

This technique will improve on time, because your prediction power will improve with time and knowledge! And It forces you to always think: “How I would do this?”

I’m already writing a lot, but to wrap up.

  • Understand the context the software you are working with was built.
  • Think about inputs and outputs, and how you would built this
  • Great books to read:
    • Clean Architecture (General concepts)
    • Refactoring - Improving Design of Existing Code (General concepts)
    • Desiging Elixir Systems With OTP (Elixir oriented)
    • Elixir in Action (Introdutory book to elixir lang, but the practical advices about reasoning about elixir systems are gold)
1 Like

Slightly off-topic but I wanted to link to an interesting talk based on something @oliveiragahenrique wrote:

No hate, sorry. :smile:

There is an interesting talk from Richard Feldman where he dives into the history of various mainstream languages and why they are popular. OOP being a great paradigm for modeling isn’t among them though. It’s a super interesting and entertaining talk, highly recommended!


Just for clarity, I don’t think OO is the best paradigm for modeling. You can do great models with any paradigm. But I do think that OO modeling is closer to the way humans naturally think, which could help if you are starting out with the topic.

Gonna check out the talk! Looks great! Thanks! :smiley:

I think maybe it’s closer to the way computers think, maybe because we made them. It is very different to “model” anything without objects, all processes have state. But conversely there is no state not in process. I am relatively new to the functional paradigm, but it seems like it emphasizes the flow of data through a program in comparison to the state of the data. So the modules you add and what functions they contain have more to do with the way data is processed than what is being processed. It encourages spreading complexity out over transformations with relatively simple data being passed along at each step. Whereas OO encourage spreading complexity out through the state, with relatively short “distances” to travel before a data flow gets represented as state again. But each can be taken too far. The key to good engineering, “elegance” perhaps, is striking the proper balance. Bad OO projects, IMO, end up with either God objects or (arguably) superfluous objects like ThingThatDoesASoThatWeCanDoBNext, which seem like less of a threat in FP, but that doesn’t mean FP doesn’t have it’s own balance. I think I’m still too much of a beginner to define it as confidently though.


In my opinion, the skill to understand huge and complex systems comes by trying to understand them.

In the first iterations there should be someone with you, explaining the compononts and how they interact together. Later on you will be able to get into those on your own.

There is a reason why we plan about a week for onboarding new devs into our comnpanies product. This week includes a shallow training in python and go (as we use those languages) as well as a brief tour of the components/services of the product, but without showing any code.

During daily business (which is sadly often enough unrelated to the product) there is usually found some time inbetween to get the new one into smaller issues in the product, they have to fix in a pairing session (only bug fixing, no feature adding!). After they have done this a couple of times and they feel proficient enough, they can access the ticketsystem freely and pick their tickets. And we are still open for questions.

Understanding complex systems is about how well they are explained/presented to you. You won’t be able to understand those on your own without putting in an amount of time that is close to the initial developing time.

Beeing able to zoom in and out during this introduction helps, but is not necessary if the system is documented well, or if its documented even better, then zooming happens without you even realizing, but I have seen such a documentation only once so far and that was for a system built at university solely for educational purpose.


This… if you can understand the difference between complexity inherent in the problem and accidental complexity introduced into the solution as a result of technical trade-offs (or more likely poor engineering), and become acutely aware of it in very technical decision you make (or any assessment you make of existing systems) you will go far. There’s almost an infinite amount of consulting work available to people willing to reverse accidental complexity out of existing systems.

One point on why applications won’t be built by AI in the short term related to this - complex real-world problems are often simplified to be represented by computer systems. Often this is for lack of available data (particularly in engineering / scientific systems), but also to keep the system simple enough for end-users to use. Joel Spolsky has a typically thought-provoking article on this - Understanding what complexity to expose to the user, and what complexity to remove altogether and what the likely impact is, is something that is going to require human judgement for a long time yet.


Beyond a CRUD app, there’s a lot of complexity to be tackled by gaining expertise in these topics:

  • Beck’s Test-Driven Development
  • Evans’ Domain-Driven Design
  • Garsiel’s How browsers work
  • Martin’s The System Design Primer
  • Kleppmann’s Designing Data-Intensive Applications
  • Kim & Humble & Debois & Willis’ The DevOps Handbook