Can we create a system design guide similar to System Design Primer?

It is the last thing a beginner need. Build a system first, any system. Facebook started out with a bunch of lame ass PHP files. Maybe you don’t need to scale. Maybe you can rewrite the bottleneck as you found them. Thinking beyond 10X of your current peak is counter-productive.

Just because Facebook did it, or others did, doesn’t make it the right way of doing it.

When building a system that you aim to be a success you should design it from day one to scale for that success, otherwise you will have a lot of pain when the success arrives, and the fact that you didn’t care initially maybe even adversely affect your capability to cope with the success and/or let it fade away, because users aren’t happy with the performance of the system.

What I am saying is a strong believe I have from many years, but I also saw it recently in the Real-Time Phoenix - Build Highly Scalable Systems with Channels book:

This chapter looks at several common scaling challenges and best practices
to help avoid performance issues as you develop and ship your application.
We’re covering these topics before we build a real application (in part II)
because it’s important to consider them at the design stage of the development
process, and not after the application is already written.

So, I am not saying that you need to get it right from day one, but you should aim for it from day one, unless you are just coding a pet project, or one that you know for sure that will not need to scale.

Hi @derek-zhou, I think you’ve misunderstood the reason for making such guides. I’m not the one who’s facing the problem and asking help here. It is meant to serve as a pathway for all the Elixir / Erlang users in the future who need help scaling systems. As the Elixir / Erlang way is different from the traditional stateless system.
Anyway, I’m planning on building one myself and organise the knowledge I have into a repo. Since the only way to get answers in the internet is to post wrong answers only.

Hello, @Exadra37, Thanks for the reply. I have read the book as well. One of the things that is bug me is that when we connect Elixir/Erlang nodes, It forms a fully connected mesh. As you add more and more nodes to the network, the overall system performs degrades because the nodes needs to communicate a lot. This is the problem that a lot of talks in the conferences address. I want to organise such knowledge in clear manner for the community.

Feel free to contribute and correct mistakes.
Elixir Erlang System Design Repo

Don’t get me wrong, I am very much interested in seeing people sharing real world scaling problem and learning here. What I argued against is only premature optimization. Also scaling is a problem that has to take the current scale into account. Scaling from one node to 10 and scaling from 100 nodes to 1000 is very different. So I doubt the usefulness of a universal guideline.

What I argued against is only premature optimization

Understandable.

Also scaling is a problem that has to take the current scale into account. Scaling from one node to 10 and scaling from 100 nodes to 1000 is very different. So I doubt the usefulness of a universal guideline

The guide is suppose to start from a single node server and move to greater scale as the application scales stating the pros and cons of each approach.
For example, do you know how elixir handled 2M concurrent connections? I believe there were system configurations made to support such high number of connections. Similarly, The Whatsapp engineer who gave a talk on how his team was able to handle the same number of connections in a server, he said that he had to patch FreeBSD os to achieve such high number of connections. But we don’t know what they did to achieve it. That’s why I wanted to organise such knowledge. It may not be a universal guide. But it is better than nothing.
If you have experienced any scaling issues or know how to scale without any downtime, feel free to contribute.

This is great! Keep it up!
Looking forward to seeing the guide grow.

Even if its not heavily optimized, it would be a blessing to have such guide(s) that shows how to achieve higher connections just by small tweaks and/or picking low hanging fruit(s). Small wins they’d say but still a win to me nonetheless.