Why did Facebook leave Erlang

Is this a worry for Pheonix/Elixir devs?
https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-for-Facebook-Chat

2 Likes

From my point of view, there is a lack of technical information to make a judgement based on this article.

1 Like

Seems to be like they had some issues like everyone does and since Facebook has a ton on C++ devs they just decided to rewrite it in a language they know really well and is popular in their organization.

2 Likes

Interesting as they then went and bought WhatsApp that was serving millions of chat connections on a single server using Erlang. I wonder if they are (or have) phased Erlang out of that product as well.

4 Likes

I think that’s doubtful - WhatsApp has to be one of the most stable services around, given the sheer amount of messages they send that is really some feat.

I loved WhatsApp until they sold out to FB - now I use it begrudgingly (and hate their recent introduction of intrusive ‘data-sharing’ between the two companies).

10 Likes

I couldn’t agree more!

3 Likes

History repeating itself?

Y Store now C++ (2003-Feb-25)
Reddit: Why was Yahoo! Stores rewritten in C++/Perl from Lisp?

For me this begs the question: How likely is it that a “C++ programmer” who is unwilling or incapable of learning Lisp/Erlang can competently wield the techniques from Andrei Alexandrescu’s Modern C++ Design: Generic Programming and Design Patterns Applied (2001) (the point at which C++ actually can become interesting)?

5 Likes

I think the take-away from all this is that yes, if you throw enough $$$, programmers and time, you can rewrite anything in the platform that works for your company. Having a unified platform at a large company is FAR more important than the efficiency of a single application within the company.

Also, once an architecture is proven to solve the problem, it’s much simpler to copy that architecture in another language. Sometimes a buggy, half-implemented duplication of Erlang is enough.

7 Likes

The industry also has a history of shooting itself in the foot - while it can be tricky to create an application that manages a complex domain it typically it is even more difficult to keep its functionality current in the face of a continually and rapidly evolving business requirements.

You don’t want bigger teams - typically a small team of (competent) veteran developers should be able tackle complex tasks with “productive tools” much more effectively than a larger team of junior developers with the imperative tools they learned as part of or shortly after completing their education - fewer people require less (inter-)communication and experience should increase the quality of communication.

Scott Ambler aptly describes in the introduction of Agile Modeling (2002) p.4 the typical career path of a corporate developer:

I also believe that the way that developers learn their trade has a few unique dysfunctions. For the most part our colleges and universities are doing a reasonable job of educating developers for entry-level jobs. However, even if the schools were doing a perfect job and everyone was getting a degree or diploma, I suspect that we’d still have a problem due to the inherent nature of software developers. When software developers are young, in their teens or early twenties, they typically focus on learning and working with technology. They describe themselves as PERL programmers, Linux experts, Enterprise JavaBeans (EJB) developers, or .NET developers. To them technology is the important thing. Because technology is constantly changing, younger developers have the tendency to just barely learn a technology, apply it on one or two projects, and then start over again learning a new technology or the latest incarnation of what they worked with previously. The problem is that they keep learning the same different flavors of the same low level, fundamental skills over and over again.

Luckily, many developers become aware of this after several rounds of technologies - once you’ve written code for transaction control in COBOL, Java, and C#, you start to realize that the fundamentals don’t change. The same is true of database access in various environments, user interface design, and so on. Before long, developers begin to realize that many of the fundamentals, which they may or may not have been taught in school, remain the same regardless of the technology. This realization often comes when reach their late twenties or early thirties, typically the time when people start to settle down, get married, and buy a house. This is fortuitous because these new demands mean that developers can longer afford to invest vast amounts of time learning new technologies; instead, they want to spend that time with their families. Suddenly higher-level roles such as project lead, project-manager, and (non-agile) modeler become attractive to them because these roles don’t require the constant and intensive effort needed to learn new technologies. So, by the time that developers begin to truly learn their craft they’re in the process of transitioning out of their roles as developers. Luckily, new “young punks” come along and the cycle repeats itself. The end result is that the majority of people actively developing software are typically not the one best qualified to do it, and they don’t even know it.

In essence the industry that is most in need of competent veteran developers actively interferes with their late maturation process (by not paying them what they are worth).

And in terms of using more “productive tools” from Against the Grain: How We Built the Next Generation Online Travel Agency using Amazon, Clojure, and a Comically Small Team:

There were concerns about finding talent that had a kernel of truth in them. My response was that I was looking for the veteran software craftsmen I described above, and they were damned hard to find no matter what, and that pretty much anyone we hired would have to be trained to use the language. I’m not sure that went over so well.

Another question was, “How is Clojure suited to large programming teams?" My response was that I never intended us to have a large programming team. Programming in the large is a higher-order "code smell” (“architecture smell”?) that means you’re doing it wrong. Decoupled, distributed systems mean you shouldn’t be worrying about this problem any more. “Just pass messages." I don’t think that was quite what they expected to hear, either.

I laid some of the skepticism to rest once I was able to explain that Clojure was a JVM-hosted language, which meant that much of the Java ecosystem could be leveraged, including debugging tools, profiling, etc. Although no one said so aloud, I think that they took this to mean that if the acquisition was completed, the system could be migrated to Java. Heh heh.

I guess nobody ever got fired for selecting C++ (or Java).

From 2 myths and 2 facts about Clojure that change everything:

There are fantastic developers in many languages, but a lower bar to entry can make them harder to identify in crowds of applicants. For many organizations, Clojure isn’t just an advantage for building great systems; it is equally effective for building high-performing teams. Hiring managers tell us Clojure is their “secret weapon. It self-selects for smart developers”.

Given that this comes from Cognitect it’s about Clojure. But if you know where to look you can typically find similar statements with insert your favorite functional language here.

Somebody at Facebook thinks its going to be easier to find and cheaper to use mainstream language developers - that doesn’t mean they are correct; if anything Facebook is likely to take on some unwanted inertia.

11 Likes

There is also the issue of deploying and running the software. For something like Facebook that’s a huge issue. The more uniform the environment, the easier it is to keep things actually running. You don’t only have to train the dev team, you have to train the SRE’s as well.

For a startup, scaling to global size with only 50 engineers is a huge win. For Facebook, the economies are quite different, keeping 50 engineers around to run just one service isn’t a win.

2 Likes

I recommend https://telegram.org/ as messenger app (screw Facebook :slight_smile: ).

9 Likes

fb never really used erlang. they used ejabberd to bootstrap chat until they could write their own thing

i don’t think erlang was ever really considered for long term use outside of a handful of low level employees

2 Likes

Telegram is a sweet alternative

2 Likes

And anyone who ever managed ejabberd-based system can easily understand why they wanted to make the switch

2 Likes

Yeah, I like their games and bots. They’re a step ahead of Whatsapp. Last I saw it’s unclear what the Telegram stack is.

2 Likes

Telegram is full C++

2 Likes

Well, by far the most interesting part of Alexandrescu’s work was functors, and with C++11 lambdas offer a much easier way to get the same functionality and seem to be getting quite mainstream…

2 Likes

Well and then one of FB former Erlang guys built this tiny app called WhatsApp :slight_smile:

3 Likes

where did you get this info? I wondered myself few months ago, but it seemed to be unclear what they use

1 Like

From a dev that work with them.

4 Likes