cs-victor-nascimento

cs-victor-nascimento

The state of Riak Core, Lasp and distributed programming in BEAM

Hi everyone! Just some general question about the state of distributed programming in BEAM.

One of the strongest feat of the BEAM is its concurrency and we all know that (and after this post I think the situation is highlighted dearly). One thing though that is sometimes mentioned but not always discussed is distributed computing.

The thing is: most of the state based libraries we have published on Hex are not meant to be used on a distributed system. This responsability is left to whoever depends on them. Take “cache” libraries for instance: most of them are not truly distributed in the sense that their state would be lost on a node down event.

Currently it appears we have 2 options to deal with that: Riak Core and Lasp lang.

Riak Core is living through different forks with the community making patches to support recent Erlang versions but I guess we are all waiting for the Bet365 position on the matter. Will they put someone in charge of accepting patches, reviewing code and things like that?

Lasp is a bigger departure from traditional Erlang. The projects has a big motto (“Planetary Scale Applications”) but the documentation, samples and whatnot is really difficult to grasp (at least for me).

This is something that, in my opinion, should be included in the ecosystem. If Elixir maintainers could add some “flag” to mix new to make it more distributed friendly it would be awesome (like an easier riak-core). Maybe for Elixir 2.0?

Am I lunatic? How would you people go about with a distributed app? What is the state of distributed programming in BEAM?

Most Liked

dimitarvp

dimitarvp

IMO that’s true everywhere. Not going Elixir fanboy mode here but everywhere I was, be it a wage programmer, or consultant, or hired-for-a-project role, and no matter the language and stack, everybody was focused on delivering the next sprint or, in the best case scenario, mid-version release (say, going from 1.1.5 to 1.2.0). Strategic investments in tech blessed by the business people are as rare as flying unicorns.

There are no places like the original labs where UNIX, Plan-9 and a lot of other tech we use to this day were conceived. Too much short-term capitalism for those to ever see the light of day again. :009:

I respectfully both agree and disagree.

  • Java doesn’t have robust solutions per se. And where it has good solutions they are very rarely out of the box and painless. The ecosystem has 500 metric tons of crap and are riding the horse named “look how many companies are using us, we have to be doing things right, we are sure!” – while in truth the slogan should be “everybody is too damn scared to move away from us because nobody has an idea how their systems work anymore”. Sorry for being a cynic but for 8 years in the Java space (2003 - 2011) I’ve never, ever, seen one Java team that legitimately knew what they were doing. And I’ve never seen a team actually being on top of all their business logic and code. Maybe only a bunch of very bright young people in SAP Bulgaria where I contracted once. And that was because the project wasn’t big.

  • Java has a lot of useful libraries for hundreds of protocols and integrations with many 3rd party APIs, I am not denying that – and that’s its main selling point in the last 7-8 years and is what maintains its momentum. But in terms of distributed computing Java is, to this day, worse off than Erlang/OTP and is almost on par with Go. Their only chance is their new functional constructs and producing a zero-fuss deployment of those to something like the cloud serverless functions or short-term specialized containers. But they can’t do that yet because the JVM takes a long time to start (there is a lightweight distribution though, so they might get there). Ad infinitum.

  • Go is not much better but I still like its community much more than Java’s. Go programmers seem to be a very particular bunch and they don’t mind getting their hands dirty. And most of them are very well aware of the pros and cons of the language and the runtime and they mostly stick to highly specialized libraries, CLI tools or microservices. Go’s community is a really excellent example of programmers who are realists. That being said, goroutines and channels are very far away from the robustness of the OTP. Deadlocks usually don’t happen but a subclass of the multicore problems still apply. This is well documented over the net; try googling about for “goroutines deadlock” or “go panics channels” or along those lines, you will find some quite disheartening articles… Gotta say, these discoveries shook my faith in the language’s multicore story.


I will agree that this community is small. But you must not underestimate the core team. Many things that have to be reinvented in every Erlang project are being gradually added to Elixir (Task.async_stream, GenStage and Flow being pretty good examples). Lately there was a huge thread with tons of discussion about how to solve the compile-time / runtime / app-startup-time configuration and a lot of progress has been made in terms of the core team gaining better understanding of the shape and the area of the problem.

The vibe I am getting from Jose and the others is that they are well-aware of the current pain points but are gradually creating the building blocks of the solutions. The Elixir core team is very judicious before adding stuff and many of us are applauding them for that. When they do it, it will be done right.

That being said, nobody forces you to wait. By all means, use ZooKeeper or a particular Kubernetes setup in the meantime; we all have paid jobs to do after all and we cannot wait indefinitely.

Last but not least, distributed computing is one of the most stubborn and complex problems in this computing age. It would be unfair to judge Elixir by its inability to provide transparently and out of the box functionality that 99% of all others cannot provide even with a bunch of tooling attached.

19
Post #9
michalmuskala

michalmuskala

I will still encourage people to actually go out and build things. Waiting around and expecting the core team to solve all problems isn’t going to work :wink:. The main goal of the core team is to maintain elixir itself - nothing more, nothing less. The language is designed to be extensible and it’s up to the community to leverage that and build libraries for solving problems they face. Relying on a handful of people does not scale.

jeremyjh

jeremyjh

Am I the only person who feels disappointed over-all in the maturity of this space? From the way some people describe Erlang it seems like these kind of solutions are the entire raison d’être yet the actual state of it is fragmented and largely of dubious production value. If I want to build a large distributed application today in Elixir and the scale outstrip the capabilities of :global (maybe 30-40 nodes) I feel like I’m on my own and honestly would probably base it on Zookeeper or Consul and grpc rather than using a native beam solution.

11
Post #7

Where Next?

Popular in Discussions Top

andre1sk
A big advantage to Elixir is all the distributed goodness but for many applications running on multiple nodes having integrated Etcd, Zoo...
New
blackode
Elixir Upgrading is so Simple in Ubuntu and It worked for me Ubuntu 16.04 git clone https://github.com/elixir-lang/elixir.git cd elixir...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31265 143
New
AstonJ
Are there any Elixir or Erlang libraries that help with this? I’ve been thinking how streaming services like twitch have exploded recentl...
New
Crowdhailer
I’ve been hearing much about the new formatter and it’s something I have been keen to try. I find examples buy far the most illuminating...
248 19327 150
New
mbenatti
Following https://github.com/tbrand/which_is_the_fastest |> https://raw.githubusercontent.com/tbrand/which_is_the_fastest/master/imgs...
New
RudManusachi
What configs will make sense to put to runtime.exs? – A bit of how I configure apps: I have generic configs in config/config.exs, dev...
New
ben-pr-p
In general I’ve been sticking to this community style guide GitHub - christopheradams/elixir_style_guide: A community driven style guide ...
New
Qqwy
Looking at the stacks that existing large companies have used, WhatsApp internally uses Mnesia to store the messages, while Discord uses ...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31265 143
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52673 488
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement