Hi, i wanted to create a post and get some feedback regarding elixir ecosystem and a project that is on the way. The short story
My boss assigned me to research around the ecosystem landscape in order to see if a specific ecosystem would help us create robust services in the edge. So far the only other language i am researching around is Golang
About me
I mainly use Node.js but also i use Golang for creating tools/scripts and whatever is needed that i need to build fast and light weight, i am not an Elixir devloper, i just happened to start learning some of the concepts this ecosystem has some months ago and they got me excited enough to have it in the top of the list before i start this research.
The concept
The main idea is not new, i should be able to connect to multiple different controllers (Modbus TCP/RTU), collect the data and either persist or push to a Message Queue. As i imagine, i would have multiple different server applications near the Field, communicating data in one way or another on a Centralized Server. Of course if possible some of the Data Processing could take place in the Field.
[Think of an X amount of controllers connected to a PC-Router per Physical Place]
Proposed Setup in the Field
Industrial Grade Router
Industrial Mini-PC (probably with at least 8 GB of RAM)
The Problems
Some things that need to be considered while researching on this topic are the following
Being able to run in a lightweight manner
Being able to change configurations or update parts of the app on the fly
Fault tolerance as much as possible considering that there are a lot of points of failure (network, legacy systems, etc)
Some Golang Findings
Seems that there is a maturing Ecosystem around the Edge Computing like (EdgeXFoundry)[https://www.edgexfoundry.org/]
Fast to Develop
Runs Lightweight
Lessons from the Past
In the past i have built a similar solution as a prototype that used Node.js, Mosquitto and TimeScaleDB but seems that in this setup it would be tough to scale (as far as my DevOps is explaining )
Main Question
Do you think that the Elixir Ecosystem could possibly offer an advantage on a project like this ? Have you heard of similar Industrial Grade projects run on Elixir/Erlang/BEAM ? Any information would be useful so i could have a more complete picture for this Ecosystem when i will need to present it to my Boss.
Thanks in Advance
PS
This was my first post so please forgive me in case i use the wrong Category. If you need any further clarification, please let me know.
The 3 bullet points of your “the problem” sections seems to be well within some of erlang/elixir’s core proposition.
Throw in the excellent Nerves project and I believe you got everything you need to focus your research on.
Thanks for your reply, to be honest i alreeady took a look on the Nerves Project and i got the impression that it is aimed to solve these kind of problems in “smaller” devices so i quickly stopped considering this since in the case of an industrial PC (even with low resources) i was thinking of using Elixir and leveraging the OTP for fault tolerance, i am not sure if i got it wrong here. Good to know that the solutions to the “problem” section seem that could be solved using this ecosystem, although i am trying to see if there are any “good pactices” for projects like these, some system design to help me understand how to properly use the OTP to solve the problems in a correct and future proof way.
Yes i already saw that project some time ago but i was not certain if it is production ready. Also, please forgive my ignorance but from reading the docs of Erlang/Elixir i noticed that most of the strengths reside in the BEAM so the VM is the place that magic happens (connect remotely on a running Instance for debug/inspection or hot code swap) so in theory even if Ergo seems to be production ready, still some of the BEAM features would not be available correct ?
I haven’t used it but my understanding is that it’s production ready. From their GitHub repo https://kaspersky.com , https://www.ringcentral.com and https://lilithgames.com are using it. Sure some BEAM features will be missing. But it seems so compatible with Erlang that you can use some Erlang tools like Observer with it. It will bring things like supervision trees for example that should help with app stability by restarting crashed services. I’ve seen code hotswap talked about quite a bit, but I don’t think almost anyone uses it.
I’m not professional Elixir developer but these things come to mind
Elixir pros
Immutable
Process/Memory isolation making it more robust. Process can’t access other processes memory directly.
Functional language (just functions and data). It being functional might be con as well for hiring people.
Probably better 99th percentile latency because of memory isolation. If you create a process, do work and exit GC might not even run and memory is released when process exits.
Better remote inspection and able to run code remotely.
Single popular web framework called Phoenix. Our company moved from C# to NodeJS/TypeScript, I hate that there are so many different web frameworks with some cons in them all.
Phoenix LiveView that you can use to create real-time admin interfaces easily.
Elixir as language has reached point where it’s pretty much ready according to its creator. So it won’t be evolving like C# for example. So you can just focus on building things with it.
Go pros
Statically typed.
Better IDE integration because it’s statically typed. My understanding is that JetBrains GoLand IDE is excellent.
Faster running code.
Compiles to single binary.
Quite likely more maintained good quality libraries available.
Easy to learn. I learned Go fully in one day (before generics) coming from C#.
Thanks for the detailed list, yes my findings point to these in one way or another as well and all of these points are excellent. I suppose i can take a second look on the Ergo and see if it could help me in any way.
Golang’s creator’s view that their audience is "not capable of understanding a brilliant language" rubs me the wrong way — YMMV. I’ve also found their writing and messaging to be defensive and irrational. I worked at a household-name software company, and the engineers with experience hated the language.
I found Go very difficult to learn. For example, how to accomplish simple things. How to do application-level multi-threading. How to write safe code, and code defensively. How to write maintainable code. Circa 09/2021, several of these were unsolved problems IMO in Golang.
I started using C# professionally starting from I think version 1.2. Before it had generics. Anyway I knew all about thread safety and other pitfalls that could bite you. So yes even if it was very easy for me to learn, might not be that for others with different background.
I agree with you that in Go you sometimes have to write lot more code to achieve some simple things compared to other languages. It’s partly because Go is very sinple and language wants to have only one way to do a thing. Go not having generics was big part of this problem. But latest Go 1.18 adds generics to the language. It might take a while until libraries catch up. Even core libraries yet don’t seem to have things like generic sort.
I agree. And by making it so simple — in the way Assembly is simple — the designers failed in this goal. There are 100 ways to do any particular thing.
For me it was a programming task along the lines of “Give me a sorted list of the map keys.” Something that’d be one line, 20 characters in Python. In Go, this involved a tedious trip to Stack Overflow and a choice between multiple low-level ways to implement it. Each several lines of un-expressive code.
That’s excellent - it has the potential to solve half of Go’s problems. Still outstanding will be (1) the goal of making an (IMO) nerfed language and (2) the lack of error handling in the language.
Also in C# that is very short, just dict.Keys.Sort(). So yes it’s bit awful when you have to write lot of code to achieve something like this in Go. Hopefully added generics starts solving that. But I think it will take many years until generics are fully utilized in the language.
Go doesn’t really make any assumptions on what you need app and data structure wise, but does make a fair number of them when utilising the standard library for networking, database pools etc. It’s always been irritating that Go forces the user to write known structures, such as sets because Go strives for simplicity (when in truth, is anything but).
Anyone can walk through the excellent Go by Example and up out knowing how to write Go provided that you come from a C background, but I find that alone is insufficient in any sort of application design. While the messaging from Go is that there’s “one” way to do things, it’s hardly the case as pages such as Effective Go and CodeReviewComments · golang/go Wiki · GitHub are necessary to get some confidence in Go itself.
While Go is incredibly fast, it lacks robust in-built fault tolerance which is a deciding factor at scale. There’s an excellent video The Robustness of Go • Francesc Campoy • GOTO 2018 - YouTube which illustrates the key differences between Go and Erlang; especially on what Erlang does (process isolation, in-code reloaded etc), which simply doesn’t exist in Go outside a third library. In short, the video concludes that tools such as Kubernetes smooth the edges with Go, especially around Go panics which can, if done incorrectly, can tear your entire application down abruptly. This reinforces my prior point; Go can be written quickly, but you have to be extremely defensive and write code to handle those instances, made somewhat worse by Go’s anemic error handling.
If you do decide with Go, you have to cater to the possibility of unexpected application shutdown and ingrain some sort of behaviour that kicks in when it occurs. It could be systemd, or Kubernetes, but you’re forced to acknowledge that runtime errors can propagate to application errors and have to plan accordingly. In Erlang, while this can happen, you have in-built measures in places to ensure that the application isn’t compromised in the presence of runtime errors.
Just a clairification on Nerves - it is still full OTP / Elixir, but bundled for embedded systems.
A couple of other points worth considering:
Erlang & Elixir’s binary parsing approach using pattern matching is very powerful. If you are having to implement different control system protocols this approach can potentially significantly reduce effort. You can see a worked example here: Binary pattern matching in Elixir with PNG parsing example (first result from a google search)
The BEAM was designed specifically to hold lots of concurrent conversations with extremely high reliability - it was built for telco switching systems. It isn’t good if 100,000 conversations get killed because of one mishap in one single conversation. Erlang/Elixir almost certainly gives you the shortest path to fault-tolerance while holding multiple “conversations” with multiple in-field devices.
While go has the big advantage of packaging pretty much everything into a single binary through the compiler & linker, mix release does an excellent job of bundling an Elixir application into a single folder for deployment.
The performance characteristics when a BEAM-based system is under load are generally quite attractive. Latency/performance tends to gradually fade as load increases, which means you have time to do something about it before it becomes a disaster. If I can recall the talk I’ll add the link! (@wanton7 mentions this in hist list)
Batteries included - caching, queuing, multi-CPU concurrency, background jobs, web-serving, API creation / consumption etc can all run off the one codebase. While Elixir applications may not deploy as a single executable, they can deploy as a very simple stack.
Good luck with your project - it sounds pretty cool! I spent a few years working for an industrial control system vendor and do think Elixir could take a lot of complexity out of SCADA systems & similar where hard real-time isn’t needed. If you do go this route, I’d love to hear how you go.
You make a lot of good points. For sure in case I would move forward with Golang I should have a complementary set of tools in order to achieve the necessary scalability, fault tolerance, etc. The reason Golang is in the evaluation process so far is the simplicity, the standard library and the very small produced executables as well the fact that the resources that a Go app needs are significantly less compared to any language that runs on top of a whole VM
Woah!! I had to take some time to digest your response. All the points you make are very helpful to me, thanks for that. My 2 main concerns when it comes to Elixir are:
The time needed for me first and my team afterwards to write code in a correct functional way
The case that the apps will need a large amount of resources in order to run because the most important thing in such projects is stability and sustainability.
I will keep searching around the points you made and ofc let you know if I would move forward with Elixir on this one. For the record I see a lot of Apache, Eclipse and LF projects in a space close to the project I am researching but it’s tough to pick cause of multiple different factors hence the first thing that came to mind would be what I could possibly build myself first and if I see that I am reinventing the wheel, only then to research further on these projects.
I tried to learn functional language called F# before Elixir. It was very hard and stopped before learning it. Tried Elixir and it was very easy to learn compared to it. Functional language is basically just functions and data and in my view Elixir is lot simpler than C# for example. Immutability takes some times to get used to, but it really makes code/data flow much easier to understand. It’s usually easy to find what is wrong by just looking at the data that is flowing from function to function. I feel like testing functional code is also much easier.
My background is that I started using C# in 2003 professionally. I think it takes about two weeks to start getting your head around how the language & platform works to start building things. If you come from non-functional language background and you are a senior programmer. I would also suggest that you don’t write any Elixir macros if you don’t absolutely need to.
On resource utilisation… bear in mind that Erlang first appeared in around 1986 when the 386 was the dominant desktop chip and RAM was measured in single digit megabytes rather than double digit gigabytes. When I’m sick of Windows &/or Mac, I pull out my trusty old Thinkpad x220 running Kubuntu (11 years old, and has the best keyboard ever put on a laptop), and everything performs fine. There are others on the forum who can talk to performance on a Raspberry Pi.
While Erlang/Elixir do run on a VM with the inefficiencies that come from that, there are some pretty smart optimisations to maximise the work not done. For example, web applications end up being super-efficient and performant due to some architectural choices in the way “strings” are processed into output on the wire - see Elixir and IO Lists, Part 1: Building Output Efficiently - Big Nerd Ranch
Yes I can believe that since according to the docs, this ecosystem is built to start and never stop as I get it. And in this regard I am interested in that aspect of course. Which means that as long as it could be able to provide this fault tolerance along with other good things others mentioned (for example background jobs/tasks) and keep the resources in a predictable manner this could be enough at least for my boss to give a thumbs up and for me to continue.
)
