Vlahunter

Vlahunter

Elixir language potential evaluation

Hi, i wanted to create a post and get some feedback regarding elixir ecosystem and a project that is on the way.
The short story
My boss assigned me to research around the ecosystem landscape in order to see if a specific ecosystem would help us create robust services in the edge. So far the only other language i am researching around is Golang

About me
I mainly use Node.js but also i use Golang for creating tools/scripts and whatever is needed that i need to build fast and light weight, i am not an Elixir devloper, i just happened to start learning some of the concepts this ecosystem has some months ago and they got me excited enough to have it in the top of the list before i start this research.

The concept
The main idea is not new, i should be able to connect to multiple different controllers (Modbus TCP/RTU), collect the data and either persist or push to a Message Queue. As i imagine, i would have multiple different server applications near the Field, communicating data in one way or another on a Centralized Server. Of course if possible some of the Data Processing could take place in the Field.
[Think of an X amount of controllers connected to a PC-Router per Physical Place]

Proposed Setup in the Field

  • Industrial Grade Router
  • Industrial Mini-PC (probably with at least 8 GB of RAM)

The Problems
Some things that need to be considered while researching on this topic are the following

  • Being able to run in a lightweight manner
  • Being able to change configurations or update parts of the app on the fly
  • Fault tolerance as much as possible considering that there are a lot of points of failure (network, legacy systems, etc)

Some Golang Findings

  • Seems that there is a maturing Ecosystem around the Edge Computing like (EdgeXFoundry)[https://www.edgexfoundry.org/]
  • Fast to Develop
  • Runs Lightweight

Lessons from the Past
In the past i have built a similar solution as a prototype that used Node.js, Mosquitto and TimeScaleDB but seems that in this setup it would be tough to scale (as far as my DevOps is explaining :stuck_out_tongue: )

Main Question
Do you think that the Elixir Ecosystem could possibly offer an advantage on a project like this ? Have you heard of similar Industrial Grade projects run on Elixir/Erlang/BEAM ? Any information would be useful so i could have a more complete picture for this Ecosystem when i will need to present it to my Boss.
Thanks in Advance

PS
This was my first post so please forgive me in case i use the wrong Category. If you need any further clarification, please let me know.

Most Liked

mindok

mindok

Just a clairification on Nerves - it is still full OTP / Elixir, but bundled for embedded systems.

A couple of other points worth considering:

  1. Erlang & Elixir’s binary parsing approach using pattern matching is very powerful. If you are having to implement different control system protocols this approach can potentially significantly reduce effort. You can see a worked example here: Binary pattern matching in Elixir with PNG parsing example (first result from a google search)
  2. The BEAM was designed specifically to hold lots of concurrent conversations with extremely high reliability - it was built for telco switching systems. It isn’t good if 100,000 conversations get killed because of one mishap in one single conversation. Erlang/Elixir almost certainly gives you the shortest path to fault-tolerance while holding multiple “conversations” with multiple in-field devices.
  3. RabbitMQ - a pretty well known message queue infrastructure - is written in Erlang (same platform as Elixir). You can take a look at the source here: GitHub - rabbitmq/rabbitmq-server: Open source RabbitMQ: core server and tier 1 (built-in) plugins · GitHub. Again - speaks to the reliability and ability to do many things at once.
  4. While go has the big advantage of packaging pretty much everything into a single binary through the compiler & linker, mix release does an excellent job of bundling an Elixir application into a single folder for deployment.
  5. The performance characteristics when a BEAM-based system is under load are generally quite attractive. Latency/performance tends to gradually fade as load increases, which means you have time to do something about it before it becomes a disaster. If I can recall the talk I’ll add the link! (@wanton7 mentions this in hist list)
  6. Batteries included - caching, queuing, multi-CPU concurrency, background jobs, web-serving, API creation / consumption etc can all run off the one codebase. While Elixir applications may not deploy as a single executable, they can deploy as a very simple stack.
  7. Serious engineers in the community. So I don’t know much about the Golang community (although I do know that there are some very talented folk in it), but the Elixir & Erlang community has, in my experience, been very open & generous, but also very thoughtful about how long-term, robust, low-hype, sustainable software should be built. It has a low presence on stack overflow relative to other languages (possibly downside when trying to convince pointy-haired bosses that it’s the right choice), but if you know where to look and make an effort to help yourself, help is never far away. The telco platform roots almost certainly have something to do with the serious side, but the Elixir ecosystem leaders such as José, Chris & @AstonJ keep it very human and welcoming.

If you haven’t watched this video yet, you should: https://www.youtube.com/watch?v=JvBT4XBdoUE - it really brings to life a few of the concepts covered in this thread and a very worthwhile investment of 42mins IMO.

Good luck with your project - it sounds pretty cool! I spent a few years working for an industrial control system vendor and do think Elixir could take a lot of complexity out of SCADA systems & similar where hard real-time isn’t needed. If you do go this route, I’d love to hear how you go.

Sebb

Sebb

We are doing a very similar thing right now.

The system we have to replace is a RPi-CM based mini-server which

  • serves a visualisation for a smarthome (HTTPS server, OAuth2)
  • connects to various field busses (RF and cable)
  • provides a Hub for those (similar to, but not as feature rich as: https://www.home-assistant.io/)
  • handles some hard-keys and LEDs
  • is OTA-updateable

The current system uses:

  • python (Hub, some scripts)
  • node (server)
  • C and Cpp (bus stacks)
  • .deb (updates)
  • bash (some scripts)
  • systemd
  • crazy test system
  • …more

We decided to go for Elixir/Nerves for the next generation.
@mindok has very nicely summarized the reasons for that decision, but I want to point out:

  • functional language (Elixir or Erlang is the only functional language I’d consider for an embedded system. Functional languages have great advantages I won’t go into here, but the biggest plus I expect is better testability)
  • we can do everything (but some fast-hard-realtime stuff) in one language. This is a big plus, especially for a small team.
  • possibility to run on small platforms like iMX6UL - even without Linux, see GRiSP2
  • erlang’s binary pattern matching is a game changer when implementing protocols (I did put Elixir to the test by implementing the part of the system I saw it least fit for: the KNX protocol stack. This turned out very nicely)
  • Nerves is a great tool for building embedded software systems and makes OTA updates simple. (Our current desktop-OS and .deb system is a pain in the ass with thousands of servers in the field)
  • OTP (Supervision tree kills systemd, we can create on (umbrella/poncho?) project for the whole system.)
  • Elixir is a great language, with a very good ecosystem including a great community. There are lots of people here, that are here because they’ve already seen some other stuff.
beepbeepbopbop

beepbeepbopbop

Go doesn’t really make any assumptions on what you need app and data structure wise, but does make a fair number of them when utilising the standard library for networking, database pools etc. It’s always been irritating that Go forces the user to write known structures, such as sets because Go strives for simplicity (when in truth, is anything but).

Anyone can walk through the excellent Go by Example and up out knowing how to write Go provided that you come from a C background, but I find that alone is insufficient in any sort of application design. While the messaging from Go is that there’s “one” way to do things, it’s hardly the case as pages such as Effective Go and CodeReviewComments · golang/go Wiki · GitHub are necessary to get some confidence in Go itself.

While Go is incredibly fast, it lacks robust in-built fault tolerance which is a deciding factor at scale. There’s an excellent video https://www.youtube.com/watch?v=40d26ZGfhR8 which illustrates the key differences between Go and Erlang; especially on what Erlang does (process isolation, in-code reloaded etc), which simply doesn’t exist in Go outside a third library. In short, the video concludes that tools such as Kubernetes smooth the edges with Go, especially around Go panics which can, if done incorrectly, can tear your entire application down abruptly. This reinforces my prior point; Go can be written quickly, but you have to be extremely defensive and write code to handle those instances, made somewhat worse by Go’s anemic error handling.

If you do decide with Go, you have to cater to the possibility of unexpected application shutdown and ingrain some sort of behaviour that kicks in when it occurs. It could be systemd, or Kubernetes, but you’re forced to acknowledge that runtime errors can propagate to application errors and have to plan accordingly. In Erlang, while this can happen, you have in-built measures in places to ensure that the application isn’t compromised in the presence of runtime errors.

Where Next?

Popular in Discussions Top

JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 53690 245
New
arpan
Hello everyone :wave: Today I am very excited to announce a project that I have been working on for almost 3 months now. The project is...
New
cvkmohan
The upcoming Phoenix 1.6 release looks very interesting. Became a habit to watch the commits - and - what they are bringing in. phx.gen...
New
praveenperera
How We Replaced React with Phoenix By: Thought Bot
New
New
PragTob
Hey everyone, this has been brewing in my head some time and it came up again while reading Adopting Elixir. GenServers, supervisors et...
New
Qqwy
I would like to spark a discussion about the static access operator: .. For whom does not know: it is used in Elixir to access fields of...
New
AstonJ
If so I (and hopefully others!) might have some tips for you :slight_smile: But first, please say which area you’re finding most challen...
New
acrolink
How does the two languages compare when it comes to server side application development? Any experiences or ideas? Thank you.
New
paulanthonywilson
I like Umbrella projects and pretty much always use them for personal Elixir stuff, especially Nerves things. But I don’t think this is ...
New

Other popular topics Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement