Elixir language potential evaluation

If your machine specs are 8GB+ of RAM what’s the concern with system resources? Is that box specifically for this task or will it be running a SCADA client or something as well?

How do you fulfill the robust requirement with Go?

Sounds fun, good luck!

1 Like

So this box will act as a beefy edge compute box. Imagine multiple different master controllers connected on the end via network switch. It’s job will be to collect data (from the registers) save some of them (was thinking Timescale or Influx for TS and Postgres for the rest) then batch process some of them (daily, weekly, monthly) but on top of all it should be able to handle the Real time aspects very well (alarms, notifications) coming from the PLCs. Just for reference, from previous experience, a normal master controller can hold easily more than 2000 data points (ofc not all of them needed). Now regarding Go, yes indeed in case I would go that road I would need to build this module by module (for modbus connectivity, saving data and all the rest) plus use docker and Kubernetes for the orchestration and fault tolerance part (which I don’t feel confident) and all the rest scripts and tools in the same manner.

I dunno what industry you’re working with but we often jam much more than 2000 tags per PLC.

You’d run Docker and Kubernetes on that edge box? Have you looked at erlgo? I think someone mentioned it earlier.

For reference, I have a system that processes events (only ~600,000/day) I suck out of SCADA servers on an 8 core/8GB server and it barely touches the sides. I calculate metrics on the fly when a user accesses the page for that period and it is quick enough to calculate the metrics for a month on the fly. It makes good use of the CPU when it does though :smile:. All the performance issues I’ve run into so far have all been to do with database.

Another great thing about the BEAM is that you can jump on and inspect the system when things are going wrong.

2 Likes

Yes, normally a PLC can have many more than 2000 data points but on average I could say 2000 is a good reference number. As you will know not all of them matter, only a fraction of it actually. Truth is that the bottlenecks in these systems as far as I can tell is indeed in some cases the DBs and the actual controller, it’s a good practice for example to request up to 200 registers for most controllers in case of polling, otherwise the controller can have a stroke which is a terrible scenario, now add on top of that the possibility to have 10 or 30 or more Masters. Regarding Ergo.services, I took an initial look in to it but not yet in depth. May I ask if you used Elixir/otp for the mentioned PLC projects? Should I assume your feedback/experience is positive when it comes to the ecosystem/libraries and overall system stability?

Well, yes. I only talk to SCADA though, no PLCs. I do dream of writing a SCADA system in Elixir just for fun :slight_smile:. I tried out a bunch of languages but modelling a system with lots of moving parts didn’t click until I came upon the BEAM, the land of processes, message passing and supervision trees.

For data processing pipelines there is Flow, GenStage and Broadway. Modelling your connections can be done with state machines, of which you have several options. If you need to use something from another language you can write a Port (I have one in F# that calls a DLL for me). A lot of it you can write yourself using OTP.

That sounds super helpful. I haven’t thought about this that far cause now I spend some time to think how to solve the first step which is to be able to connect to multiple PLCs, find a way to reload configurations dynamically in order to add/remove controllers cause I am not sure it could be possible to be fully automated in a native way. After that step I should persist, as I mentioned above and of course the final step would be to build data pipelines as you kindly suggest flow, broadway and genStage. Regarding the state_machines that you mentioned though, I am trying to understand how and why I should go that route, I mean in theory I have an expectation on specific ReadRegs for example, I should expect a specific number of values in their corresponding expected data type which means that I could potentially model it with dynamically created objects per different type of controller, am I seeing this wrong here? And would a state machine help in the modeling part? Cause in my head a state machine would be awesome for the case I would need to Write a register (send a command back to the PLC) or when I would like to keep a state in the case of an alarm for example.

Imagine you have a process for each PLC - a GenServer or a gen_statem (or some other state machine representation). Thought you might end up implementing your own controller behaviour that wraps a GenServer.

If you model the connection using a state machine you might have the states disconnected and connected. If you’re connected, you poll the PLC for updates, check what you get back is what you’re expecting and persist it, send it on or whatever. If you’re disconnected, you retry connecting every now and then.

If you want to modify the tags read from that PLC, you send a message to that PLC’s process with the new list (of maps or tuples… e.g.%{tag: "SOME_TAG", type: :dint}). If you want to add another PLC, you add it under a DynamicSupervisor.

You could keep the alarms in :ets.

Woah!! i am not experienced in Elixir so i ll need to keep digging to understand exactly how these work, but if the way you describe is doable then it could potentially help in a lot of different aspects (basically all the ones i mentioned in my previous comment) .

Thanks a ton for these ideas, i have a lot of research to do and docs to read :stuck_out_tongue:

I’ve never had cause to use it, but FWIW Erlang has a substantial SNMP implementation built-in:

https://www.erlang.org/doc/apps/snmp/users_guide.html

Never used SNMP, as i see in the docs this is protocol works over UDP. Normally (in the previous project i have been working) i would connect over TCP (Modbus TCP) and have my client read/write according to the task.

As most things Erlang, i love the fact that they have a whole huge PDF explaining this. Thanks for sharing this. I definitely need some time to consume it.

Not sure if i would be able to use it (with an initial search, i see that a given PLC should also support this protocol) but as i see the concepts in this protocol seem to be really industrial grade, also, as i see in such a Network many different HW/SW devices can be connected and managed which is crucial.

PS The more i dig in to this project the more i feel less competent, in the past i just worked my way through with simple small apps, one task at a time. Now that i have the desire to build this correctly as a prototype i see that there so many more things i was not aware of their existence.

We are doing a very similar thing right now.

The system we have to replace is a RPi-CM based mini-server which

  • serves a visualisation for a smarthome (HTTPS server, OAuth2)
  • connects to various field busses (RF and cable)
  • provides a Hub for those (similar to, but not as feature rich as: https://www.home-assistant.io/)
  • handles some hard-keys and LEDs
  • is OTA-updateable

The current system uses:

  • python (Hub, some scripts)
  • node (server)
  • C and Cpp (bus stacks)
  • .deb (updates)
  • bash (some scripts)
  • systemd
  • crazy test system
  • …more

We decided to go for Elixir/Nerves for the next generation.
@mindok has very nicely summarized the reasons for that decision, but I want to point out:

  • functional language (Elixir or Erlang is the only functional language I’d consider for an embedded system. Functional languages have great advantages I won’t go into here, but the biggest plus I expect is better testability)
  • we can do everything (but some fast-hard-realtime stuff) in one language. This is a big plus, especially for a small team.
  • possibility to run on small platforms like iMX6UL - even without Linux, see GRiSP2
  • erlang’s binary pattern matching is a game changer when implementing protocols (I did put Elixir to the test by implementing the part of the system I saw it least fit for: the KNX protocol stack. This turned out very nicely)
  • Nerves is a great tool for building embedded software systems and makes OTA updates simple. (Our current desktop-OS and .deb system is a pain in the ass with thousands of servers in the field)
  • OTP (Supervision tree kills systemd, we can create on (umbrella/poncho?) project for the whole system.)
  • Elixir is a great language, with a very good ecosystem including a great community. There are lots of people here, that are here because they’ve already seen some other stuff.
5 Likes

Very detailed explanation. Seems that the existing system is complex enough so i suppose you have researched enough to decide to go with Elixir. Of course, as i see in your case you need to work in smaller devices (hence nerves and Grisp).

So if i may ask and from reading your comment, do i understand correctly that you want to replace almost all the current system’s functionalities to run in the BEAM? And also, if we are talking about a low resources device, are you confident it could run cold ? (meaning from the point of view of resources).

Last but not least, since you are researching in a similar field, have you found any efforts from the Elixir community that could help someone advance faster? I mean if you have found some guidelines, system design techniques using the OTP and leveraging the BEAM? Or is this specific field a bit of uncharted for the moment being from the Ecosystem’s side ?

I believe SNMP may only be popular in certain industries - the BEAM implementation is (presumably) because Ericsson used/uses it, and I’ve heard about major cable-TV operations controlling set-top boxes remotely with it.

1 Like

Do you think Elixir memory usage would be a problem? If you are worried about that then to my understanding Go usually uses more memory.

I really have to conduct tests for that. For a simple system that only has READ functionality and then handles the data (persist, send to MQ, etc) although i have not experience in Elixir/BEAM, i do not believe the RAM would not be sufficient.

As i think it so far, the data must be handled in batches and asynchronously and live for a short time, hence my belief for low RAM usage in this part. When i was working in the past with Node.js this part was very low consuming, it was the handling from the TS DB that would raise Errors, and i should raise the V8 memory allowance.

My concerns about the memory have to do with the fact that if the prototype would be successful, i would have to build a lot more apps under BEAM. Tasks, data handling, network communications, if possible (not even sure if it is) some kind of stream pre-processing so there lies the reason for feeling insecure regarding this.

On the other hand as we discussed above, in case of Go (not Ergo) i should feel a lot more confident with 8GBs of RAM and 4 cores but still, i should build almost everything myself, plus one more DevOps to help me on the fault-tolerance and orchestration part.

Yes I did. (Go was also on the list). I do not see any system out there coming even close to Elixir/OTP/Nerves. Only drawback is: small community, dynamically typed (but there is sth gleaming at the end of the tunnel).

Yes, it will be a complete rewrite. But as I said, fast-hard-realtime stuff has to be in C. But Elixir is not the problem here, but Linux. So it also has to be on a separate microcontroller. (Which we could drop when using Erlang under RTEMS, then we could most likely also drop the the C code and use Erlang, not sure).

I do not know what “run cold” means.
The IMX6ULL will have enough resources to handle our requirements.
The server just has to handle the smart-home traffic of one house. That is about 1 frame/sec. Most of it is KNX which has 9600bps.

I read Elixir in Action, which is a great book and a great starting point if you have some experience. Then I started to implement the KNX stack - and failed. But I was put back on track by this great response from Sasa Juric: The GenServer from hell (needs some refactoring) - #8 by sasajuric

Core point is, that when using a functional language you have to structure your code and data(!) differently, here I followed the advise to implement the functional core, imperative shell pattern.

When coming from JS, you’ll (like me) have some trouble adjusting to functional programming. But I’m sure its worth it.

2 Likes

In BEAM if you plan to model your system using GenServers, they support hibernation. Where they can be minimized to memory after idling specified time. Also if you create a GenServer to process some data then exit all memory that GenServer used is freed. But Go for example has global garbage collector and can’t do this.

You could also use ZRAM in your system to “increase” available memory. If you set ZRAM to compress 50% (4GB) of memory you could get about 16-20GB out of 8GB RAM depending on compression algorithm used.

2 Likes

How many would that be?
Modbus over RS-xxx has 38400bps afaik. You could connect loads of those to a 4 Core machine.

Thanks for clarifications. Yes i am also coming from JS so some simple stuff for functional programmers are a bit more tough to understand and really apply them in practice.

Run cold meaning not running in high % of cpu/ram, sorry this is how i personally call it :stuck_out_tongue: .

I think that your case as well described is a bit more tough but i see now completely the reasons that you want to follow the Elixir/OTP/Nerves way.

No, the thing will be bored (and cold).