So this box will act as a beefy edge compute box. Imagine multiple different master controllers connected on the end via network switch. It’s job will be to collect data (from the registers) save some of them (was thinking Timescale or Influx for TS and Postgres for the rest) then batch process some of them (daily, weekly, monthly) but on top of all it should be able to handle the Real time aspects very well (alarms, notifications) coming from the PLCs. Just for reference, from previous experience, a normal master controller can hold easily more than 2000 data points (ofc not all of them needed). Now regarding Go, yes indeed in case I would go that road I would need to build this module by module (for modbus connectivity, saving data and all the rest) plus use docker and Kubernetes for the orchestration and fault tolerance part (which I don’t feel confident) and all the rest scripts and tools in the same manner.
Vlahunter
cmo
I dunno what industry you’re working with but we often jam much more than 2000 tags per PLC.
You’d run Docker and Kubernetes on that edge box? Have you looked at erlgo? I think someone mentioned it earlier.
For reference, I have a system that processes events (only ~600,000/day) I suck out of SCADA servers on an 8 core/8GB server and it barely touches the sides. I calculate metrics on the fly when a user accesses the page for that period and it is quick enough to calculate the metrics for a month on the fly. It makes good use of the CPU when it does though 
. All the performance issues I’ve run into so far have all been to do with database.
Another great thing about the BEAM is that you can jump on and inspect the system when things are going wrong.
Vlahunter
Yes, normally a PLC can have many more than 2000 data points but on average I could say 2000 is a good reference number. As you will know not all of them matter, only a fraction of it actually. Truth is that the bottlenecks in these systems as far as I can tell is indeed in some cases the DBs and the actual controller, it’s a good practice for example to request up to 200 registers for most controllers in case of polling, otherwise the controller can have a stroke which is a terrible scenario, now add on top of that the possibility to have 10 or 30 or more Masters. Regarding Ergo.services, I took an initial look in to it but not yet in depth. May I ask if you used Elixir/otp for the mentioned PLC projects? Should I assume your feedback/experience is positive when it comes to the ecosystem/libraries and overall system stability?
cmo
Well, yes. I only talk to SCADA though, no PLCs. I do dream of writing a SCADA system in Elixir just for fun 
. I tried out a bunch of languages but modelling a system with lots of moving parts didn’t click until I came upon the BEAM, the land of processes, message passing and supervision trees.
For data processing pipelines there is Flow, GenStage and Broadway. Modelling your connections can be done with state machines, of which you have several options. If you need to use something from another language you can write a Port (I have one in F# that calls a DLL for me). A lot of it you can write yourself using OTP.
Vlahunter
That sounds super helpful. I haven’t thought about this that far cause now I spend some time to think how to solve the first step which is to be able to connect to multiple PLCs, find a way to reload configurations dynamically in order to add/remove controllers cause I am not sure it could be possible to be fully automated in a native way. After that step I should persist, as I mentioned above and of course the final step would be to build data pipelines as you kindly suggest flow, broadway and genStage. Regarding the state_machines that you mentioned though, I am trying to understand how and why I should go that route, I mean in theory I have an expectation on specific ReadRegs for example, I should expect a specific number of values in their corresponding expected data type which means that I could potentially model it with dynamically created objects per different type of controller, am I seeing this wrong here? And would a state machine help in the modeling part? Cause in my head a state machine would be awesome for the case I would need to Write a register (send a command back to the PLC) or when I would like to keep a state in the case of an alarm for example.
cmo
Imagine you have a process for each PLC - a GenServer or a gen_statem (or some other state machine representation). Thought you might end up implementing your own controller behaviour that wraps a GenServer.
If you model the connection using a state machine you might have the states disconnected and connected. If you’re connected, you poll the PLC for updates, check what you get back is what you’re expecting and persist it, send it on or whatever. If you’re disconnected, you retry connecting every now and then.
If you want to modify the tags read from that PLC, you send a message to that PLC’s process with the new list (of maps or tuples… e.g.%{tag: "SOME_TAG", type: :dint}). If you want to add another PLC, you add it under a DynamicSupervisor.
You could keep the alarms in :ets.
Vlahunter
Woah!! i am not experienced in Elixir so i ll need to keep digging to understand exactly how these work, but if the way you describe is doable then it could potentially help in a lot of different aspects (basically all the ones i mentioned in my previous comment) .
Thanks a ton for these ideas, i have a lot of research to do and docs to read 
al2o3cr
I’ve never had cause to use it, but FWIW Erlang has a substantial SNMP implementation built-in:
Vlahunter
Never used SNMP, as i see in the docs this is protocol works over UDP. Normally (in the previous project i have been working) i would connect over TCP (Modbus TCP) and have my client read/write according to the task.
As most things Erlang, i love the fact that they have a whole huge PDF explaining this. Thanks for sharing this. I definitely need some time to consume it.
Not sure if i would be able to use it (with an initial search, i see that a given PLC should also support this protocol) but as i see the concepts in this protocol seem to be really industrial grade, also, as i see in such a Network many different HW/SW devices can be connected and managed which is crucial.
PS The more i dig in to this project the more i feel less competent, in the past i just worked my way through with simple small apps, one task at a time. Now that i have the desire to build this correctly as a prototype i see that there so many more things i was not aware of their existence.
Sebb
We are doing a very similar thing right now.
The system we have to replace is a RPi-CM based mini-server which
- serves a visualisation for a smarthome (HTTPS server, OAuth2)
- connects to various field busses (RF and cable)
- provides a Hub for those (similar to, but not as feature rich as: https://www.home-assistant.io/)
- handles some hard-keys and LEDs
- is OTA-updateable
The current system uses:
- python (Hub, some scripts)
- node (server)
- C and Cpp (bus stacks)
- .deb (updates)
- bash (some scripts)
- systemd
- crazy test system
- …more
We decided to go for Elixir/Nerves for the next generation.
@mindok has very nicely summarized the reasons for that decision, but I want to point out:
- functional language (Elixir or Erlang is the only functional language I’d consider for an embedded system. Functional languages have great advantages I won’t go into here, but the biggest plus I expect is better testability)
- we can do everything (but some fast-hard-realtime stuff) in one language. This is a big plus, especially for a small team.
- possibility to run on small platforms like iMX6UL - even without Linux, see GRiSP2
- erlang’s binary pattern matching is a game changer when implementing protocols (I did put Elixir to the test by implementing the part of the system I saw it least fit for: the KNX protocol stack. This turned out very nicely)
- Nerves is a great tool for building embedded software systems and makes OTA updates simple. (Our current desktop-OS and .deb system is a pain in the ass with thousands of servers in the field)
- OTP (Supervision tree kills systemd, we can create on (umbrella/poncho?) project for the whole system.)
- Elixir is a great language, with a very good ecosystem including a great community. There are lots of people here, that are here because they’ve already seen some other stuff.