FBP-ish (Flow Based Programming) and GenServers

I would like to let you know that I am not quite sure what I am doing, I have an idea and I am trying to look for answers, sorry for any non-sense I could say.

I am trying to come out with some design for a FBP (Flow Based Programming) platform but I want some extra features on the implementation.

My vague thoughts

I want every single operation on the software to be a Node on the Graph, even multiple, sum or which ever one (still thinking why but eventually the same platform is used for program itself). Also, I want to make a Smart systems running in the connection between those Nodes, so I could scale the parallelism or take some decisions on the execution of specific program.

Right now, I am thinking on create a GenServer for every single Node (remember it is any computation) but I am not sure if I shouldn’t use GenServer for this since the latency will be involve.

Also, how I could load dynamically into the compiled code new code so the new Nodes becomes available and the software could start using it.

I need your feedbacks

I would like to know the implications of me using new processes for anything, my understand of the BEAM and ErLang is to little for understand what I am asking for (hopefully I will get there). Should I use GenServer or go the most low level implementation of dealing with processes? What could be the duplicated work I will be doing?

The Smart Graph Connections will take a look to the throughput and try to be smart about it, either by using more resources of the computer or maybe auto scale where you could do more parallel process at specific bottleneck on the execution.

Loading new graph node dynamically, the idea is that you would load specific Node implementation or replace the existing one, propagate that to other VMs when it is done correctly so they are aware of the new code without you having to do the same process in each VM.

Anyway, probably my thoughts are too high level and vague but this is something that I am trying to figure out myself what I am trying to do even, it makes sense to me on my head, I need to figure out if makes sense on the real world.

btw, essentially what I need from this project is to have a huge graph where later on I could use smart algorithms to take decisions on it and even implement software by them-self given the human instructions.

Although a graph of processes would ‘scale’, it would certainly be slower than the more usual methods in almost all cases.

First of all, instead of making a graph can you instead make a little ‘interpreter’ perhaps of simple instructions that link together (a simple list), perhaps by putting them into a named map that can call on each other and other bits for data storage? That’d be single-process but would be very fast in that single process.

And if you are just wanting to run other instructions from humans, what about embedding in erlua or so, you can still bind between different ‘scripts’ in it if you want, but having that full language gives you a decently safe way of running things from outside sources. :slight_smile:

Right now I am using the graph because essentially from data structure probably becomes easier to train a computer to take decisions around it by analyzing the possible connections between the available Nodes and the desire output.

Ideally, the computer could communicate with you asking you questions based on your desire program. If the computer definitely knows what you are looking for it will code for you the desire subgraph that executes your code, if it needs more information then I want will build machine learning with voice communicate or what have you.

I was thinking that if I just create the set of nodes that allow the computer to be turning complete from the set of instructions on the graph I will no need to add any extra nodes but maybe load some soft of language that connects nodes using those primitives.

That is why I want to go to most low level detail on how the graph work so I build everything on top of that. Essentially, that is what I would use for program the VM.

I want ErLang so I can get the OTP behaviors (specially around the recovering) , right now I wouldn’t matter to make it slow, that I could work later in how to improve it or worst scenario, you need a really powerful computer to run the software which I wouldn’t matter as well (Ideally I would like to not let this happens)

Btw, I dont want to have processes for everything because of the auto scaling thing, imagine that nobody should think about concurrency at all and the software should take all the resources available and be smart about it, probably based on the same execution the computer could become smarter about itself and know how to manage those resources.