Broadcasting a message to multiple linked processes without pid?

Hi - sorry for another beginner question…

Is it possible in Elixir to spawn e.g process1, process2, process3, and process4
Then link process1 with both process2 and process4 but not process3
Then process1 broadcast a message e.g {:greeting “hello”} without knowing pids of the linked processes, and that message is received only by the linked processes 2 & 4?

A code snippet or a link of an example demonstrate the above is deeply appreciated.

Thanks in advance
Hunar

Can you speak a bit more about what you’re trying to do at a higher level? Links are used to establish lifecycle constraints, not really to setup messaging routes or something. If you can articulate what you’re after we can suggest a more idiomatic approach.

4 Likes

Thanks for the reply.
For example, in case of neural network we do have many inter-connected nodes (neurons) each takes input from several lower nodes, do some computations and then send the result to some other higher level nodes. If Elixir allows us to dynamically link and un-link selective processes together and allow massage passing among the connected process (without bothering about pids), then this will be a nice feature and makes Elixir a very good programming language for this purpose or other similar purposes.

Hunar

As @benwilson512 noted, the ‘linking’ process that is normally used is only to ensure that two processes that depend on each other will die as a whole when one of them dies.

The linking you are referring to would just mean exchanging the PIDs and sending a message to the PIDs you know at a later time. Why would it be wrong here to ‘bother about PIDs’? You can see the neural network represented then in the same way as a social network is represented with people using phones: When I get an important message, I can (and will) send it to the people whose phone number I have. PIDs are processes’ phone numbers, nothing more, nothing less.

2 Likes

But in your phone number analogy, isn’t useful to have a utility function so that you can send a message to “All”?. In that case, you don’t need to bother remembering (i.e retrieving) all the stored numbers (pids), you just send to “All” and the utility function takes care of that.
As you said, similar to the news-feed of the social media where only those who are subscribed - friends see each others’ messages. Linking and unlinking processes in Elixir which is used to propagate :exit message along the connected processes seems to be a very useful feature if it can be adapted to propagate other messages e.g tuples across the connected processes. This level of abstraction makes programming networks much easier.

In this case, you could maybe use pg2, which is an Erlang built-in module that handles process groups, allowing processes to register under a given name, and then sending a message to one, all or some group members.

Another thing you might try, is to use something like Flow, which allows you to work with a producer-consumer-type interface directly. So after setting up the connections and specifying what each of the neurons should do, Flow will handle the inter-process passing on of messages for you.

But maybe the neurons of a neural network are not the proper level of abstraction to split it into multiple processes, since each of these neurons is probably extremely simple: If you have a system in which most of the processes cannot do a lot of work without communicating with a large number of other processes, the message-passing part itself might become more of a bottleneck than what you solve by splitting your neural network up into processes like this.

3 Likes

https://github.com/CorticalComputer/Book_NeuroevolutionThroughErlang/blob/master/Ch_6/simplest_nn.erl

shows the simplest approach that was used for the Handbook of Neuroevolution Through Erlang. Each process holds the pids of the other processes it needs to communicate within its state.

  • The sensor process sends messages to its neuron process (N_PId).
  • The neuron process expects messages from its sensor process (S_PId) and sends messages to its actuator (A_PId) process.
  • The actuator process receives messages from its neuron process (N_PId).
  • The cortex process sends a sync message to the sensor process (Sensor_PId) to initiate the sense_think_act cycle. It knows all three (Sensor_PId,Neuron_PId,Actuator_PId) so it can terminate them.
4 Likes

As @Qqwy said, that mechanism doesn’t need to be adapted to handle this. You can have process groups that comprise a set of processes for some kind of purpose; in this case which processes to send to for any given process in the network.

Using :pg2, you might register A, B, C and D in an :all_neurons group, but also register B, C and D in a {:receiver, A} group that will be used to dynamically look up where A needs to send its results, etc… What you get in this model is essentially a process group registry-based friends list for each process.

Just as a note: If you ever do choose to save PIDs and keep them around in state you should remember that PIDs are essentially process pointers and there is no guarantee that what you have actually points to a running process, so will need to also make sure you monitor each process you’re interested in so that you’ll know whenever they die and should be removed from your state. They then need to somehow notify that process when they spawn back up so they can be added. Those two events need to be asynchronous.

I personally would just go for the process group approach as it handles processes coming up and down without issue and :pg2 itself only potentially becomes an issue in a distributed system with many erlang nodes. Each time you send messages you can just get the absolute latest state of which a process’ receivers are and that’ll be that.

Edit: As an addendum to this: Any way a neuron could know it was part of a certain set of receivers is fine as it’ll be best to register it in groups in the init function of the neuron. If you truly just want to send to everyone else you can just have only the :all_neurons group and then you’ll just have the neuron ignore its own PID.

1 Like

Thanks for all the replies and useful hints. I’ll look to them and see which one is more convenient for my purpose.
Also I have came across this question in Stack overflow which looks relevant.

Another option, especially if it is a multi-directional graph, would be to use the built-in :graph module to hold the PID ‘links’ then with a wrapper module around that then you could ‘broadcast’ to all that are linked to you or vice-versa. If the graph is built once and never/rarely changed that embedding it into that module would be super fast, otherwise could probably encode it into ETS carefully as well sans the graph module).

If however it is just a simple list of send to things in this group, then pg2 or gproc (or maybe Registry) would be dead simple (I’d probably use gproc, but only because I know it best).

3 Likes

I couldn’t find :graph but found “digraph” in Erlang docs. Erlang -- digraph
It looks very interesting and I was surprised to find that Erlang has implemented graph theory in its standard library!. I wonder if it is taking advantage of the Erlangs concurrency model!

1 Like

Er, yeah, that’s it. ^.^;

And nope, it is entirely data-driven. Spreading out to concurrency is very algorithm dependent (and not free) so that depends on what you need to do.

1 Like