Socket + REST linkage

Lets say I have a running API that is updating data onto a database and I want an user connected to a specific node or machine of my network to receive a notification upon data modification. How would I go on about notifying that user?

Just so we understand what I mean; lets say an user’s data is updated and I want that user to be notified, I would go and somehow locate this user on the network and send over to that specific node a message. How does this apply to a large network with a high number of events per second (i.e. user data being updated 8,000 times per second) and 1,000 nodes?

Would the events be sent to all nodes (the 8,000 events sent to all 1,000 nodes) and then be processed whether the affected user is connected?

Let’s set aside the exact numbers you picked for a second, because a thousand node cluster is really rather astronomical.

The way pubsub systems generally work is that when an event happens (like a change in user data) that you want to push to subscribers you go to some kind of registry and you look to see if anyone is subscribed to that data. A subscription here generally represents some kind of active way to contact the subscriber. If there are subscribers, you push the data to each of them.

Based on your title you seem interested in using sockets to handle subscriptions, so the way this works is that each node tracks the sockets that are connected to it, and what each socket is subscribed to. When an event happens on node A, node A broadcasts the result to each socket on node A that is subscribed to that event (which may be nobody, in which case no work is done). Additionally, node A will publish the result to every other node in the cluster, which then performs the same operation on any sockets connected to it.

In this model, all events must be broadcasted to all nodes, because each node tracks its own set of subscribers separately.

I’m not entirely sure how the REST question fits in here. One of the issues with REST is that because it’s built around HTTP semantics REST oriented api tools don’t really translate well into sockets. If you are interested in a more unified API approach you may want to look into GraphQL.

2 Likes

Well true, those numbers were a bit of an exaggeration :smiley:

How stable of a model is this? Would the system ever reach a point where a node wouldn’t process events fast enough (as it would be receiving all events, not just the ones its connected users are interested on)? What other models are there, and which of all performs best?

I used REST more like as an example, I wanted to have something responsible for updating the data. Although, how would this go when changing data outside the elixir network - as in i.e. having my REST API running on another environment? Would I just hit a generic node upon data modification which then, again, sends an event to every other node?

Is there any specific pubsub solution(s) and/or model(s) you have deep experience with and would recommend?

As far as the BEAM is concerned, your network IO would be the issue before the BEAM would be, so look at your hardware to answer that question, not the BEAM (the BEAM is made for this). ^.^

1 Like

Hi!

This isn’t a solution, but here is a cool blog post by Discord:

Quentin

1 Like