scherrey

scherrey

How to scale Phoenix Pubsub event publishing?

When using Phoenix.Pubsub.Subscribe and having a large number of subscribers to the same event (3k+), is there a single process pushing all those events out sequentially? If so, what would it take to distribute this to a pool of publisher processes which are assigned subscribers based on their hash so the process of fulfilling subscriptions is better distributed over the system cores? Has anyone done this before?

Ultimately we’re trying to significantly reduce latency at scale by allowing the scheduler to put multiple processors in action to satisfy a large number of subscriptions.

Sounds like something that can’t easily be done without altering the Phoenix pubsub code base.

Most Liked

LostKobrakai

LostKobrakai

It’s not a single process, but it’s a single process per node, which dispatches the message to all the subscribers of that node (implemented using Registry.dispatch). Registry.dispatch actually does have a parallel option, which kicks in once registry stores registered processes on multiple ets table shards if enabled.

Phoenix PubSub does also support custom dispatchers though, which e.g. phoenix channels use to fastlane messages sent to external clients. Fastlaning in this case means encoding the message only once based on the serialization format and faning out only the encoded message directly to the sockets instead of faning out to channel processes, encoding messages individually and letting channel processes forward the encoded message to their sockets.

So you might be able to enable parallel sending by Registry by configuring a custom dispatcher.

Where Next?

Popular in Discussions Top

vans163
So useless benchmarks aside, Its possible to write a webserver that can serve 300k requests per second (perhaps more with optimizations)....
New
jeramyRR
This is an interesting article to read. Elixir’s performance, like usual, is excellent. However, it seems like the high CPU usage is co...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
Qqwy
Looking at the stacks that existing large companies have used, WhatsApp internally uses Mnesia to store the messages, while Discord uses ...
New
mmmrrr
Just saw that dhh announced https://hotwire.dev/ Is it just me or is this essentially live view? :smiley: Although I like the “iFrame-e...
New
lorenzo
Hey everone! I created a prototype for my app using Nodejs for the api. But the framework I chose wasnt great (in general theresnt any g...
New
AstonJ
Are there any Elixir or Erlang libraries that help with this? I’ve been thinking how streaming services like twitch have exploded recentl...
New
AstonJ
I’ve just started the Phoenix part of the utterly brilliant online course by @pragdave. On generating the Phoenix app he uses the --no-ec...
New
rms.mrcs
A couple of days ago I was discussing with a friend about different approaches to write microservices. He said that if he was going to w...
New
ben-pr-p
In general I’ve been sticking to this community style guide GitHub - christopheradams/elixir_style_guide: A community driven style guide ...
New

Other popular topics Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement