scherrey
How to scale Phoenix Pubsub event publishing?
When using Phoenix.Pubsub.Subscribe and having a large number of subscribers to the same event (3k+), is there a single process pushing all those events out sequentially? If so, what would it take to distribute this to a pool of publisher processes which are assigned subscribers based on their hash so the process of fulfilling subscriptions is better distributed over the system cores? Has anyone done this before?
Ultimately we’re trying to significantly reduce latency at scale by allowing the scheduler to put multiple processors in action to satisfy a large number of subscriptions.
Sounds like something that can’t easily be done without altering the Phoenix pubsub code base.
Trending in Discussions
Other Trending Topics
Latest Phoenix Threads
Chat & Discussions>Discussions
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #elixirconf-us
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #performance
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First Post!
LostKobrakai
It’s not a single process, but it’s a single process per node, which dispatches the message to all the subscribers of that node (implemented using
Registry.dispatch).Registry.dispatchactually does have aparalleloption, which kicks in once registry stores registered processes on multiple ets table shards if enabled.Phoenix PubSub does also support custom dispatchers though, which e.g. phoenix channels use to fastlane messages sent to external clients. Fastlaning in this case means encoding the message only once based on the serialization format and faning out only the encoded message directly to the sockets instead of faning out to channel processes, encoding messages individually and letting channel processes forward the encoded message to their sockets.
So you might be able to enable parallel sending by
Registryby configuring a custom dispatcher.