smanza
Hello.
I’m designing an application where multiple node have to communicate but I’m not using the Erlang distribution only raw tcp messaging. And I want to set backpressure for the incoming messages to avoid overflow of the system and be able to scale with multiple processors.
I ended up to define something like this using Broadway:
- Listener which spawns a process for each connection
- Each connection spawns a Broadway pipeline with a producer which queue the tcp messages and a given number of processors to deal with the request and send back data
This is working fine, but I’m wondering if it’s a good design to have n Broadway Pipeline or how will it be better to have a single pipeline for the entire app (listener) and provide more processors to scale.
Thanks you
Where Next?
Trending in Discussions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Bringing learning, work and compliance together since 1987.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #ai
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
cmo
Depends on the scale. If you’re expecting a million connections then you’re going to have a million * processor concurrency processors running. As far as I understand it, you define the level of concurrency in your pipeline that is ideal for your system. If you have a changing number of pipelines, then you’re providing backpressure to each connection individually but not the system as a whole. Can you pass an id or the pid of the process as part of the message and respond in
ackor something?smanza
I think you are right about the global concurrency limit and control for the upstream messages. For now I have 5 processors by requests but if the system will get 1_000 connections, this will end up With 5_000 processors instead of maybe 100 global processors .
In fact I think I did like this also so each pipeline would have a given transport and socket in its context but I guess I can pass this a message from the producer (which is a forwarder a incoming upstream messages)