kwando
Broadway; consume multiple queues in the same pipeline
I have built a Broadway pipeline with a custom producer that reads from a single queue but I want to start broadway with 3 producers where each one reads from a separate queue. Is there any way to accomplish this?
Most Liked
josevalim
You can start three different topologies but use the same callback module for each of them. Something like
defmodule MyBroadway do
use Broadway
def start_link(queue) do
Broadway.start_link(__MODULE__, ...., queue_name: queue)
end
end
and then in your supervision tree:
children = [
{MyBroadway, "queue-1"},
{MyBroadway, "queue-2"},
{MyBroadway, "queue-3"}
]
If you want them to be consumed precisely from the same topology (let’s say because you need to share batchers), then that’s a feature that yo need to add to your producer.
rogerweb
Jose’s example is a simplification. You have to give a different id for each supervised process (like described in the error message) and a different name when calling Broadway’s start_link/2. You can do both at the same time by defining a child_spec function in the module you use Broadway.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #hex









