th3mus1cman
Is it possible to start a Broadway pipeline with a DynamicSupervisor? Well, it is possible start at least one but when I try to start multiple only the first one starts.
I tried overriding the default child_spec with
%{
id: "#{__MODULE__}:#{unique_id}",
start: {__MODULE__, :start_link, [destination: destination]},
shutdown: :infinity
}
but then none of the pipelines start and I am getting zero errors.
I overrode the default child_spec because this is the implementation:
default = %{
id: unquote(module),
start: {__MODULE__, :start_link, [arg]},
shutdown: :infinity
}
I have also defined a process_name:
@impl Broadway
def process_name({:via, module, {registry, name}}, base_name) do
{:via, module, {registry, "#{base_name}:#{name}"}}
end
This is how I am starting the pipeline:
DynamicSupervisor.start_child(
ER.DynamicSupervisor,
{ER.Destinations.Pipeline.S3, [destination: destination]}
)
def start_link(opts) do
broadway_config = get_broadway_config(opts[:destination])
context = %{
destination: broadway_config.destination
}
result =
Broadway.start_link(__MODULE__,
name: broadway_config.name,
producer: [
module:
{OffBroadwayEcto.Producer,
client: {ER.Destinations.Pipeline.Client, [destination: broadway_config.destination]}}
],
context: context,
processors: [
default: [concurrency: broadway_config.processor_concurrency]
],
batchers: [
s3: [
concurrency: broadway_config.batcher_concurrency,
batch_size: broadway_config.batch_size,
batch_timeout: broadway_config.batch_timeout
]
]
)
Anyone have any ideas what I could be doing it wrong? Or if it is possible? Thanks
Trending in Questions
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
Hello,
I know there is an approach for handling lists that allows for optimized traversal, but I can’t recall the specific method (somet...
New
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New
Documentation
While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
So my question is quite simple and i have found no conclusive answer on forum, google or AI.
Should we use :erlang.float for Integer to ...
New
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
New
If a change or preparation module uses Ash.Changeset.get_argument/2 or Ash.Query.get_argument/2 (or any of the other get_argument functio...
New
Other Trending Topics
I am happy to introduce the very α version of the new programming language compiled to BEAM.
Welcome Cure.
It has literally three kille...
New
Hi there! We created Gust: A task orchestrator inspired by Airflow.
For those who have never heard about Aiflow, it’s a Python-based wor...
New
Hi everyone!
The first release candidate for the Expert language server project is now available!
We’ve published a press release detai...
New
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming










Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
dimitarvp
Have you been able to solve this? Admittedly I can’t offer help as I haven’t worked with Broadway for a while – but I also never tried what you do.
th3mus1cman
Sadly I have not yet solved the problem. I did open an issue on the Broadway repo on Github. At this point I am thinking it is something internal to Broadway that is the issue but always open to it being my code.
If I can’t structure my pipelines this way I have been thinking about alternatives. I just don’t like anything that I have come up with yet. I have thought about trying to use a regular supervisor and not a dynamic supervisor but I just don’t know why that difference would matter. I am not an expert on the differences between the two so maybe there is something there that is causing an issue.