Good news, I found a workaround. It appears reducing to a single producer makes the issue go away. I’m still having trouble balancing CPU and networking but I no longer have single workers that are hoarding messages (and therefore causing them to go to the DLQ).
Is it possible that when the first worker asks the producer for demand that multiple producers respond with a single message which creates the bottleneck on a single worker if they cannot get through the message fast enough?
Has there been any traction on this? Is this enough info to submit an issue or are there other things I can try in order to run multiple producers?
Apologies, I only check the forum a couple times per week. Let’s move this to a Broadway issue and discuss there. But one issue is for sure: if you have multiple producers, the first processor would subscribe to all of them and be the first in the list to receive messages from all of them. The minimum we can do here is to shuffle the list of subscription. I have some ideas to try out, please open up an issue on Broadway and we can continue the convo there!