I have n number of identical rabbitmq stream consumers with prefetch count set to 1. All the messages have a tag, lets suppose user_id in this case. The messages with the same user_id tag must be processed in order they arrive in the stream.
The consumer fetches the state corresponding to the tag from ets, acquires a lock ,does processing, update the row and release the lock. Now, lets suppose consumer 1,2,3 all have messages with the same tag while consumer 2 and 3 are trying to acquire the lock in a loop and consumer 1 having the lock. How do I ensure that consumer 2 gets the lock first/message 2 is processed first after consumer 1 has released the lock.
Thanks for reading:)
My current implemetation has a single consumer consuming all the messages with the same tag, therefore order is maintained automatically, but in this case I need to have as many consumers as the number of different tags/users on the platform which doesnt scale out very well.
Without knowing too much about RabbitMQ (more like, I let it lapse and forgot it) I’d say your current implementation seems like the best and safest. Are you encountering any delays or any runtime performance struggles with it?
Hi. No, the current implementation works fine and we have achieved pretty good scale and performance in our application, which is latency critical.
But to further cut down costs, we hope to move the design from 1 consumer per queue per user to generic consumers consuming from rbmq streams.
From observation, a lot of the consumers are just sitting idle and eating up memory. This impacts performance as we try to add new users. Another queue, another consumer. Therefore we currently have more machines running than we actually need.
In that case I am doubly sorry to have forgotten RabbitMQ’s specifics.
Can’t you increase the prefetch count of just one consumer and let it consume all messages (and remove all other consumers) and have it dispatch them to where they must go?
Or indeed as you said – have one consumer per queue?
Admittedly I have no idea what led to the current design so my questions might very well be ignorant.
Nothing to be sorry about. The second solution is already implemented, the first solution will I guess converge to second solution when you want to retain unacked messages in rbmq.
This talk walks through a very similar problem statement:
Short short version: it presents an algorithm for routing RabbitMQ messages so that ones that need consistent ordering are always processed by the same consumer so (for instance) the ones with user_id = 1 are seen in submission order.
Hi Matt. In our current implementation, we have this setup already where the messages with same tag are consumed by a single consumer. However, in that case we have to have as many consumer as users on out platform.
We are trying to design a generic consumer and handle sequence using ets locks. So, can workers trying to receive lock acquire in the order of demand made for the lock.
