samkit
Acquire locks in ets table row in order they were requested
I have n number of identical rabbitmq stream consumers with prefetch count set to 1. All the messages have a tag, lets suppose user_id in this case. The messages with the same user_id tag must be processed in order they arrive in the stream.
The consumer fetches the state corresponding to the tag from ets, acquires a lock ,does processing, update the row and release the lock. Now, lets suppose consumer 1,2,3 all have messages with the same tag while consumer 2 and 3 are trying to acquire the lock in a loop and consumer 1 having the lock. How do I ensure that consumer 2 gets the lock first/message 2 is processed first after consumer 1 has released the lock.
Thanks for reading:)
My current implemetation has a single consumer consuming all the messages with the same tag, therefore order is maintained automatically, but in this case I need to have as many consumers as the number of different tags/users on the platform which doesnt scale out very well.
Most Liked
al2o3cr
This talk walks through a very similar problem statement:
Short short version: it presents an algorithm for routing RabbitMQ messages so that ones that need consistent ordering are always processed by the same consumer so (for instance) the ones with user_id = 1 are seen in submission order.
Last Post!
samkit
Hi Matt. In our current implementation, we have this setup already where the messages with same tag are consumed by a single consumer. However, in that case we have to have as many consumer as users on out platform.
We are trying to design a generic consumer and handle sequence using ets locks. So, can workers trying to receive lock acquire in the order of demand made for the lock.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security









