krasenyp
Hi everyone. I have an interesting case at work. I’m developing a service which ingest data from an external data provider via an exclusive AMQP queue. Since it’s an exclusive queue, when the consumer disconnects, the queue gets destroyed and having more than one queues means I have to de-duplicate messages. My task is to minimize the chance of missing messages in case of flaky connection or if something happens to the container the service is running in.
By using supervision, I can minimize the missed messages when the connection is flaky but how to handle containers crashing? I’ve identified two ways to tackle the problem both of them presume a cluster of at least two instances of the services. One approach is to register the AMQP connection process as a global process by using :global or horde. The second approach consists of creating a process group with :pg and synchronize the connection and channel where the queue is declared. I couldn’t manage to make it working reliably when nodes are started at the same time.
Are there any alternatives and am I missing something? What approach would you advise me to use? Thanks in advance.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 4- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
benwilson512
Ideally, the external data provider needs to require that you ACKnowledge data that you receive. This way if the connection crashes, they can re-send data that they dont’ have acknowledged. This requires some minor de-duplification on your end but it also guarantees that you don’t miss messages.
krasenyp
The problem is the queue is exclusive and when the service reconnects, a new query is created and all the messages between the lost connection and the reconnect are lost.
benwilson512
This is an intrinsically fault intolerant design. This cannot be made to not drop messages without changing something about that. You can try to minimize faults by keeping things online as much as possible, but the very idea of “fault tolerance” is that a fault can happen, but things can recover.
krasenyp
For the sake of continuity, I managed to minimize the possibility of losing messages with a two-node cluster, using
hordeto manage a global AMQP connection. If a connection dies, it gets restarted, and if a node dies, the connection is restarted on the other node. It’s not perfect but it’s better than nothing.