oliveiragahenrique

oliveiragahenrique

How to model a queue and consumer with GenServer?

I’m building a system which must have a queue and a kind of consumer, that will consume the events of the queue and remove them only if succeeds.

I’m thinking about model the queue and the consumer as different gen servers:

Queue process: Handle insert, get first event and delete operations
Consumer process: Handle consume operation (series of steps over the event data, which may fail)

Both processes have a 1:1 relationship, which means a specific queue has only one consumer and the consumer only consumes from one queue.

The consumer will call it self periodically, requesting to consume an event! This processes consist in a series of calls to the queue process, ultimately removing the event from the queue.

I’m currently thinking about some decisions:

1 - Would be useful to separate the registration of both process under two different Registry?
2 - Since consumer do not make sense without a queue, and the consumer can crash more easily, I was thinking if the Queue process could supervise the consumer process, thoughts on this?

How would you guys solve it, have some repository which implements something similar?

First Post!

kokolegorille

kokolegorille

Producer, Consumer… => GenStage

And other related tools, like Flow and Broadway. in case of error, it could be pushed into an error queue.

It’s nice to build something like this, but GenStage allow creation of pipeline of Producer, Consumer/Producer, Consumer with back pressure support.

Where Next?

Popular in Discussions Top

ricklove
I was just introduced to Elixir and Phoenix. I was told about the 2 million websocket test that was done 2 years ago. From my research, t...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
lorenzo
Hey everone! I created a prototype for my app using Nodejs for the api. But the framework I chose wasnt great (in general theresnt any g...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 40165 209
New
RudManusachi
What configs will make sense to put to runtime.exs? – A bit of how I configure apps: I have generic configs in config/config.exs, dev...
New
nunobernardes99
Hi there Elixir friends :vulcan_salute: In a recent task I was on, I needed to check in two dates which of them is the maximum and which...
New
fireproofsocks
I’ve been working on an Elixir project that has required a lot of scripting. I usually reach for Elixir because I like it more (and in th...
New

Other popular topics Top

nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement