I am beginning a proof of concept for an internal application to help promote Elixir. I would like to showcase the power and performance of the language platform. I have an existing messaging application using RabbitMQ. My thought were to build a publisher and consumer to connect with the existing implementation.
GenStage and Flow seem to provide value with this architecture pattern. However, as I am new to GenStage/Flow and RabbitMQ specifically, I am curious if introducing GenStage/Flow is redundant or an unnecessary abstraction.
Would you recommend using the Elixir capabilities on top of RabbitMQ? Or are the native features of RabbitMQ far superior to what could be demonstrated by GenStage/Flow?
I think that RabbitMQ is a good tool to manage back pressure, unless you have a very huge amount of data that needs to be handled really fast to the workers to not saturate RabbitMQ itself.
And also, rabbit workers, adopt a “push” pattern, so by adopting a back pressure system like GenStage, where you “pull” messages from rabbit, you are using it in a really strange way. There is already prefetch count and prefetch size from the amqp protocol that works pretty well with back pressure.
I think that a postgres table (with Repo.stream) or a big file could be a better example. A situation where the amount of data that comes in could eventually saturate your application and slow everything down.
Thanks for the explanation. I did feel like I was trying to connect the positive ends of two magnets, but thought maybe there was some justification or specific use case.
Yes. I saw this library earlier. However, I am curious if the library is necessary. Or if, as @matteosister states, does GenStage make sense with RabbitMQ.
Do you know of a more robust example of using RabbitMQ than the example tutorial on the RabbitMQ site? Seems overly simple and does not take advantage of GenServer.
