marcuslankenau
How to use a set of worker to handle sqs messages?
Hi there,
we are implementing a client for AWS EQS queuing (that we already use in other [ruby] projects). Sqs is providing a http entpoint that is returning a list of messages. In case of an empty queue it is possible to let that call block for a couple of seconds (to save http calls and money). We would like to have one process polling sqs and handing the messages to a pool of worker, that do the real work. The number of worker should be limited to avoid sucking all sqs messages into elixir and dying an overload death.
We have a kind of hacked solution using honeydew, but there we need busy waiting loops to wait for a free worker. That we would like to avoid.
It would be really nice to have a worker pool that we can pass a message and it blocks (the caller) in case all worker are busy. I think we would be able to handcraft that, but prefer to use otp or standard libraries.
Solutions and ideas are highly appreciated. 
Cheers
Marcus
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 6 of 6 Posts
StevenXL
Hi @marcuslankenau,
I would love to hear what the folks that have actually worked with pools would do in this case, but pool management is a (elegantly) solved problem in Elixir / Erlang. The most common package I’ve seen used for this is poolboy. It’s a dependency of Ecto, in case that increases your confidence in the library. I found a very quick tutorial / example on using poolboy - I don’t know how much the library has changed since that post was written, but it will certainly help you understand the intent of the library.
Does it seem to fit your need?
OvermindDL1
Actually, instead of poolboy, this seems like something that the new
GenStagemodule that is being added to Elixir (and currently available via the above link on hex.pm) is explicitly and specifically designed for. Hard to get more ‘standard built-in library’ than that (at least until it is added to, likely, the next Elixir version base).EDIT: /me is really loving the GenStage module
marcuslankenau
@StevenXL: I was considering poolboy as well knowing it is used in ecto. But it looks like there is to much logic we have to add ourself like blocking if all workers are busy.
The GenStage looks super promising, we will try to use it. Hope it is covering the usecaser.
Thx for your help!
Marcus
OvermindDL1
I’m curious as to how it works out for you, would love a followup post on these forums.
Now that I’ve gotten used to it, it is invaluable in a few specific uses (only two for me thus far, but highly useful for flowrate control).
StevenXL
I’ll second @OvermindDL1’s request for a follow-up, if it’s not to much trouble. Also, @OvermindDL1 thanks for pointing out GenStage - I wasn’t aware of it.
OvermindDL1
It was announced on the official Elixir blog not long ago: Announcing GenStage - The Elixir programming language
