marcuslankenau
How to use a set of worker to handle sqs messages?
Hi there,
we are implementing a client for AWS EQS queuing (that we already use in other [ruby] projects). Sqs is providing a http entpoint that is returning a list of messages. In case of an empty queue it is possible to let that call block for a couple of seconds (to save http calls and money). We would like to have one process polling sqs and handing the messages to a pool of worker, that do the real work. The number of worker should be limited to avoid sucking all sqs messages into elixir and dying an overload death.
We have a kind of hacked solution using honeydew, but there we need busy waiting loops to wait for a free worker. That we would like to avoid.
It would be really nice to have a worker pool that we can pass a message and it blocks (the caller) in case all worker are busy. I think we would be able to handcraft that, but prefer to use otp or standard libraries.
Solutions and ideas are highly appreciated. 
Cheers
Marcus
Most Liked
OvermindDL1
Actually, instead of poolboy, this seems like something that the new GenStage module that is being added to Elixir (and currently available via the above link on hex.pm) is explicitly and specifically designed for. Hard to get more ‘standard built-in library’ than that (at least until it is added to, likely, the next Elixir version base).
EDIT: /me is really loving the GenStage module
StevenXL
Hi @marcuslankenau,
I would love to hear what the folks that have actually worked with pools would do in this case, but pool management is a (elegantly) solved problem in Elixir / Erlang. The most common package I’ve seen used for this is poolboy. It’s a dependency of Ecto, in case that increases your confidence in the library. I found a very quick tutorial / example on using poolboy - I don’t know how much the library has changed since that post was written, but it will certainly help you understand the intent of the library.
Does it seem to fit your need?
StevenXL
I’ll second @OvermindDL1’s request for a follow-up, if it’s not to much trouble. Also, @OvermindDL1 thanks for pointing out GenStage - I wasn’t aware of it.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Move fast and fix things! Error tracking with less noise for Elixir devs who love to ship. Start your free dev account.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"