Hi there,
we are implementing a client for AWS EQS queuing (that we already use in other [ruby] projects). Sqs is providing a http entpoint that is returning a list of messages. In case of an empty queue it is possible to let that call block for a couple of seconds (to save http calls and money). We would like to have one process polling sqs and handing the messages to a pool of worker, that do the real work. The number of worker should be limited to avoid sucking all sqs messages into elixir and dying an overload death.
We have a kind of hacked solution using honeydew, but there we need busy waiting loops to wait for a free worker. That we would like to avoid.
It would be really nice to have a worker pool that we can pass a message and it blocks (the caller) in case all worker are busy. I think we would be able to handcraft that, but prefer to use otp or standard libraries.
Solutions and ideas are highly appreciated. 
Cheers
Marcus
1 Like
Hi @marcuslankenau,
I would love to hear what the folks that have actually worked with pools would do in this case, but pool management is a (elegantly) solved problem in Elixir / Erlang. The most common package I’ve seen used for this is poolboy. It’s a dependency of Ecto, in case that increases your confidence in the library. I found a very quick tutorial / example on using poolboy - I don’t know how much the library has changed since that post was written, but it will certainly help you understand the intent of the library.
Does it seem to fit your need?
1 Like
Actually, instead of poolboy, this seems like something that the new GenStage module that is being added to Elixir (and currently available via the above link on hex.pm) is explicitly and specifically designed for. Hard to get more ‘standard built-in library’ than that (at least until it is added to, likely, the next Elixir version base).
EDIT: /me is really loving the GenStage module
3 Likes
@StevenXL: I was considering poolboy as well knowing it is used in ecto. But it looks like there is to much logic we have to add ourself like blocking if all workers are busy.
The GenStage looks super promising, we will try to use it. Hope it is covering the usecaser.
Thx for your help!
Marcus
I’m curious as to how it works out for you, would love a followup post on these forums.
Now that I’ve gotten used to it, it is invaluable in a few specific uses (only two for me thus far, but highly useful for flowrate control).
I’ll second @OvermindDL1’s request for a follow-up, if it’s not to much trouble. Also, @OvermindDL1 thanks for pointing out GenStage - I wasn’t aware of it.
1 Like
It was announced on the official Elixir blog not long ago: http://elixir-lang.org/blog/2016/07/14/announcing-genstage/
