Is there a limit to creating process transactions in Poolboy?

Questions / Help
, ,
1

Is there a limit to creating process transactions in Poolboy?

The problem:

I upload a file with several ids, for each id it creates a poolboy transaction. There may be up to 10000 processes waiting to run and the machine crash after a while.

I need a limit on those processes, I work with 150 poolboy workers.

Maybe it’s not the best way enqueue processes in poolboy, but for now I dont know a better implementation.

Thanks in advance

2

This sounds like precisely the use-case for GenStage/Flow. :slight_smile:

But in poolboy limits, I’m unsure? I really wouldn’t think so…

1 Like
3

@GusGA yeah it sounds like you want a proper queue here of some kind or another. 10_000 processes each trying to checkout a poolboy process is a lot of contention. Each process will only wait the pool_timeout amount. Poolboy is great for holding on to resources, but it isn’t a queueing system.

I would consider GenStage, Flow, or one of the persisted worker queue libraries depending on your persistence needs.

2 Likes