GusGA
Is there a limit to creating process transactions in Poolboy?
Is there a limit to creating process transactions in Poolboy?
The problem:
I upload a file with several ids, for each id it creates a poolboy transaction. There may be up to 10000 processes waiting to run and the machine crash after a while.
I need a limit on those processes, I work with 150 poolboy workers.
Maybe it’s not the best way enqueue processes in poolboy, but for now I dont know a better implementation.
Thanks in advance
Most Liked
benwilson512
@GusGA yeah it sounds like you want a proper queue here of some kind or another. 10_000 processes each trying to checkout a poolboy process is a lot of contention. Each process will only wait the pool_timeout amount. Poolboy is great for holding on to resources, but it isn’t a queueing system.
I would consider GenStage, Flow, or one of the persisted worker queue libraries depending on your persistence needs.
OvermindDL1
This sounds like precisely the use-case for GenStage/Flow. 
But in poolboy limits, I’m unsure? I really wouldn’t think so…
Popular in Questions
Other popular topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"