GusGA
Is there a limit to creating process transactions in Poolboy?
The problem:
I upload a file with several ids, for each id it creates a poolboy transaction. There may be up to 10000 processes waiting to run and the machine crash after a while.
I need a limit on those processes, I work with 150 poolboy workers.
Maybe it’s not the best way enqueue processes in poolboy, but for now I dont know a better implementation.
Thanks in advance
Where Next?
Trending in Questions
Other Trending Topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
OvermindDL1
This sounds like precisely the use-case for GenStage/Flow.
But in poolboy limits, I’m unsure? I really wouldn’t think so…
benwilson512
@GusGA yeah it sounds like you want a proper queue here of some kind or another. 10_000 processes each trying to checkout a poolboy process is a lot of contention. Each process will only wait the pool_timeout amount. Poolboy is great for holding on to resources, but it isn’t a queueing system.
I would consider GenStage, Flow, or one of the persisted worker queue libraries depending on your persistence needs.