alexcastano
I’ve been working on a pool of remote sessions, where each session is created and closed with HTTP requests. Sessions are limited in number, so I figured, why not implement a pool? An additional requirement is that these sessions must be refreshed every 15 minutes if idle.
I decided to use a GenServer to handle the sessions. When it starts, it makes a request to create a session and stores the secret. On termination, it makes another request to close the session. It has a public API that makes an HTTP request using the session, parses the response, and sends it back to the client. And if it’s not used for 15 minutes, it’ll refresh the session itself.
To manage the pool, I’ve got a Supervisor looking after two children:
- Finch HTTP pool
- Poolboy worker pool
So when a Poolboy worker needs to make a request, it grabs a connection from the Finch HTTP pool. However, things start to go south when it’s time to shut everything down.
When the app needs to shut down, the Supervisor sends a shutdown signal to Poolboy. But it seems Poolboy isn’t that patient and doesn’t wait for its children to stop. From what I can tell, the exit/2 function works asynchronously. So, the children get the shutdown signal, but the Poolboy worker pool doesn’t stick around.
https://github.com/devinus/poolboy/blob/master/src/poolboy.erl#L285
Then the Finch pool shuts down, the Supervisor follows suit, and finally, the workers. But, here’s the twist: the workers need the Finch pool to send out a final request to close the remote session. So, they crash without closing the remote sessions.
What’s throwing me for a loop is why a library as popular as Poolboy would behave like this. Am I missing something here?
I also thought about using NimblePool, but decided against it because:
NimblePool may not be a good option to manage processes. After all, the goal of NimblePool is to avoid creating processes for resources. If you already have a process, using a process-based pool such as
poolboywill provide a better abstraction.
Any ideas on this? Maybe a different approach?
PD: I opened an issue in Poolex 
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixir-ls
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 7- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
mpope
If you need a solution before a fix is released from the libraries you could use a process that each worker links with called the
WorkerManager. This process can add each registerd worker to a set for tracking, and listen for their deaths. Once all of the processes that were linked die, then this process can terminate itsself. You can use a supervisor is has arest_for_onestrategy, then I think if you have the order:[FinchPool, WorkerManager, WorkerPool]. The WorkerPool can shutdown, the manger will wait for all the workers, then terminate once the final worker dies, and then finally the FinchPool can shutdown. This ordering should allow for a worker that is shutting down to grab a Finch connection.alexcastano
Thank you for your proposal. Last commit in Poolboy repo was more than 4 years ago, so I’m hopeless. I’ll try your solution if I cannot find a good alternative to Poolboy.
I don’t like that the worker has to know about the manager to link, but I’m sure it will eradicate the bug
mpope
Hopefully it works, its a common enough pattern described in Adopting Erlang’s Supervisor section. Might be overkill but you could checkout GitHub - inaka/worker_pool: Erlang worker pool · GitHub, too
dimitarvp
Poolex is another alternative.
cmo
pooler exists too
alexcastano
Hello, sorry for the delay. I am still working on this but don’t have a final decision.
It is very cool. I tried, but the pools must be created under the
:pooler_suptree. This is very inconvenient in our case. We want to launch the worker and the Finch pool under the same supervisor. This way both pools are started and shut down together, and in the right order.We tried poolex, and we found some minor issues that have been fixed. The maintainer is very friendly. However, I feel like it is not very used or “production ready”. Maybe, we keep using it and trying to fix all the issues, but it is not the best for project for doing this
We didn’t try worker_pool because it doesn’t have the “overflow workers” feature, which is very useful for our current situation.
Thank you for your proposals
dimitarvp
In that case I think that the Erlang
:jobslibrary could serve you because it also tries to auto-adapt depending on the observed (BEAM) system health.