general-CbIC

general-CbIC

Hi :waving_hand:t2:

Poolex is an Elixir library for managing pools of workers.

In short, this is a poolboy written in Elixir.

When I started this project, I had the following goals:

  • To solve the problem of missing documentation of public interfaces and “official” library usage examples.
  • To bring this library back to life. poolboy is not actively maintained. Even if poolboy is written perfectly, there may be a chance of incompatible OTP changes in the future or the appearance of new features we’d like to use.
  • Try to rewrite this library in Elixir. It’s not a problem, but I’d like to use Elixir dependencies when I’m writing in Elixir.
  • To add the ability to use different strategies for getting a worker. I think a developer may have more needs than just choosing a LIFO / FIFO. So I added the ability to describe and use implementations for operating with worker and caller process queues.

Some project links:

Showing Posts 1 to 10

general-CbIC

general-CbIC OP

Release 0.7.0

Added FIFO implementation for getting workers from the pool. This is the same mechanism as the :fifo strategy in poolboy.

general-CbIC

general-CbIC OP

Release 0.8.0

Since the latest major release, many improvements have been made:

  • Shutting down the pool and its workers is more accurate.
  • The work has been optimized by eliminating unnecessary ETS tables.
  • Fixed many bugs with handling exit of workers and callers processes.
  • Added missed validation for some initialization options.

Also, there are some breaking changes. I will quote the release note for 0.8.0:

  • Option :timeout renamed to :checkout_timeout.

    • Reason: This option configures only the waiting time for worker from the pool, not the task’s work time. This naming should be more understandable on the call site.

      # Before
      Poolex.run(:my_awesome_pool, fn worker -> some_work(worker) end, timeout: 10_000)
      
      # After
      Poolex.run(:my_awesome_pool, fn worker -> some_work(worker) end, checkout_timeout: 10_000)
      
  • Poolex.run/3 returns tuple {:error, :checkout_timeout} instead of :all_workers_are_busy.

    • Reason: It is easier to understand the uniform format of the response from the function: {:ok, result} or {:error, reason}.
  • Poolex.caller() type replaced with struct defined in Poolex.Caller.t().

    • Reason: We need to save unique caller references.
  • Poolex.run!/3 was removed in favor of Poolex.run/3. The new unified function returns {:ok, result} or {:error, :checkout_timeout} and not handles runtime errors anymore.

    • Reason: We should not catch errors in the caller process. The caller process itself must choose how to handle exceptions and exit signals.

As always, here are some important links:

general-CbIC

general-CbIC OP

Release 0.9.0

With the last release, metrics were added to observe, analyze, and optimize the production configuration of your pools.

For now, implemented sending of pool’s size metrics: idle/busy workers count and current state “Is pool overflowed right now?”. The library starts to send these metrics via :telemetry if you pass pool_size_metrics: true on pool initialization:

children = [
  {Poolex,
    pool_id: :worker_pool,
    worker_module: SomeWorker,
    workers_count: 5,
    pool_size_metrics: true}
]

Also, I have added a plugin for PromEx library. You can check it out here.


dimitarvp

dimitarvp

Nice library, thought of writing something similar in the past. Will review. Thanks for your work! :heart:

general-CbIC

general-CbIC OP

Your kind words mean a lot to me. Thank you!

general-CbIC

general-CbIC OP

Release 0.10.0

Two new functions, add_idle_workers! and remove_idle_workers!, allow you to control the pool’s size at runtime.

For example, this is how you can add additional workers to an already running pool:

Poolex.add_idle_workers!(:my_lovely_pool, 5)

Dynamic pool size management can help you utilize resources more efficiently, depending on the workload.


Poolex package on Hex

general-CbIC

general-CbIC OP

Release 1.0.0

The new monitoring implementation no longer restricts the use of pool_id as an atom. Now you can use any valid GenServer.name() as pool_id.

PromEx plugin also bumped to v1.0.0.

Thanks to @spacebat and @al2o3cr for their discussion, this helped me realize the naming problem :pray:

darnahsan

darnahsan

Hi @general-CbIC I am facing this issue with poolboy, do you think using poolex could help

general-CbIC

general-CbIC OP

Hi @darnahsan :wave:

I am not sure about it because I haven’t tried poolex with AMQP yet. You can quickly check it since it is easy to migrate from poolboy — Migration guide.

If I understand the problem correctly, AMQP connections terminate for some reason but do not leave the pool. It’s strange because both libraries (poolboy and poolex) should monitor workers and restart them when they die.

darnahsan

darnahsan

Thanks will give poolex a try to see . I think what happens to my AMQP connections is this heartbeat=0 is negotiated but not actually respected · Issue #112 · rabbitmq/rabbitmq-erlang-client · GitHub but why polboy get the dead conections is mystery

Where Next? Top

Trending in Announcing Top

restlessronin
The repo is at GitHub - cyberchitta/openai_ex: Community maintained Elixir library for OpenAI API · GitHub. Docs are at OpenaiEx User Gu...
152 12366 136
New
type1fool
WebAuthnLiveComponent WebAuthnComponents See this post about renaming the package. Passwordless authentication for Phoenix LiveView app...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
kip
In 2021 I started a new library called Tempo with the objective of modelling time as a set of intervals - not as instants. In 2022 I gave...
New
gtcode
Aloha gang, There’s a new class of AI model out from TypeSafe. Jev is their flagship model. You can learn about it here: ...
New

Other Trending Topics Top

budgie
A little off-topic, but I feel like people here have a good head on their shoulders. I used to be quite good at making software. Was luc...
New
KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
budgie
I love Elixir. It’s one of 2 programming languages I’ve ever fallen in love with. But I don’t use it anymore. Serverless was the promis...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
Null-logic-0
What IDE or editor are you using for Elixir development? Personally, I use Zed, and I really like it, but sometimes I wish there were a ...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews