Poolex - A library for managing pools of workers

[general-CbIC]

Hi, @ruslandoga!

@ruslandoga

I learned recently that it might not be necessary to store the associations in the monitoring process since we can provide tags to :erlang.monitor which would be included in the DOWN message. And since the caller gives ref and resource back on checkin, we don’t need lookups there either.

Thanks a lot for the tip! At first glance, with the help of tags, we can refuse to store monitoring links to the “types of process” we monitor (worker or waiting caller). I’ll have to check it out.

[Asd]

Oh, and I also found some more tiny bugs.

1. Consider this situation: we have a pool, all workers are busy and there is a queue of waiting callers. Then pool receives a call to create more idle workers, but these workers won’t be returned to waiting callers.

2. And there’s also this nasty bug which is very rare in normal conditions but quite possible in extreme load scenario. Caller sits in get_idle_worker, then timeout expires, caller sends the cancel_waiting and before Poolex receives the cancel message, some worker becomes available and gets sent to this waiting caller. Then Poolex will handle the cancel_waiting message and we’ll have a worker which is considered busy, but caller may never use it and this worker will be busy until caller dies

[general-CbIC]

Release 1.1.0

Several small improvements have been made that were mentioned above. Thank you very much for your comments!

1. The unnecessary Agent process has been removed from the monitoring logic.
2. Poolex.get_state/1 deprecated in favor of :sys.get_state/1.
3. Poolex processes now have higher priority.

Also added adobe/elixir-styler to CI and made a few minor refactoring changes.

---

• Docs
• Package on hex.pm
• GitHub

[general-CbIC]

Release 1.2.0

And a few more improvements!

These are mainly related to library maintenance improvements:

• added Elixir version 1.18 on CI;
• tests rewritten with new ExUnit parameterization feature;
• tests no longer throw many expected errors into the console when executed.

However, there has also been a slight improvement in the library’s use. Now, during initialization, there is no need to explicitly specify pool_id since, judging by user feedback, it often coincides with worker_module. Therefore, you can not set pool_id; by default, it will have a value equal to worker_module.

Example:

Poolex.start_link(worker_module: MyLovelyWorker, workers_count: 10)
Poolex.run(MyLovelyWorker, fn pid -> do_something(pid) end)

---

• Docs
• GitHub

[general-CbIC]

Release 1.2.1

Fixed some small bugs:

• The busy_worker will be restarted after the caller's death. Read more in issue.
• The function add_idle_workers!/2 now provides new workers to waiting callers if they exist. Read more in issue.

Attention

I dragged these bugs ahead to stabilize the library’s work. However, in the next release, I plan several breaking changes:

• Hide function get_debug_info/1 (issue).
• Bump the minimum required Elixir version to 1.17 to align the code with the latest Styler recommendations.

If you need to force some updates before the bump of the minimum Elixir version, write, and we will discuss it.

[general-CbIC]

I decided to solve this issue before making any breaking changes.

[general-CbIC]

Release 1.3.0

So I’ve published a new version of the Poolex with fixes for the problem mentioned above.
Now the pool will not crash when some workers fail for any reason. Poolex will start with healthy workers and will try to restart failed workers.

You can use a new option failed_workers_retry_interval on pool initialization to configure the interval between retry start attempts.

Some other little improvements were also made; see the changelog for more info.

---

• 1.3.0 changelog
• Docs