<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="346037" data-post-id="346037">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Asd" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Asd
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Hi, nice library, I’ve read the code and I have several questions:</p>
<ol>
<li>Why did you name the option <code>pool_id</code> instead of more common <code>name</code> for name registration?</li>
<li><code>poolboy</code> uses high priority for the pool owner genserver process. Why didn’t you implement the same here?</li>
<li>Why do you use <code>Agent</code> for monitor references storage. It seems that it is redundant and just decreases the performance of the whole pool (I am preparing a PR with change to plain map).</li>
<li>Why do you use the <code>Project.Private.Module</code> naming schema? It is the first time I see something like this. It is strange because these <code>Private</code> modules are exposed in documentation which declares that they can be used by developer (which means opposite of <code>Private</code>)</li>
<li>There’s a very bad bug with <code>monitor_caller</code> function which spawns a monitoring process for every <code>run</code> call what goes against the idea of having a pool of processes in the first places (since you end up spawning process for every call anyways). I’d suggesting moving caller monitoring to the <code>handle_call get_idle_worker</code> logic and not removing the monitor until the worker is released. Overall, following checkout/checkin pattern would solve the problem. Plus, these processes will live forever while the caller is alive. Consider some long-living process (for example very common pattern of GenServer which executes some command periodically) which calls this pool frequently. These monitoring processes would pile up until all memory is exhausted which is essentially a memory leak and will result in the whole BEAM shutting down.</li>
<li>Poolex is not handling worker start errors and whole pool will die with <code>MatchError</code> if some extra worker fails to start. It makes this pool inapplicable for usage in environments where workers connect to external services (like databases or HTTP services which are most pooling use-cases) which can be unavailable or return 422 for example.</li>
<li>Starting and stopping supervisor manually is a strange approach. I’d suggest to start one_for_all supervisor which has Poolex and Dynamic as it’s children instead of starting a Dynamic supervisor as a direct link to Poolex. This would work faster in application stop and would disable the unnecessary <code>trap_exit</code> flag</li>
</ol>
<p>Also I’d like to mention that this approach of <code>BusyWorkers</code> and <code>IdleWorkers</code> module which manage the same structure is uncommon (I personally see it for the first time) but very nice to read and it makes it really easy to follow the algorithm.</p>
<hr>
<p>I’ve found that you work in Авиасейлс and there’s a chance that you use this library in production. If you want an expert review of your other solutions, codebase and development practices to find more issues like I did just now, please leave me the message, my rates are low.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346037" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-346037" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346037"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="346085" data-post-id="346085">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="general-CbIC" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-CbIC/120/29604_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  general-CbIC
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Hi, <a class="mention" href="/u/asd" rel="nofollow">@Asd</a>!<br>
(<a class="mention" href="/u/hst337" rel="nofollow">@hst337</a> ? <img src="https://forum.elixirforum.com/images/emoji/apple/thinking.png?v=15" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20">)</p>
<p>First of all, thank you very much for reading the code and describing several problems you found! Writing a project without a code review was quite tricky, and I no longer saw the issues you wrote about.</p>
<blockquote>
<p>Why did you name the option <code>pool_id</code> instead of more common <code>name</code> for name registration?</p>
</blockquote>
<p>Initially, I used only <code>atom()</code> as the first parameter since I did not see the need to complicate it. I hadn’t thought about using Registry, and it turns out that in my years of working on Elixir, I’ve never had to use anything other than atoms. In general, when a pool was intended to have a unique atom as its identifier, <code>pool_id</code> was an appropriate name. Most likely, I made a mistake by not changing the name of this option when supporting the <code>GenServer.name()</code> type. But I don’t see any great criticality in this. I will make an alias <code>:name</code>, deprecate the old key <code>:pool_id</code>, and then slowly migrate it.</p>
<blockquote>
<p><code>poolboy</code> uses high priority for the pool owner genserver process. Why didn’t you implement the same here?</p>
</blockquote>
<p>Unfortunately, I didn’t understand what we were talking about. Can you please describe it in more detail?</p>
<blockquote>
<p>Why do you use <code>Agent</code> for monitor references storage. It seems that it is redundant and just decreases the performance of the whole pool (I am preparing a PR with change to plain map).</p>
</blockquote>
<p>You’re right! Thank you! It seems to me that there was a similar reason in one of the previous implementations because not in all cases, when working with monitoring, I had a State available. It looks like there are no more reasons, but I forgot to check this and remove the public storage for monitoring.</p>
<p>I’ll be waiting for your PR <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>There’s a very bad bug with <code>monitor_caller</code> function which spawns a monitoring process for every <code>run</code> call what goes against the idea of having a pool of processes in the first places (since you end up spawning process for every call anyways).</p>
</blockquote>
<p>These new processes are very lightweight and not linked to the caller process. In any case, we need to monitor the caller, and I don’t yet understand why a process that waits to see if the caller will crash is not suitable for this task.</p>
<blockquote>
<p>Overall, following checkout/checkin pattern would solve the problem.</p>
</blockquote>
<p>It seems that this is excessive control over the execution logic on the call side. When using a pool, you should just perform some operation on it instead of juggling worker processes. What do you think?</p>
<blockquote>
<p>Plus, these processes will live forever while the caller is alive.</p>
</blockquote>
<p>I’m afraid that’s not right. The monitoring process is always killed after the worker is released.</p>
<p>Pay attention to this line: <a href="https://github.com/general-CbIC/poolex/blob/develop/lib/poolex.ex#L182" class="inline-onebox" rel="noopener nofollow ugc">poolex/lib/poolex.ex at develop · general-CbIC/poolex · GitHub</a></p>
<p>Please tell me where I’m wrong if I don’t see something. <img src="https://forum.elixirforum.com/images/emoji/apple/pray.png?v=15" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>Poolex is not handling worker start errors and whole pool will die with <code>MatchError</code> if some extra worker fails to start. It makes this pool inapplicable for usage in environments where workers connect to external services (like databases or HTTP services which are most pooling use-cases) which can be unavailable or return 422 for example.</p>
</blockquote>
<p>This is an excellent observation. Thank you very much! I focused on carefully handling errors from an already running worker and forgot about controlling their launch.</p>
<blockquote>
<p>Starting and stopping supervisor manually is a strange approach. I’d suggest to start one_for_all supervisor which has Poolex and Dynamic as it’s children instead of starting a Dynamic supervisor as a direct link to Poolex.</p>
</blockquote>
<p>I didn’t understand <img src="https://forum.elixirforum.com/images/emoji/apple/frowning.png?v=15" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
Why is this behavior strange, and how is it conceptually different from general supervisory behavior?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346085" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-346085" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346085"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="346101" data-post-id="346101">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="general-CbIC" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-CbIC/120/29604_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  general-CbIC
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="Asd" data-post="12" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/a/c68b51/48.png" class="avatar"> Asd:</div>
<blockquote>
<p>Why do you use the <code>Project.Private.Module</code> naming schema? It is the first time I see something like this. It is strange because these <code>Private</code> modules are exposed in documentation which declares that they can be used by developer (which means opposite of <code>Private</code>)</p>
</blockquote>
</aside>
<p>I missed this question. As far as I know, Elixir does not have private modules, and there is no way to limit module accessibility. I added the <code>Private</code> space to clarify to the developer that he is doing something wrong.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346101" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-346101" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346101"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="346114" data-post-id="346114">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Asd" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Asd
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="general-CbIC" data-post="13" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-cbic/48/29604_2.png" class="avatar"> general-CbIC:</div>
<blockquote>
<p>I didn’t understand <img src="https://forum.elixirforum.com/images/emoji/apple/frowning.png?v=15" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
Why is this behavior strange, and how is it conceptually different from general supervisory behavior?</p>
</blockquote>
</aside>
<p>First, the difference in application stop. When supervisor will stop Poolex process, this process will have some time to handle some messages and then terminate (if it has a <code>trap_exit</code> flag) and only then the DynamicSupervisor will start terminating. Usually, Supervisors give some time for children to terminate and then, when some child is terminating for too long, Supervisor kills it. With spawning link to DynamicSupervisor directly, it would have no such timeout and the Supervisor of Poolex won’t even know that there’s some DynamicSupervisor which is terminating, therefore DynamicSupervisor won’t have time to terminate workers. If these workers were HTTP connections acceptors, they would just die, while they could terminate gracefully with controllable timeout.</p>
<p>Second, the difference in fault tolerance. Right now if Poolex dies, all workers die with it. However, it is possible to just restart Poolex without restarting workers if dynamic supervisor and Poolex were started under the rest_for_one supervisor. Poolex would then just initialize it’s state and monitors from <code>which_children</code> of DynamicSupervisor.</p>
<aside class="quote no-group" data-username="general-CbIC" data-post="13" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-cbic/48/29604_2.png" class="avatar"> general-CbIC:</div>
<blockquote>
<p>In any case, we need to monitor the caller</p>
</blockquote>
</aside>
<p>And you can monitor it in the Poolex process without spawning extra one. This is what this checkout/checkin pattern is about: you store an association between a caller and a worker once worker is found and when caller or worker dies, you release the alive one.</p>
<aside class="quote no-group" data-username="general-CbIC" data-post="13" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-cbic/48/29604_2.png" class="avatar"> general-CbIC:</div>
<blockquote>
<p>Unfortunately, I didn’t understand what we were talking about. Can you please describe it in more detail?</p>
</blockquote>
</aside>
<p>I was talking about <a href="https://www.erlang.org/doc/apps/erts/erlang.html#process_flag_priority" rel="nofollow">high priority</a> for the dispatcher/manager process (Poolex process in your case) so that it checks out processes faster, since it manages the queue of work on it’s own and there’s no need to keep the messages in the messagebox. I thought that <code>poolboy</code> uses it, but it does not, while a lot of other pools with similar architecture do (like <code>hackney</code>, <code>lhttpc</code>). But it would still make sense to use it in <code>Poolex</code></p>
<aside class="quote no-group" data-username="general-CbIC" data-post="13" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-cbic/48/29604_2.png" class="avatar"> general-CbIC:</div>
<blockquote>
<p>I’m afraid that’s not right. The monitoring process is always killed after the worker is released.</p>
</blockquote>
</aside>
<p>True, I am wrong, I missed this line completely.</p>
<aside class="quote no-group" data-username="general-CbIC" data-post="13" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-cbic/48/29604_2.png" class="avatar"> general-CbIC:</div>
<blockquote>
<p>(<a class="mention" href="/u/hst337" rel="nofollow">@hst337</a> ? <img src="https://forum.elixirforum.com/images/emoji/apple/thinking.png?v=15" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20">)</p>
</blockquote>
</aside>
<p>That’s another user, not me.</p>
<aside class="quote no-group" data-username="general-CbIC" data-post="14" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-cbic/48/29604_2.png" class="avatar"> general-CbIC:</div>
<blockquote>
<p>I added the <code>Private</code> space to clarify to the developer that he is doing something wrong.</p>
</blockquote>
</aside>
<p>But developer can use the <code>Private.DebugInfo</code> and <code>Private.Metrics</code> modules, right?</p>
<hr>
<p>And I also found one more bug. Consider this scenario:<br>
Caller gets a worker, sends a long-running job to the worker, then caller dies and Poolex just returns this worker to the idle queue. In this case the next caller, can receive this worker which is still executing the long-running job and this caller won’t be able to execute anything with this worker. So I have a feeling that it might make sense to just restart the worker when caller dies before releasing this worker</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346114" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-346114" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346114"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="346133" data-post-id="346133">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sodapopcan" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sodapopcan/120/34668_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sodapopcan
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="general-CbIC" data-post="13" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-cbic/48/29604_2.png" class="avatar"> general-CbIC:</div>
<blockquote>
<p>(<a class="mention" href="/u/hst337" rel="nofollow">@hst337</a> ? <img src="https://forum.elixirforum.com/images/emoji/apple/thinking.png?v=15" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20">)</p>
</blockquote>
</aside>
<p>LOL, I miss <a class="mention" href="/u/hst337" rel="nofollow">@hst337</a>.</p>
<aside class="quote no-group" data-username="general-CbIC" data-post="14" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-cbic/48/29604_2.png" class="avatar"> general-CbIC:</div>
<blockquote>
<p>I added the <code>Private</code> space to clarify to the developer that he is doing something wrong.</p>
</blockquote>
</aside>
<p>The “official” way to mark a module private is to add <code>@moduledoc false</code>, that way they don’t show up in the documentation (I can find a citation if needed).   Looking through your <code>Private</code> namespace none of the moduledocs are very involved and could just be comments.  Though I don’t think it’s a big deal either way, just pointing it out.  Earmark uses a similar approach with <a href="https://github.com/pragdave/earmark/blob/master/lib/earmark/internal.ex" rel="noopener nofollow ugc"><code>Earmark.Internal</code></a>.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346133" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-346133" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346133"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #15"></div>
  </section>
</div>
    <div class="postbit" id="346346" data-post-id="346346">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="general-CbIC" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-CbIC/120/29604_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  general-CbIC
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="Asd" data-post="15" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/a/c68b51/48.png" class="avatar"> Asd:</div>
<blockquote>
<p>First, the difference in application stop. When supervisor will stop Poolex process, this process will have some time to handle some messages and then terminate (if it has a <code>trap_exit</code> flag) and only then the DynamicSupervisor will start terminating. Usually, Supervisors give some time for children to terminate and then, when some child is terminating for too long, Supervisor kills it. With spawning link to DynamicSupervisor directly, it would have no such timeout and the Supervisor of Poolex won’t even know that there’s some DynamicSupervisor which is terminating, therefore DynamicSupervisor won’t have time to terminate workers. If these workers were HTTP connections acceptors, they would just die, while they could terminate gracefully with controllable timeout.</p>
</blockquote>
</aside>
<p>If I understand correctly, you propose changing the circuit, as shown in the figure below. It seemed that if the Poolex process receives an <code>EXIT</code> message, it starts executing the <code>terminate()</code> callback after a short time. The first step in this callback is to turn off DynamicSupervisor: <a href="https://github.com/general-CbIC/poolex/blob/develop/lib/poolex.ex#L531" class="inline-onebox" rel="noopener nofollow ugc">poolex/lib/poolex.ex at develop · general-CbIC/poolex · GitHub</a>.</p>
<p>On the Application Supervisor side, both trees are represented through a single input process: Poolex or Poolex Supervisor. I think that killing the input process will result in the same termination logic in both trees.</p>
<aside class="quote no-group" data-username="Asd" data-post="15" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/a/c68b51/48.png" class="avatar"> Asd:</div>
<blockquote>
<p>Second, the difference in fault tolerance. Right now if Poolex dies, all workers die with it. However, it is possible to just restart Poolex without restarting workers if dynamic supervisor and Poolex were started under the rest_for_one supervisor. Poolex would then just initialize it’s state and monitors from <code>which_children</code> of DynamicSupervisor.</p>
</blockquote>
</aside>
<p>I completely agree with this. However, it is unclear where you can dump the state with information about which workers are busy and which are not.</p>
<p></p><div class="lightbox-wrapper"><a class="lightbox" href="https://forum.elixirforum.com/uploads/default/original/3X/6/5/65d2ae67614bc5924734b368c791a7384bdb9355.png" data-download-href="https://forum.elixirforum.com/uploads/default/65d2ae67614bc5924734b368c791a7384bdb9355" title="image" rel="nofollow"><img src="https://forum.elixirforum.com/uploads/default/optimized/3X/6/5/65d2ae67614bc5924734b368c791a7384bdb9355_2_272x300.png" alt="image" data-base62-sha1="ewLBDtwE8NZTj66H4zchV7bdKa9" width="272" height="300" srcset="https://forum.elixirforum.com/uploads/default/optimized/3X/6/5/65d2ae67614bc5924734b368c791a7384bdb9355_2_272x300.png, https://forum.elixirforum.com/uploads/default/optimized/3X/6/5/65d2ae67614bc5924734b368c791a7384bdb9355_2_408x450.png 1.5x, https://forum.elixirforum.com/uploads/default/optimized/3X/6/5/65d2ae67614bc5924734b368c791a7384bdb9355_2_544x600.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">781×859 18.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://forum.elixirforum.com/uploads/default/original/3X/4/3/4322eedf004c5353501e54744d8b493cfd2a83a9.png" data-download-href="https://forum.elixirforum.com/uploads/default/4322eedf004c5353501e54744d8b493cfd2a83a9" title="image" rel="nofollow"><img src="https://forum.elixirforum.com/uploads/default/optimized/3X/4/3/4322eedf004c5353501e54744d8b493cfd2a83a9_2_334x300.png" alt="image" data-base62-sha1="9zUPJ7Zlgq0OlraoeeKh9dzjKxX" width="334" height="300" srcset="https://forum.elixirforum.com/uploads/default/optimized/3X/4/3/4322eedf004c5353501e54744d8b493cfd2a83a9_2_334x300.png, https://forum.elixirforum.com/uploads/default/optimized/3X/4/3/4322eedf004c5353501e54744d8b493cfd2a83a9_2_501x450.png 1.5x, https://forum.elixirforum.com/uploads/default/optimized/3X/4/3/4322eedf004c5353501e54744d8b493cfd2a83a9_2_668x600.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">948×849 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><p></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346346" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-346346" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346346"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #16"></div>
  </section>
</div>
    <div class="postbit" id="346348" data-post-id="346348">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="general-CbIC" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-CbIC/120/29604_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  general-CbIC
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="Asd" data-post="15" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/a/c68b51/48.png" class="avatar"> Asd:</div>
<blockquote>
<p>And you can monitor it in the Poolex process without spawning extra one. This is what this checkout/checkin pattern is about: you store an association between a caller and a worker once worker is found and when caller or worker dies, you release the alive one.</p>
</blockquote>
</aside>
<p>I need to think about it.</p>
<aside class="quote no-group" data-username="Asd" data-post="15" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/a/c68b51/48.png" class="avatar"> Asd:</div>
<blockquote>
<p>I was talking about <a href="https://www.erlang.org/doc/apps/erts/erlang.html#process_flag_priority" rel="nofollow">high priority</a> for the dispatcher/manager process (Poolex process in your case) so that it checks out processes faster, since it manages the queue of work on it’s own and there’s no need to keep the messages in the messagebox. I thought that <code>poolboy</code> uses it, but it does not, while a lot of other pools with similar architecture do (like <code>hackney</code>, <code>lhttpc</code>). But it would still make sense to use it in <code>Poolex</code></p>
</blockquote>
</aside>
<p>Oh, cool! I will check this out <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="Asd" data-post="15" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/a/c68b51/48.png" class="avatar"> Asd:</div>
<blockquote>
<p>But developer can use the <code>Private.DebugInfo</code> and <code>Private.Metrics</code> modules, right?</p>
</blockquote>
</aside>
<p>Yep. And I can’t do anything with it. I can only say that it is for internal library usage. I love Elixir, but unfortunately, it doesn’t have a feature to hide library interfaces.</p>
<aside class="quote no-group" data-username="Asd" data-post="15" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/a/c68b51/48.png" class="avatar"> Asd:</div>
<blockquote>
<p>And I also found one more bug. Consider this scenario:<br>
Caller gets a worker, sends a long-running job to the worker, then caller dies and Poolex just returns this worker to the idle queue. In this case the next caller, can receive this worker which is still executing the long-running job and this caller won’t be able to execute anything with this worker. So I have a feeling that it might make sense to just restart the worker when caller dies before releasing this worker</p>
</blockquote>
</aside>
<p>Thank you very much! I will add it to issues to not forget <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<hr>
<aside class="quote no-group" data-username="sodapopcan" data-post="16" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sodapopcan/48/34668_2.png" class="avatar"> sodapopcan:</div>
<blockquote>
<p>The “official” way to mark a module private is to add <code>@moduledoc false</code>, that way they don’t show up in the documentation (I can find a citation if needed). Looking through your <code>Private</code> namespace none of the moduledocs are very involved and could just be comments. Though I don’t think it’s a big deal either way, just pointing it out. Earmark uses a similar approach with <a href="https://github.com/pragdave/earmark/blob/master/lib/earmark/internal.ex" rel="noopener nofollow ugc"><code>Earmark.Internal</code></a>.</p>
</blockquote>
</aside>
<p>However, I want to document private modules and functions for “future-me” or other Poolex contributors. Hexdocs is not all because developers still get access to each module’s public interfaces even if they aren’t documented.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346348" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/18">Post #17</a>
	                </div>
	            </div>
              <div id="likers-container-346348" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346348"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #17"></div>
  </section>
</div>
    <div class="postbit" id="346351" data-post-id="346351">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="ruslandoga" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ruslandoga/120/22193_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  ruslandoga
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<blockquote>
<p>you store an association between a caller and a worker once</p>
</blockquote>
<p><img src="https://forum.elixirforum.com/images/emoji/apple/wave.png?v=15" title=":wave:" class="emoji only-emoji" alt=":wave:" loading="lazy" width="20" height="20"></p>
<p>I learned recently that it might not be necessary to store the associations in the monitoring process since we can provide tags to <a href="https://www.erlang.org/doc/apps/erts/erlang#monitor/3" rel="nofollow">:erlang.monitor</a> which would be included in the DOWN message. And since the caller gives ref and resource back on checkin, we don’t need lookups there either.</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">def checkout(pool, callback, timeout) do
  {ref, resource} = GenServer.call(pool, :out, timeout)

  try do
    callback.(resource)
  after
    GenServer.cast(pool, {:in, ref, resource})
  end
end

# assume no queue
def handle_call(:out, {_ref, pid}, state) do
  # ... queue handling and stuff ...
  [resource | resources] = state.resources
  ref = :erlang.monitor(pid, tag: {:DOWN, resource})
  {:reply, {ref, resource}, %{state | resources: resources}}
end

def handle_cast({:in, ref, resource}, state) do
  # ...
  Process.demonitor(ref, [:flush])
  {:noreply, %{state | resources: [resource | state.resources]}}
end

def handle_info({{:DOWN, resouce}, _ref, ...}, state) do
  # ...
  {:noreply, %{state | resources: [resource | state.resources]}}
end
</code></pre>
<p>Disclaimer: I didn’t read the whole thread or the Poolex code, but thought this little bit about monitors might be useful <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Sorry if it’s completely irrelevant to the discussion!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346351" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/19">Post #18</a>
	                </div>
	            </div>
              <div id="likers-container-346351" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346351"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #18"></div>
  </section>
</div>
    <div class="postbit" id="346367" data-post-id="346367">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="ruslandoga" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ruslandoga/120/22193_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  ruslandoga
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><s>And maybe cleanups like <code>GenServer.cast(pool_id, {:cancel_waiting, caller_reference})</code> can be replaced with <a href="https://www.erlang.org/doc/system/ref_man_processes#process-aliases" rel="nofollow">process aliases</a> (available since OTP-24+, Elixir 1.15+) and the caller queue can maybe be replaced with just the message queue!</s> Reading the docs a bit more carefully, it might not work. I’ll need to try out in a project tomorrow.</p>
<p>Either way, it’d be really cool if Poolex used all the modern features of the Erlang VM! Then it would be another good reason to use it over Poolboy <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346367" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/20">Post #19</a>
	                </div>
	            </div>
              <div id="likers-container-346367" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346367"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #19"></div>
  </section>
</div>
    <div class="postbit" id="346452" data-post-id="346452">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="general-CbIC" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/general-CbIC/120/29604_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  general-CbIC
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Hi, <a class="mention" href="/u/ruslandoga" rel="nofollow">@ruslandoga</a>!</p>
<aside class="quote no-group" data-username="ruslandoga" data-post="19" data-topic="54800">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ruslandoga/48/22193_2.png" class="avatar"> ruslandoga:</div>
<blockquote>
<p>I learned recently that it might not be necessary to store the associations in the monitoring process since we can provide tags to <a href="https://www.erlang.org/doc/apps/erts/erlang#monitor/3" rel="nofollow">:erlang.monitor</a> which would be included in the DOWN message. And since the caller gives ref and resource back on checkin, we don’t need lookups there either.</p>
</blockquote>
</aside>
<p>Thanks a lot for the tip! At first glance, with the help of tags, we can refuse to store monitoring links to the “types of process” we monitor (<code>worker</code> or <code>waiting caller</code>). I’ll have to check it out.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="346452" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/poolex-a-library-for-managing-pools-of-workers/54800/21">Post #20</a>
	                </div>
	            </div>
              <div id="likers-container-346452" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="346452"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #20"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/54800/load_more?page=3">Load more posts (12 remaining)</a>
</div></template></turbo-stream>