<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="151809" data-post-id="151809">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="drl123" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/drl123/120/8307_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  drl123
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><a class="mention" href="/u/axelson" rel="nofollow">@axelson</a> if this is on Heroku, from command line use <code>heroku pg:info DATABASE_URL</code> and you’ll get something like this to see how many connections are being used.  You can also see some statistics if you click on the PG addon in the Dashboard GUI for the app and go to the Diagnose tab.  There you’ll see a limited set of query stats.  There may be more you can do with the CLI under the <code>heroku pg</code> command if you check out their documentation.  Hope that helps.</p>
<p>PS…FWIW the issue I was seeing in iex above is unrelated as I found out and I don’t want to clutter your thread with details…it was a PICNIC (Problem In Chair, Not In Computer)! <img src="https://forum.elixirforum.com/images/emoji/apple/wink.png?v=15" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p></p><div class="lightbox-wrapper"><a class="lightbox" href="https://forum.elixirforum.com/uploads/default/original/3X/8/7/8791a2efaf3a3b27fa3176b42f11f2eabfe9c6d4.png" data-download-href="https://forum.elixirforum.com/uploads/default/8791a2efaf3a3b27fa3176b42f11f2eabfe9c6d4" title="07%20AM" rel="nofollow"><img src="https://forum.elixirforum.com/uploads/default/original/3X/8/7/8791a2efaf3a3b27fa3176b42f11f2eabfe9c6d4.png" alt="07%20AM" data-base62-sha1="jlitXs1tb5NA30vyP2kdpwmAQoQ" width="427" height="318"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">07%20AM</span><span class="informations">427×318 20.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><p></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="151809" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/how-can-you-avoid-overwhelming-ecto/27064/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-151809" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="151809"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="152070" data-post-id="152070">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="OvermindDL1" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/OvermindDL1/120/2677_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  OvermindDL1
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="axelson" data-post="10" data-topic="27064">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/axelson/48/26351_2.png" class="avatar"> axelson:</div>
<blockquote>
<p>How would you ask PG what it’s usage is? Or check how checked out ecto’s pool is? I don’t see an API to get that info.</p>
</blockquote>
</aside>
<p>For PG you’d just need to run a fragment select statement (or raw sql), query the system catelogs as <a class="mention" href="/u/hauleth" rel="nofollow">@hauleth</a> said.  <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>As for Ecto’s pool, I’m unsure what pool it uses but there is always a way to get the information ‘somehow’, otherwise sounds like a bug.  ^.^;</p>
<p>But still, if the DB is being overwhelmed, you should look more at the work being done instead, it likely can be quite optimized.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="152070" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/how-can-you-avoid-overwhelming-ecto/27064/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-152070" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="152070"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="152481" data-post-id="152481">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="hkrutzer" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  hkrutzer
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<blockquote>
<p>Should I be trying to provide backpressure somehow?</p>
</blockquote>
<p>Yes.</p>
<blockquote>
<p>Should I try to prioritize the client requests over the batch requests?</p>
</blockquote>
<p>You should have a separate connection pool. You can also configure a higher queue target for it, as it probably doesn’t matter if these requests take longer.</p>
<blockquote>
<p>So if I wanted to slow down my batch processing, could I control the amount of slowdown based on how overloaded ecto/the database is?</p>
</blockquote>
<p>That sounds like a lot of work. Depending on how you are doing this processing, you could have a worker pool, throttle the amount of concurrent requests on the URL using your loadbalancer, or have a job queue.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="152481" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/how-can-you-avoid-overwhelming-ecto/27064/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-152481" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="152481"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="152482" data-post-id="152482">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="josevalim" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/josevalim/120/1787_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  josevalim
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Creator of Elixir</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="axelson" data-post="1" data-topic="27064">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/axelson/48/26351_2.png" class="avatar"> axelson:</div>
<blockquote>
<p>I understand that I can tweak the <code>queue_target</code> and <code>queue_interval</code> , however I’d like to better understand what (if anything) I should be trying to do before that.</p>
</blockquote>
</aside>
<p>Correct. You should avoid tweaking <code>queue_target</code> and <code>queue_interval</code> because increasing them mostly means your users have to wait longer.</p>
<p>You should look at the logs and generally track your database times. Ecto logs the following data:</p>
<ul>
<li>
<p>query_time / db_time - how long your queries are taking. if you have slow queries, they will get all connections busy, leading to the errors above. Your priority is to reduce those (adding indexes, denormalizing the data, etc)</p>
</li>
<li>
<p>queue_time - how long you are waiting for the connection. If query times are already low but queue time is high, it is like that you have a lot of traffic and you can’t serve it all in time. A good solution here is to increase the pool size or create separate pools based on usage as mentioned by <a class="mention" href="/u/hkrutzer" rel="nofollow">@hkrutzer</a></p>
</li>
</ul> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="152482" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/how-can-you-avoid-overwhelming-ecto/27064/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-152482" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="152482"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="152550" data-post-id="152550">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="axelson" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/axelson/120/26351_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  axelson
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Scenic Core Team</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="hkrutzer" data-post="14" data-topic="27064">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/h/e0b2c6/48.png" class="avatar"> hkrutzer:</div>
<blockquote>
<p>You should have a separate connection pool. You can also configure a higher queue target for it, as it probably doesn’t matter if these requests take longer.</p>
</blockquote>
</aside>
<p>Ah, that is a good idea for the bulk requests, although it might be a little difficult passing the repo to use through the rest of the system.</p>
<aside class="quote group-livebook_core_team" data-username="josevalim" data-post="15" data-topic="27064">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/josevalim/48/1787_2.png" class="avatar"> josevalim:</div>
<blockquote>
<p>You should look at the logs and generally track your database times. Ecto logs the following data:</p>
<ul>
<li>query_time / db_time - how long your queries are taking. if you have slow queries, they will get all connections busy, leading to the errors above. Your priority is to reduce those (adding indexes, denormalizing the data, etc)</li>
<li>queue_time - how long you are waiting for the connection. If query times are already low but queue time is high, it is like that you have a lot of traffic and you can’t serve it all in time. A good solution here is to increase the pool size or create separate pools based on usage as mentioned by <a class="mention" href="/u/hkrutzer" rel="nofollow">@hkrutzer</a></li>
</ul>
</blockquote>
</aside>
<p>Thanks for the tips! I’ll take a look at those values (looks like they’re exported via telemetry as well) and then choose the approach from there.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="152550" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/how-can-you-avoid-overwhelming-ecto/27064/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-152550" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="152550"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>