<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="26394" data-post-id="26394">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="pdilyard" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/pdilyard/120/3287_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  pdilyard
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Updating on disk had already proven to be too slow (each request requires a<br>
pool of other related data, and an update to 3+ “objects”, plus there are<br>
background jobs going on constantly to re-calculate scores and things). I<br>
did a lot of benchmarking to get to the genserver implementation…The app<br>
is really a lot like a game.</p>
<p>Maybe the approach I should go for is to use genservers to route requests<br>
to the proper node, but use ETS to actually maintain the state of each<br>
object. Any thoughts on that?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26394" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-26394" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26394"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="26397" data-post-id="26397">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dom" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dom/120/9313_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dom
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Can you get recon from Hex, then try :recon.bin_leak(10) (<a href="http://ferd.github.io/recon/recon.html#bin_leak-1" class="inline-onebox" rel="noopener nofollow ugc">recon — recon v2.5.3</a>) when your system is using a lot of memory, and check 1) does usage go down a lot? and 2) what type of process shows up in the top list? (User vs Thought vs something else)</p>
<p>This will help pinpoint if GC really is the issue and where. Another possible problem would be if you receive large binary messages, then extract and keep in the state a small substring, which is really a pointer into the larger string and prevents it from being garbage collected.</p>
<p>You might find chapter 7 of Erlang in Anger useful (<a href="http://www.erlang-in-anger.com/" rel="noopener nofollow ugc">http://www.erlang-in-anger.com/</a>).</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26397" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-26397" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26397"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="26400" data-post-id="26400">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="pdilyard" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/pdilyard/120/3287_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  pdilyard
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Yes, usage went down a ton after running that function actually (~1.2gb to 120mb on my machine). Can you expand a little bit on what this does and what it means (and how I might be able to fix my problem using these results)?</p>
<p>The 10 processes listed were not my User/Thought processes, they where <code>Phoenix.Endpoint.CodeReloader</code>, <code>:ssl_manager</code>, <code>Logger</code>, and a few other <code>:gen_server</code>s that didn’t have obvious names.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26400" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-26400" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26400"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="26401" data-post-id="26401">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="pdilyard" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/pdilyard/120/3287_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  pdilyard
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Just for fun, I also called <code>:recon.bin_leak(50)</code>, and the list was full of mostly values like this:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">{#PID&lt;0.22905.0&gt;, -117,
  [current_function: {:gen_server, :loop, 6},
   initial_call: {:proc_lib, :init_p, 5}]},
</code></pre>
<p>I ran <code>GenServer.call(pid(0, 23540, 0), :get)</code>, and can confirm that these are my User processes.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26401" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-26401" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26401"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="26407" data-post-id="26407">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="imetallica" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/imetallica/120/38986_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  imetallica
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Which function have you used to reduce memory?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26407" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-26407" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26407"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #15"></div>
  </section>
</div>
    <div class="postbit" id="26408" data-post-id="26408">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="pdilyard" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/pdilyard/120/3287_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  pdilyard
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><code>:recon.bin_leak/1</code> as suggested by <a class="mention" href="/u/dom" rel="nofollow">@dom</a></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26408" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-26408" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26408"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #16"></div>
  </section>
</div>
    <div class="postbit" id="26426" data-post-id="26426">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dom" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dom/120/9313_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dom
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>bin_leaks forces a garbage collect on all processes, and measures how many reference-counted binaries were freed per process. So this confirms lack of GC is the issue here.</p>
<p>Some things you can do:</p>
<ul>
<li>If you have operations that generate lots of refc binary garbage, do them in a separate, short-lived process linked to your long-lived user process, so it doesn’t accumulate garbage.</li>
<li>You can use a timer to hibernate (see genserver doc) the user process after N seconds of inactivity, or when you know it won’t be getting messages for a while. The process will still be alive, but won’t hold extra memory.</li>
<li>You can also use a timer to force a gc every N seconds.</li>
<li>ETS as mentioned can help. Each process can own a table, it doesn’t have to be shared. This is a nice article about the difference it makes: <a href="http://theerlangelist.com/article/reducing_maximum_latency" class="inline-onebox" rel="noopener nofollow ugc">The Erlangelist - Reducing the maximum latency</a></li>
</ul> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26426" data-batch-url="/posts/batch_likers">
                        5
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/18">Post #17</a>
	                </div>
	            </div>
              <div id="likers-container-26426" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26426"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #17"></div>
  </section>
</div>
    <div class="postbit" id="26429" data-post-id="26429">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="DianaOlympos" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/DianaOlympos/120/4353_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  DianaOlympos
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Reduce the amount of memory you allocate to the BEAM. Why try to force some GC if you do not need it ?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26429" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/19">Post #18</a>
	                </div>
	            </div>
              <div id="likers-container-26429" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26429"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #18"></div>
  </section>
</div>
    <div class="postbit" id="26440" data-post-id="26440">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="aseigo" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/aseigo/120/10674_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  aseigo
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>If this is due to binaries being held due to being referenced, as opposed to just GC not running on those processes:</p>
<p>If you can identify which binaries are being kept around (sounds like messages from the user that are being parsed out, with the interesting components stored in those maps?), then consider using :binary.copy/1 on the binary snippets before storing them in your GenServer’s state. This will create a deep copy of those binaries, freeing the original binary they were pulled from, and then the GC can do its job on those original binaries.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26440" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/20">Post #19</a>
	                </div>
	            </div>
              <div id="likers-container-26440" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26440"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #19"></div>
  </section>
</div>
    <div class="postbit" id="26442" data-post-id="26442">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="NobbZ" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/NobbZ/120/27235_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  NobbZ
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>If really binarys from the bin-heap were the problem, observer wouldn’t count them to the genservers heapsize.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26442" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/21">Post #20</a>
	                </div>
	            </div>
              <div id="likers-container-26442" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26442"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #20"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/4035/load_more?page=3">Load more posts (13 remaining)</a>
</div></template></turbo-stream>