<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="98094" data-post-id="98094">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="PurgePJ" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/PurgePJ/120/11782_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  PurgePJ
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Honestly, not yet. I managed to lower the memory used by the entire module by reducing the amount of middles, but the main problem was, and still is, the memory / binary leak from GenServer.</p>
<p>The issue is that GenServer processes are consuming way too much memory, sadly.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="98094" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/memory-leaks-from-genserver-processes/16526/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-98094" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="98094"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="98105" data-post-id="98105">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="aseigo" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/aseigo/120/10674_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  aseigo
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>It looks like there are a number of improvements that could be made so your stages run smoother.</p>
<p>For instance, in Coxir.Struct you are using ets as an in-memory backing store and have these <code>encode/1</code> and <code>decode/1</code> functions. Those are getting used a lot and just change a Map into a list to a tuple and back… It looks like you are doing this so you can store it in an ets table which needs tuples .. but .. you could just do:</p>
<pre><code>:ets.insert @table, {id, data_as_a_map}
</code></pre>
<p>Then you will have equivalent lookups:</p>
<pre><code>[{_id, data}] = :ets.lookup @table, id
</code></pre>
<p>No moving data between different data structures constantly. Whether or not that is causing your memory usage (doubtful), it will certainly relieve some pressure on the GC and make your code faster.</p>
<p>In <code>Coxir.Stage.Middle.handle/2</code> there are many places where you are using <code>for</code> list comprehensions, but discarding the results. <code>for</code> creates a list out of every single result .. if you do not use those results, the <code>for</code> is storing them in a list for no good reason. What you probably want instead is <code>Enum.each/2</code> which just calls the function for each entry in the enumerable, IOW: for the side-effects. List comprehensions are there to build lists, not create side effects.</p>
<p>More plainly: <code>for</code> in Elixir is not equivalent to a for-loop in your typical imperative language. <code>Enum.each/2</code> is closer to loops that do not change external variables, and <code>Enum.reduce/3</code> is closer to those that do.</p>
<p>There is also interesting code in Consumer like:</p>
<pre><code>handler
|&gt; apply(:handle_event, [event, state])
</code></pre>
<p>where you could just do:</p>
<pre><code>handler.handle_event(event, state)
</code></pre>
<p>It’s equivalent and probably easier to read, esp as you can then just do:</p>
<pre><code>case handler.handle_event(event, state) do
</code></pre>
<p>which is, at least IMHO, a <em>lot</em> easier to read than the pipeline syntax used there <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> It contains lal the information needed in one line (e.g. what is the subject of the case) rather than having to track back…</p>
<p>I also noticed you have call of <code>String.to_atom/1</code> on externally supplied data .. this is a great way to exhaust the atom table (you only get so many!), run out of entries, and DoS your app as an unhappy side-effect <img src="https://forum.elixirforum.com/images/emoji/apple/confused.png?v=15" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> You should really look to replace that approach.</p>
<p>Finally .. as to where your binary “leaks” are coming from … I assume it is something like this:</p>
<ul>
<li>you get a block of json as a binary from the external service</li>
<li>Jason parses that into smaller strings for your, but each of those smaller strings are just pointers into the bigger one .. and that prevents the GC from removing those binaries until the references to the smaller strings are dropped</li>
<li>those smaller strings are being put into an ets table, so they stay forever until the entry is deleted from the ets table, which means the binaries stay forever until the entries are deleted in the ets table</li>
<li>since coxir is updating those entries over time from many different json input binaries, each entry in the ets table(s) is keeping <em>multiple</em> json binaries in memory over their lifespan.</li>
</ul>
<p>You can get around this by calling <code>:binary.copy/1</code> on the specific sub-strings you are storing in the ets table to see if that is indeed the issue.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="98105" data-batch-url="/posts/batch_likers">
                        9
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/memory-leaks-from-genserver-processes/16526/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-98105" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="98105"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-most-liked cat-most-liked" title="One of the top 3 liked posts in this thread!"></div>
  </section>
</div>
    <div class="postbit" id="98107" data-post-id="98107">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="PurgePJ" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/PurgePJ/120/11782_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  PurgePJ
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I am at work right now so I can’t test all this, but I will definitely try every single suggestion.</p>
<p>Thank you very much for such answer <img src="https://forum.elixirforum.com/images/emoji/apple/smiley.png?v=15" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="98107" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/memory-leaks-from-genserver-processes/16526/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-98107" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="98107"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="98115" data-post-id="98115">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="aseigo" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/aseigo/120/10674_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  aseigo
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>no problems; it’s really nice to see people taking on projects like yours with Elixir! best of luck and happy hacking …</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="98115" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/memory-leaks-from-genserver-processes/16526/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-98115" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="98115"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #15"></div>
  </section>
</div>
    <div class="postbit" id="98260" data-post-id="98260">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="PurgePJ" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/PurgePJ/120/11782_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  PurgePJ
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Would it be something like <a href="https://github.com/satom99/coxir/pull/25/commits/daabe4d8cb760982ff0b9d436964d322cbbd860f" rel="noopener nofollow ugc">this</a> + <a href="https://github.com/satom99/coxir/pull/25/commits/67c2f650a3e96c395e40137ec56c8218838df92f" rel="noopener nofollow ugc">this</a>?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="98260" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/memory-leaks-from-genserver-processes/16526/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-98260" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="98260"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #16"></div>
  </section>
</div>
    <div class="postbit" id="98320" data-post-id="98320">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dimitarvp" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/120/38664_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dimitarvp
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>EDIT: Ignore the <code>Enum.each</code> comments, I mistook it for <code>map</code>. My bad.</p>
<p><strong>First commit</strong>:</p>
<ul>
<li>Not sure why you use <code>Poison</code>. <code>Jason</code> is the newer library and is faster.</li>
<li>You still have not made your code use string map keys instead of atoms. You only get 1_048_476 atoms (<a href="http://erlang.org/doc/man/erl.html#+t" rel="nofollow">here is the source</a>) and they are never garbage-collected. It’s easy to execute a denial-of-service attack against Elixir code that blindly accepts all atoms thrown at it via external data. I mean this is not actually a major concern in the grand scheme of things (you will probably protect your API behind an authentication wall) but since you are trying to troubleshoot a memory consumption problem I believe it’s worth the effort to modify your code to never use atom JSON keys. I personally also believe your producer/consumer queues hold on to strings for a long time and that’s your problem but hey, why not try everything since you decided to try and tackle the problem?</li>
<li>Outside of those, I believe the commit mostly covers what <a class="mention" href="/u/aseigo" rel="nofollow">@aseigo</a> meant earlier.</li>
</ul>
<hr>
<p><strong>Second commit</strong>: if you are not interested in return value of <code>Enum.each</code> I suggest the following instead.</p>
<p>Change all these:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">Enum.each(data.guilds, fn(guild) -&gt; handle(:GUILD_UPDATE, guild) end)
</code></pre>
<p>to these:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">Stream.each(data.guilds, fn(guild) -&gt; handle(:GUILD_UPDATE, guild) end)
|&gt; Stream.run
</code></pre>
<p>Sure it’s little longer but (1) it does not create a return value and does not pass and modify it between each iteration only for it to eventually be discarded, and (2) communicates the intent of the code better: you are not interested in the results of the loop, you only invoke it for the side effects.</p>
<p>Do this everywhere you do <strong>NOT</strong> assign anything to the <code>Enum.each</code> calls.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="98320" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/memory-leaks-from-genserver-processes/16526/18">Post #17</a>
	                </div>
	            </div>
              <div id="likers-container-98320" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="98320"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #17"></div>
  </section>
</div>
    <div class="postbit" id="98358" data-post-id="98358">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dom" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dom/120/9313_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dom
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><code>Enum.each</code> is the right thing to use when you only care about side-effects, it always returns <code>:ok</code> and doesn’t accumulate a list.</p>
<p><a href="https://github.com/elixir-lang/elixir/blob/v1.7.3/lib/elixir/lib/enum.ex#L764" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/elixir-lang/elixir/blob/v1.7.3/lib/elixir/lib/enum.ex#L764</a></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="98358" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/memory-leaks-from-genserver-processes/16526/19">Post #18</a>
	                </div>
	            </div>
              <div id="likers-container-98358" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="98358"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #18"></div>
  </section>
</div>
    <div class="postbit" id="98359" data-post-id="98359">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dimitarvp" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/120/38664_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dimitarvp
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>OMG, I have been reading it as <code>map</code> all the time! Editing post.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="98359" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/memory-leaks-from-genserver-processes/16526/20">Post #19</a>
	                </div>
	            </div>
              <div id="likers-container-98359" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="98359"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>