<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="63254" data-post-id="63254">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="vertti" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/vertti/120/7509_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  vertti
                      <span class="op-star" title="Thread Starter">
                        <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                      </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Yes, I meant one (or splitted into two) <code>GenServer</code> per topic too. You’re right, load testing is the only way to get any solid idea about the bottlenecks, and this will not be hard to test.</p>
<p>I will start building a <code>ETS</code>-<code>GenServer</code> solution for topic state management. I had no idea transactions were such a hassle and that there are so many things to take into consideration. Having a <code>GenServer</code> serialize writes, event at a performance penalty, makes sense to me now as the reads can still use the efficient ETS directly.</p>
<p>I’ve been looking into <code>GenStage</code> but don’t understand it well yet.</p>
<p>Thanks for all the insightful advice!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="63254" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/channels-where-to-store-state/11076/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-63254" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="63254"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="63653" data-post-id="63653">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="vertti" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/vertti/120/7509_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  vertti
                      <span class="op-star" title="Thread Starter">
                        <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                      </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><a class="mention" href="/u/sasajuric" rel="nofollow">@sasajuric</a> I’ve been looking into your con_cache, it looks very solid and I’ll likely end up using that over <code>Cachex</code> or <code>mnesia</code>.</p>
<p>I’d like to try storing the keyword list in <code>con_cache</code>, utilizing the janitor process that checks for expired timestamps. However, my data will be in the format of <code>{keyword, timestamp}</code>. So the value of each key in <code>con_cache</code> would be something like:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">[
 {"Politics", #DateTime&lt;2017-12-30 12:08:18.598684Z&gt;},
 {"Football", #DateTime&lt;2017-12-30 12:08:18.598708Z&gt;}
]
</code></pre>
<p>As your <code>con_cache</code> checks for expiry only for each key, not within dynamic lists, I was wondering if it’s possible to somehow extend the janitor process to have a custom <code>Enum</code> function to iterate through lists and check for timestamps there, then invoke another custom function for removal if it is past the given timestamp?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="63653" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/channels-where-to-store-state/11076/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-63653" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="63653"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="63660" data-post-id="63660">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sasajuric" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sasajuric/120/991_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sasajuric
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Elixir In Action</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I’m a bit confused. Do you want to remove just one part of the key, for example “politics”, or do you want to determine the expiry time of the entire element based on the entries in the key list?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="63660" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/channels-where-to-store-state/11076/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-63660" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="63660"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="63661" data-post-id="63661">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="vertti" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/vertti/120/7509_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  vertti
                      <span class="op-star" title="Thread Starter">
                        <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                      </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Yes, I want to iterate through those tuples, and only remove the tuple whose timestamp is past the expiry timestamp. I think <code>Enum</code> creates a new list, which I could then insert as the new value for the key. I would only need to iterate through them once every 30 seconds or so.</p>
<p>So say, <code>[{"politics", timestamp}, ...]</code> tuple’s timestamp is past expiry upon check → remove that tuple from the list.</p>
<p>Does it make sense to write a separate janitor process that goes through the <code>con_cache</code> periodically, going through each value (list of tuples) checking for expiry?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="63661" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/channels-where-to-store-state/11076/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-63661" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="63661"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="63662" data-post-id="63662">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="mgwidmann" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/mgwidmann/120/7249_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  mgwidmann
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>In case you haven’t read it, this blog post by Discord may help you with scaling out as it looks like you’ll encounter some of the same issues they did.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://discord.com/blog">
  <header class="source">
      <img src="https://cdn.prod.website-files.com/5f8dd67f8fdd6f51f0b50904/60ae916347747e71167e21cc_favicon.png" class="site-icon" alt="" width="32" height="32">

      <a href="https://discord.com/blog" target="_blank" rel="noopener nofollow ugc">discord.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://cdn.prod.website-files.com/6257adef93867e50d84d30e2/665643dd8c7ac752237b5cef_Discord-OG-1200x630.jpg" class="thumbnail" alt="" width="690" height="362"></div>

<h3><a href="https://discord.com/blog" target="_blank" rel="noopener nofollow ugc">Discord Blog</a></h3>

  <p>Discover the latest news and app updates, stories, company insights, and plenty more about Discord, all in one place.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="63662" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/channels-where-to-store-state/11076/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-63662" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="63662"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #15"></div>
  </section>
</div>
    <div class="postbit" id="63668" data-post-id="63668">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sasajuric" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sasajuric/120/991_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sasajuric
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Elixir In Action</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>What you’re describing here is a composite operation. When you change the value of the key (for example, by removing one element from its list), you’re effectively removing an item and inserting a new one, under a different key. As far as I know, ETS doesn’t support this operation atomically. Functions such as <code>:ets.update_element</code> will fail if you try to modify a key.</p>
<p>Currently, I don’t think key modifications should be supported by <code>con_cache</code>. However, I think you have all the pieces to make this happen.</p>
<p>First, you can set TTL per each row (see <a href="https://hexdocs.pm/con_cache/ConCache.Item.html" rel="noopener nofollow ugc">here</a>). Therefore, when storing a new value, you could find the nearest future time, compute TTL, and set the expiry for that item. When starting the cache, you would provide the <code>:callback</code> option to <code>ConCache.start_link</code> (see <a href="https://hexdocs.pm/con_cache/ConCache.html#start_link/2" rel="noopener nofollow ugc">here</a>). Then, before the item is deleted, your callback will be invoked. In this callback, you can fetch the current value, modify the key, compute the next TTL, and insert the new key-value with this TTL.</p>
<p>However, I’m a bit puzzled with your cache organization. Why do you have such complex keys? How are you looking things up in such table?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="63668" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/channels-where-to-store-state/11076/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-63668" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="63668"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #16"></div>
  </section>
</div>
    <div class="postbit" id="63888" data-post-id="63888">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="vertti" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/vertti/120/7509_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  vertti
                      <span class="op-star" title="Thread Starter">
                        <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                      </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Yes actually, I was thinking about simply fetching the current key (the whole list), and then updating it with by basically replacing it with a totally new list as the value that holds the new state (keywords).</p>
<p>Thanks for the instructions! I was also thinking about a callback approach but didn’t quite understand how to work it out. Just so I got this right;</p>
<ul>
<li>The callback itself would insert the new computed key</li>
<li>The delete function would execute before the callback function inserts the new key-value, thus not deleting the newly computed &amp; inserted key-value itself</li>
</ul>
<p>Am I understanding the callback execution / new key-value insertion wrong here?</p>
<p>I was of course not suggesting you to extend <code>con_cache</code> to suit my domain specific needs, just trying to find a solution to my own problem, which you have been helping greatly with.</p>
<p>The keyword list itself is a list of tuples <code>{keyword, timestmap}</code>, I don’t know of a better way of organizing it. The size will be max 15 tuples in the list, typically I assume about 3-5 at any given time. For now, I was planning on always fetching the whole keyword list and sending it out to clients.</p>
<p>The rest of the values in the cache are very simple, usually a string or an atom as the value.</p>
<p>As for the message history, I built it as a topic specific dynamically spawned <code>GenServer</code> with Erlang’s <code>:queue</code>, which works spot on for that. Next I need to work out on how to supervise that.</p>
<p><a class="mention" href="/u/mgwidmann" rel="nofollow">@mgwidmann</a> thanks for that blog post link, it’s really helpful as they explain everything in a very pragmatic way there! The way they optimize message passing between nodes is particularly interesting.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="63888" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/channels-where-to-store-state/11076/18">Post #17</a>
	                </div>
	            </div>
              <div id="likers-container-63888" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="63888"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #17"></div>
  </section>
</div>
    <div class="postbit" id="63950" data-post-id="63950">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sasajuric" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sasajuric/120/991_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sasajuric
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Elixir In Action</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="vertti" data-post="18" data-topic="11076">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/vertti/48/7509_2.png" class="avatar"> vertti:</div>
<blockquote>
<p>Am I understanding the callback execution / new key-value insertion wrong here?</p>
</blockquote>
</aside>
<p>No, that’s the gist of my proposal.</p>
<aside class="quote no-group" data-username="vertti" data-post="18" data-topic="11076">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/vertti/48/7509_2.png" class="avatar"> vertti:</div>
<blockquote>
<p>The keyword list itself is a list of tuples {keyword, timestmap}, I don’t know of a better way of organizing it. The size will be max 15 tuples in the list, typically I assume about 3-5 at any given time. For now, I was planning on always fetching the whole keyword list and sending it out to clients.</p>
</blockquote>
</aside>
<p>The reason why I asked is because I wonder how are you accessing this data. Are you actually doing a key-based lookup or are you iterating through all the elements?</p>
<p>If latter, then I have some reservations about how well will ETS perform here. In any case, I advise you to properly measure and consider different options, including a plain data structure.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="63950" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/channels-where-to-store-state/11076/19">Post #18</a>
	                </div>
	            </div>
              <div id="likers-container-63950" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="63950"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>