<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="371074" data-post-id="371074">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jaybe78" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jaybe78/120/36069_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jaybe78
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Thank you all</p>
<p>Really smart to use random key from the start.</p>
<p>I have a few questions for you <a class="mention" href="/u/jstimps" rel="nofollow">@jstimps</a> just to confirm my assumption about your gist</p>
<p>You get random keys in 2 steps, the first time you start from a random key as mentioned by <a class="mention" href="/u/garrison" rel="nofollow">@garrison</a> and the second time(keys2), you use <code>first</code>, if not enough keys have been found</p>
<p>i.e</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">
start = :crypto.strong_rand_bytes(32)
 keyed_items = first_n(state.t, start, n, [])

...
case :ets.next(t, key) do
</code></pre>
<blockquote>
<p>Sample from a random index in the uniform space and just scan until you hit the quota</p>
</blockquote>
<p>This would not really be efficient, especially if there’s less keys in the table so at some point it’s better to start from <code>first</code>, right ?</p>
<p>i.e</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">keyed_items2 = if(length(keyed_items) &lt; n, do: first_n(state.t, nil, n-length(keyed_items), []), else: [])

...
case :ets.first(t) do

</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371074" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/22">Post #21</a>
	                </div>
	            </div>
              <div id="likers-container-371074" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371074"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #21"></div>
  </section>
</div>
    <div class="postbit" id="371075" data-post-id="371075">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jaybe78" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jaybe78/120/36069_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jaybe78
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<blockquote>
<p>I don’t think I fully understand the use-case here but putting that aside for a moment…</p>
</blockquote>
<p>Sorry, if I was not clear enough, what part of it you don’t understand ?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371075" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/23">Post #22</a>
	                </div>
	            </div>
              <div id="likers-container-371075" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371075"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #22"></div>
  </section>
</div>
    <div class="postbit" id="371078" data-post-id="371078">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jstimps" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/120/41062_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jstimps
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I think it should be pretty efficient due to the <code>:ordered_set</code>, which should get you O(log n). Only way to know for sure is to test in your specific use case.</p>
<p>As <a class="mention" href="/u/garrison" rel="nofollow">@garrison</a> pointed out though this scheme might not be rigorously uniform. Some simulation testing might be helpful to confirm it’s fairness.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371078" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/24">Post #23</a>
	                </div>
	            </div>
              <div id="likers-container-371078" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371078"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #23"></div>
  </section>
</div>
    <div class="postbit" id="371107" data-post-id="371107">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jstimps" data-post="24" data-topic="71997">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jstimps/48/41062_2.png" class="avatar"> jstimps:</div>
<blockquote>
<p>this scheme might not be rigorously uniform</p>
</blockquote>
</aside>
<p>If you see my edit I <em>think</em> my second proposal (combining ideas from earlier in the thread) would be uniform. The problem with “select a position and scan next N” is that you bias that group to be chosen together. Like I said for some use cases this <em>doesn’t matter</em> (nobody will ever notice or care), and I think this is one of those cases.</p>
<p>But if it did matter you can just sample random N random positions instead of sampling the range <code>[X, X+N)</code>. It probably wouldn’t even be meaningfully slower (there would be <em>some</em> difference due to cache but random access memory is still pretty alright at random access).</p>
<p>This was proposed upthread but the problem (as mentioned) is that deletions can bias the sample if you use sequential keys. Imagine you insert 1,000,000 sequential keys and then delete the first 500,000. Now a uniform random number has a 50% chance of grabbing the first user! Not great <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>But by using random keys it becomes impossible to bias the sample in any way with your insert/delete pattern.</p>
<p>I think, anyway, this stuff can be really unintuitive to reason about. If you need to confirm you can run a simulation as suggested.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371107" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/25">Post #24</a>
	                </div>
	            </div>
              <div id="likers-container-371107" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371107"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #24"></div>
  </section>
</div>
    <div class="postbit" id="371117" data-post-id="371117">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jaybe78" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jaybe78/120/36069_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jaybe78
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<blockquote>
<p>But by using random keys it becomes impossible to bias the sample in any way with your insert/delete pattern.</p>
</blockquote>
<p>Have you checked his gist ?<br>
He’s not using sequential number but a random key</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">def handle_call({:get, n}, _from, state) do
    start = :crypto.strong_rand_bytes(32)
    keyed_items = first_n(state.t, start, n, [])

    for {key, _} &lt;- keyed_items, do: :ets.delete(state.t, key)

    keyed_items2 = if(length(keyed_items) &lt; n, do: first_n(state.t, nil, n-length(keyed_items), []), else: [])

    for {key, _} &lt;- keyed_items2, do: :ets.delete(state.t, key)

    keyed_items = keyed_items ++ keyed_items2

    items = for {_key, item} &lt;- keyed_items do
      insert(state.t, item)
      item
    end
    {:reply, items, state}
  end

</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371117" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/26">Post #25</a>
	                </div>
	            </div>
              <div id="likers-container-371117" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371117"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #25"></div>
  </section>
</div>
    <div class="postbit" id="371121" data-post-id="371121">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jaybe78" data-post="26" data-topic="71997">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jaybe78/48/36069_2.png" class="avatar"> jaybe78:</div>
<blockquote>
<p>He’s not using sequential number but a random key</p>
</blockquote>
</aside>
<p>Yes, I was referring to earlier suggestions in the thread and how they work better when combined with his idea of using random keys.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371121" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/27">Post #26</a>
	                </div>
	            </div>
              <div id="likers-container-371121" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371121"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #26"></div>
  </section>
</div>
    <div class="postbit" id="371811" data-post-id="371811">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="manhvu" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/manhvu/120/39832_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  manhvu
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>It’s too later, but I hope someone else can have an idea for work with <code>:ets</code> table.</p>
<p>I think one <code>:ets</code> table with <code>:set</code> type is enough for this case.</p>
<p><strong>Solution:</strong></p>
<p>My solution is keep your tuple has format like <code>{username, index, data}</code>. <code>username</code> is key of table.<br>
You have an auto incremental number for index element in tuple.<br>
Last index number can keep in state of GenServer or :persistent_term.<br>
For new <code>username</code> just increase last index for that.</p>
<p>For get data by username it’s using <code>lookup</code> function.</p>
<p>For get random data in table you can use Enum.random(1..last_index) (or other random function) and get data by one of these ways:</p>
<ol>
<li>Use <code>match</code> &amp; <code>match_object</code> function. Increase random number if current random number is removed.</li>
<li>Use <code>select</code> function for query data with index equal or larger than random number (limit = 1).</li>
</ol>
<p>Note: I recommend create <code>:ets</code> table with <code>:protected</code> mode for other process can read directly &amp; set <code>:write_concurrency</code> + <code>:read_concurrency</code> to <code>true</code> for improve performance.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371811" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/28">Post #27</a>
	                </div>
	            </div>
              <div id="likers-container-371811" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371811"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #27"></div>
  </section>
</div>
    <div class="postbit" id="371825" data-post-id="371825">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Using match without an exact key will <a href="https://www.erlang.org/doc/apps/stdlib/ets.html#match/2" rel="nofollow">scan the entire table</a>. It doesn’t even support prefix scans (which is super annoying).</p>
<p>You would want to keep a second table as an index. Also, the solution you describe has the bias problems discussed previously.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371825" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/29">Post #28</a>
	                </div>
	            </div>
              <div id="likers-container-371825" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371825"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #28"></div>
  </section>
</div>
    <div class="postbit" id="371861" data-post-id="371861">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="manhvu" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/manhvu/120/39832_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  manhvu
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Oh, I missed this. Not like Mnesia table, ets cannot have an index for another field. So I think for this case can use Mnesia with memory table and add index can get some advantage when scale out.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="371861" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/search-random-elements-in-a-big-ets-table-1m/71997/30">Post #29</a>
	                </div>
	            </div>
              <div id="likers-container-371861" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="371861"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>