<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="225697" data-post-id="225697">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Qqwy" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Qqwy/120/1349_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Qqwy
                      <span class="op-star" title="Thread Starter">
                        <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                      </span>
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>TypeCheck Core Team</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<h3><a name="p-225697-better-graphs-1" class="anchor" href="#p-225697-better-graphs-1" aria-label="Heading link" rel="nofollow"></a>Better Graphs</h3>
<p>I did not like the graphs I constructed before very much: they are neither beautiful nor easy to understand, and took a lot of manual labour to create.</p>
<p>In these graphs, we compare the average running time (i.e., ‘lower is better’) of four common operation on sequential collections.</p>
<p></p><div class="lightbox-wrapper"><a class="lightbox" href="https://forum.elixirforum.com/uploads/default/original/3X/0/5/05cfcfa4fde43d7200f5ff0978708b8f0bafc884.png" data-download-href="https://forum.elixirforum.com/uploads/default/05cfcfa4fde43d7200f5ff0978708b8f0bafc884" title="concat" rel="nofollow"><img src="https://forum.elixirforum.com/uploads/default/optimized/3X/0/5/05cfcfa4fde43d7200f5ff0978708b8f0bafc884_2_690x552.png" alt="concat" data-base62-sha1="PpCtBvZUuHjvQ2JMKb1scaUwle" width="690" height="552" srcset="https://forum.elixirforum.com/uploads/default/optimized/3X/0/5/05cfcfa4fde43d7200f5ff0978708b8f0bafc884_2_690x552.png, https://forum.elixirforum.com/uploads/default/optimized/3X/0/5/05cfcfa4fde43d7200f5ff0978708b8f0bafc884_2_1035x828.png 1.5x, https://forum.elixirforum.com/uploads/default/optimized/3X/0/5/05cfcfa4fde43d7200f5ff0978708b8f0bafc884_2_1380x1104.png 2x" data-dominant-color="222F26"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">concat</span><span class="informations">1875×1500 195 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><p></p>
<p>Concatenation of lists seems to be optimized to an extreme extent within the BEAM, because while this should be asymptotically slower, it blows all other implementations out of the water.<br>
For larger collections, <code>A.Vector</code> is significantly faster than <code>ErlangArray</code> or <code>MapArray</code>. This is one operation in which <code>ErlangArray</code> is clearly better than <code>MapArray</code> (in most other operations, their performance is very similar).</p>
<p></p><div class="lightbox-wrapper"><a class="lightbox" href="https://forum.elixirforum.com/uploads/default/original/3X/7/e/7e83102004120cd85850d03f84eff2ee432fd0a7.png" data-download-href="https://forum.elixirforum.com/uploads/default/7e83102004120cd85850d03f84eff2ee432fd0a7" title="random_access" rel="nofollow"><img src="https://forum.elixirforum.com/uploads/default/optimized/3X/7/e/7e83102004120cd85850d03f84eff2ee432fd0a7_2_690x552.png" alt="random_access" data-base62-sha1="i3aXFB4Q8sLmYO9JQboD4Fo8HKT" width="690" height="552" srcset="https://forum.elixirforum.com/uploads/default/optimized/3X/7/e/7e83102004120cd85850d03f84eff2ee432fd0a7_2_690x552.png, https://forum.elixirforum.com/uploads/default/optimized/3X/7/e/7e83102004120cd85850d03f84eff2ee432fd0a7_2_1035x828.png 1.5x, https://forum.elixirforum.com/uploads/default/optimized/3X/7/e/7e83102004120cd85850d03f84eff2ee432fd0a7_2_1380x1104.png 2x" data-dominant-color="222E26"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">random_access</span><span class="informations">1875×1500 171 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><p></p>
<p>Random access is a prime situation for which arrays are better suited than lists (once you have more than ~20 elements; below that, the JIT will be able to optimize the list-based solution to something incredibly fast).<br>
This is another situation in which <code>A.Vector</code> is significantly faster than <code>ErlangArray</code> and <code>MapArray</code>, being just as fast as the builtin lists for small collections, and not really showing any signs of slowing down until having more than ~120_000 elements.<br>
That said, the performance of <code>MapArray</code> and <code>ErlangArray</code> is by no means bad; up to arrays of ~30_000 elements, they are only a constant amount slower than <code>A.Vector</code>.</p>
<p></p><div class="lightbox-wrapper"><a class="lightbox" href="https://forum.elixirforum.com/uploads/default/original/3X/3/0/30b02da652be29e5e769bb8109898e68168bec29.png" data-download-href="https://forum.elixirforum.com/uploads/default/30b02da652be29e5e769bb8109898e68168bec29" title="random_update" rel="nofollow"><img src="https://forum.elixirforum.com/uploads/default/optimized/3X/3/0/30b02da652be29e5e769bb8109898e68168bec29_2_690x552.png" alt="random_update" data-base62-sha1="6WInurYloumAMz1qdMQQzkAndWx" width="690" height="552" srcset="https://forum.elixirforum.com/uploads/default/optimized/3X/3/0/30b02da652be29e5e769bb8109898e68168bec29_2_690x552.png, https://forum.elixirforum.com/uploads/default/optimized/3X/3/0/30b02da652be29e5e769bb8109898e68168bec29_2_1035x828.png 1.5x, https://forum.elixirforum.com/uploads/default/optimized/3X/3/0/30b02da652be29e5e769bb8109898e68168bec29_2_1380x1104.png 2x" data-dominant-color="222D26"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">random_update</span><span class="informations">1875×1500 182 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><p></p>
<p>Updating a random element is another operation for which arrays are clearly more suitable than lists. As soon as you have more than ~20 elements, that is. Here, <code>A.Vector</code> actually is beaten by <code>ErlangArray</code> and <code>MapArray</code> (which perform similarly well), at least until your array has more than ~30_000 elements.</p>
<p></p><div class="lightbox-wrapper"><a class="lightbox" href="https://forum.elixirforum.com/uploads/default/original/3X/4/4/445f03025dbe1b2ec808c79a73732ed001e532d6.png" data-download-href="https://forum.elixirforum.com/uploads/default/445f03025dbe1b2ec808c79a73732ed001e532d6" title="append" rel="nofollow"><img src="https://forum.elixirforum.com/uploads/default/optimized/3X/4/4/445f03025dbe1b2ec808c79a73732ed001e532d6_2_690x552.png" alt="append" data-base62-sha1="9KQ1Qj2sc15xORBRgkGt52CiugK" width="690" height="552" srcset="https://forum.elixirforum.com/uploads/default/optimized/3X/4/4/445f03025dbe1b2ec808c79a73732ed001e532d6_2_690x552.png, https://forum.elixirforum.com/uploads/default/optimized/3X/4/4/445f03025dbe1b2ec808c79a73732ed001e532d6_2_1035x828.png 1.5x, https://forum.elixirforum.com/uploads/default/optimized/3X/4/4/445f03025dbe1b2ec808c79a73732ed001e532d6_2_1380x1104.png 2x" data-dominant-color="282832"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">append</span><span class="informations">1875×1500 227 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><p></p>
<p>It’s clear that appending a single element to the tail of a list is a really bad idea. In some situations, <em>pre</em> pending an element to a list and then ‘interpreting it backwards’ is a possibility, although that will still not allow fast read/write access to any of the other elements in the collection (as seen above).</p>
<p>For appends, all three kinds of arrays (<code>ErlangArray</code>, <code>MapArray</code> and <code>A.Vector</code>) seem to behave similarly well. <code>A.Vector</code> slows <em>down</em> a little here once the ~10_000 element limit has been reached.</p>
<h3><a name="p-225697-which-implementation-should-i-use-2" class="anchor" href="#p-225697-which-implementation-should-i-use-2" aria-label="Heading link" rel="nofollow"></a>Which implementation should I use?</h3>
<p>Benchmarks are nice, and from above benchmarks it seems that either <code>A.Vector</code> if reads are common and <code>ErlangArray</code> if writes/appends are somewhat more common might be a good rule of thumb.</p>
<p>But why choose based on general advice? The main fief of the <code>Arrays</code> library is that it is easy to switch out one array-implementation for another (by an app-wide config or an option passed to <code>Arrays.new</code>), so the better advice is to make your choice based on benchmarking your particular application! <img src="https://forum.elixirforum.com/images/emoji/apple/smiley.png?v=15" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<hr>
<p>I want to thank <a class="mention" href="/u/sasajuric" rel="nofollow">@sasajuric</a> for <a href="https://www.theerlangelist.com/article/sequences" rel="noopener nofollow ugc">his great article on sequences</a> of december last year, and for <a href="https://github.com/sasa1977/erlangelist/tree/master/examples/seq" rel="noopener nofollow ugc">publishing the source code</a> to make the graphs in that article. (And also to <a class="mention" href="/u/sabiwara" rel="nofollow">@sabiwara</a> for his help in this matter – hope you get around to writing that longer blogpost about <code>A.Vector</code> soon!)</p>
<hr>
<h3><a name="p-225697-whats-next-3" class="anchor" href="#p-225697-whats-next-3" aria-label="Heading link" rel="nofollow"></a>What’s next?</h3>
<p>I will probably be spending my spare time the next couple of weeks on some of the other Elixir libraries I’ve been working on <a href="https://forum.elixirforum.com/t/typecheck-fast-and-flexible-runtime-type-checking-for-your-elixir-projects/32886" rel="nofollow">(hint)</a>. After this, work on Arrays will continue. The roadmap is still to add support to more common array operations: swapping elements at two indices, sorting arrays, shuffling arrays, and maybe more if other common operations are identified.</p>
<p>I urge you to try out the library! It is very stable, well-tested and performant.<br>
At some distant point in the future a v3.0 might be released if the <code>Arrays.Protocol</code> might be refined further to include more paths for optimization. This will be a breaking change for array-structure <em>implementers</em> but probably not for <em>users</em> of the library.</p>
<p>Cheers! <img src="https://forum.elixirforum.com/images/emoji/apple/cake.png?v=15" title=":cake:" class="emoji" alt=":cake:" loading="lazy" width="20" height="20"></p>
<p>~Qqwy/Marten</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="225697" data-batch-url="/posts/batch_likers">
                        13
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/arrays-fast-and-versatile-arrays-with-swappable-implementations/42146/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-225697" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="225697"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-most-liked cat-most-liked" title="One of the top 3 liked posts in this thread!"></div>
  </section>
</div>
    <div class="postbit" id="251609" data-post-id="251609">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Qqwy" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Qqwy/120/1349_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Qqwy
                      <span class="op-star" title="Thread Starter">
                        <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                      </span>
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>TypeCheck Core Team</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I’ve been doing some more work on <a href="https://github.com/Qqwy/elixir-arrays_rrb_vector" rel="noopener nofollow ugc">ArraysRRBVector</a>:</p>
<p>Now that there is a reasonably simple way to call an Elixir function from within Rust (c.f. RustlerElixirFun: <a href="https://github.com/Qqwy/elixir-rustler_elixir_fun" rel="noopener nofollow ugc">repo</a>,  <a href="https://forum.elixirforum.com/t/rustlerelixirfun-calling-elixir-from-rust/47984" rel="nofollow">topic</a>), it seemed sensible to try this out by implementing a new version of <code>map</code> (as in <code>Enum.map</code>, but keeping it as an array) for this native array datastructure.</p>
<p>The new approach iterates over the leaf nodes in the reduced-radix-binary tree, which (when fully filled, which all except the last one are), each contain 64 elements. This means that:</p>
<ul>
<li>We call into Elixir only <s><code>O(log_64(n))</code></s> times if you have an array with <code>n</code> elements. <em>EDIT: Actually, this is incorrect as we do not drill down to a single leaf node, but rather iterate over all leaves. Thus, we still run in <code>O(n/64) = O(n)</code> time. Still a significant reduction of function call overhead though.</em></li>
<li>We only convert between Rust datastructures and Elixir datastructures 64 elements at a time. This greatly reduces memory usage if you have very big vectors (as we do not need to create side-by-side copies of the full vector anymore).</li>
</ul>
<p>I have not done any benchmarks yet. I’m pretty sure that it is still not as fast as staying in Elixir the whole time (when e.g. using an ErlangArray, MapArray or Aja.Vector), but it definitely is faster than the old approach of turning the whole vector fully into a list, mapping over that in Elixir, and then turning it back.</p>
<p>Most importantly, it might be a nice example for people of you to use <a href="https://github.com/Qqwy/elixir-rustler_elixir_fun" rel="noopener nofollow ugc">RustlerElixirFun</a> to call Elixir code from inside Rust in their own projects.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="251609" data-batch-url="/posts/batch_likers">
                        6
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/arrays-fast-and-versatile-arrays-with-swappable-implementations/42146/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-251609" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="251609"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="334826" data-post-id="334826">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="maruks" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/maruks/120/3641_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  maruks
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I got this compliation error when I tried to use your library:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">
error: undefined function .../0 (expected TypeCheck.Internals.UserTypes.TypeCheck.DefaultOverrides.Macro to define such a function or for it to be imported, but none are available)
    │
  4 │   @type! captured_remote_function() :: (... -&gt; any())
    │                                         ^^^
    │
    └─ lib/type_check/default_overrides/macro.ex:4:41: TypeCheck.Internals.UserTypes.TypeCheck.DefaultOverrides.Macro.captured_remote_function/0


== Compilation error in file lib/type_check/default_overrides/macro.ex ==
** (CompileError) lib/type_check/default_overrides/macro.ex: cannot compile module TypeCheck.Internals.UserTypes.TypeCheck.DefaultOverrides.Macro (errors have been logged)
    expanding macro: TypeCheck.Macros.__before_compile__/1
    lib/type_check/default_overrides/macro.ex:1: TypeCheck.DefaultOverrides.Macro (module)
could not compile dependency :type_check, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile type_check --force", update it with "mix deps.update type_check" or clean it with "mix deps.clean type_check"
</code></pre>
<p>Version:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">
Erlang/OTP 27 [erts-15.0] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.17.1
</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="334826" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/arrays-fast-and-versatile-arrays-with-swappable-implementations/42146/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-334826" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="334826"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="334855" data-post-id="334855">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Qqwy" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Qqwy/120/1349_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Qqwy
                      <span class="op-star" title="Thread Starter">
                        <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                      </span>
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>TypeCheck Core Team</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Thank you for reporting!</p>
<p>For anyone else coming across this problem: it seems to be caused by a change in how <code>...</code> is parsed in the Elixir AST from v1.16 to v1.17, which the TypeCheck library (a transitive dependency of the Arrays library) currently does not handle correctly.</p>
<p><a href="https://github.com/Qqwy/elixir-type_check/issues/190" rel="noopener nofollow ugc">GitHub issue here</a>.</p>
<p>Unfortunately, I’m on vacation until the start of August, so it might be a while for a fix to be implemented.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="334855" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/arrays-fast-and-versatile-arrays-with-swappable-implementations/42146/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-334855" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="334855"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>