<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="368244" data-post-id="368244">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="tmbb" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  tmbb
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Due to the fact that Explorer (the elixir library) is based on Polars (the rust library), which is based on a rust implementation of Arrow (a standard to store data in memory), I have tried to re-implement SciEx to be based on Arrow instead of ndarray, in order to make it comaptible with Explorer. It would also make it much easier to store and move data around.</p>
<p>However, I haven’t been able to make it work in a satisfying way… Arrow doesn’t even provide a multidimensional array type you can operate on as a unit. It does provide tensors, but it does not provide vectoried operations on tensors and I can’t think of a good way to implement something like a Fast Fourier Transform algorithm on Arrow datatypes.</p>
<p>If anyone can think of a way to “port” SciEx to Arrrow I’d be very interested.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368244" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/sciex-scientific-programming-for-elixir-based-on-bindings-to-rusts-ndarray/70058/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-368244" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368244"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="368245" data-post-id="368245">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="cevado" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/cevado/120/40655_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  cevado
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>instead of porting the whole library, wouldn’t be enough to either create another library or create a function to convert from the data type that SciEx uses to the data types that works with Explorer?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368245" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/sciex-scientific-programming-for-elixir-based-on-bindings-to-rusts-ndarray/70058/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-368245" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368245"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="368246" data-post-id="368246">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="tmbb" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  tmbb
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Yes, that would probably be sufficient, I guess. It seems easy to convert the containers in SciEx (Array1, Array2, etc.) and Polars (ChunkedArray, etc.) into and from a plain <code>Vec&lt;_&gt;</code>. In the worst case you can always convert things into an Elixir <code>list()</code> and then convert it into the corresponding rust type, but there is probably a way of doing it portably using the Arrow format for serialization.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368246" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/sciex-scientific-programming-for-elixir-based-on-bindings-to-rusts-ndarray/70058/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-368246" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368246"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="368368" data-post-id="368368">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="tmbb" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  tmbb
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I have just added some intersting functions to create arrays from Elixir. Now you can create a 1D-array from an elixir list (I’m still thinking on what the best way to implement the creating of multidimensional arrays) and a 2D-array from a black and white PNG image:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">iex&gt; iex(2)&gt; SciEx.Float64.Array1.from_list([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0])     
#SciEx.Float64.Array1&lt;[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0], shape=[7], strides=[1], layout=CFcf (0xf), const ndim=1&gt;
iex&gt; SciEx.Float64.Array2.from_image("test/data/visualization/heightmap.png")
#SciEx.Float64.Array2&lt;[[0.27450980392156865, 0.2549019607843137, 0.21176470588235294, 0.1843137254901961, 0.1568627450980392, ..., 0.4470588235294118, 0.44313725490196076, 0.43137254901960786, 0.3843137254901961, 0.3215686274509804],
 [0.2823529411764706, 0.25098039215686274, 0.21176470588235294, 0.1803921568627451, 0.14901960784313725, ..., 0.3843137254901961, 0.3764705882352941, 0.38823529411764707, 0.34509803921568627, 0.2980392156862745],
 [0.25882352941176473, 0.21568627450980393, 0.18823529411764706, 0.1607843137254902, 0.13333333333333333, ..., 0.3176470588235294, 0.3176470588235294, 0.3333333333333333, 0.3176470588235294, 0.2901960784313726],
 [0.22745098039215686, 0.2, 0.1803921568627451, 0.15294117647058825, 0.12941176470588237, ..., 0.3058823529411765, 0.29411764705882354, 0.27450980392156865, 0.24705882352941178, 0.23921568627450981],   
 [0.21176470588235294, 0.1803921568627451, 0.16470588235294117, 0.1411764705882353, 0.12549019607843137, ..., 0.27450980392156865, 0.2549019607843137, 0.2, 0.17647058823529413, 0.1803921568627451],     
 ...,
 [0.2980392156862745, 0.27450980392156865, 0.2627450980392157, 0.27058823529411763, 0.2823529411764706, ..., 0.23137254901960785, 0.22745098039215686, 0.2235294117647059, 0.22745098039215686, 0.21568627450980393],
 [0.30980392156862746, 0.29411764705882354, 0.2980392156862745, 0.2980392156862745, 0.2823529411764706, ..., 0.21568627450980393, 0.2235294117647059, 0.2196078431372549, 0.21176470588235294, 0.21176470588235294],
 [0.3333333333333333, 0.3254901960784314, 0.3254901960784314, 0.3215686274509804, 0.30196078431372547, ..., 0.21176470588235294, 0.21568627450980393, 0.2235294117647059, 0.2196078431372549, 0.21176470588235294],
 [0.34901960784313724, 0.34509803921568627, 0.35294117647058826, 0.34901960784313724, 0.32941176470588235, ..., 0.20392156862745098, 0.21568627450980393, 0.21176470588235294, 0.21568627450980393, 0.2196078431372549],
 [0.35294117647058826, 0.3411764705882353, 0.35294117647058826, 0.35294117647058826, 0.3411764705882353, ..., 0.17647058823529413, 0.2, 0.20784313725490197, 0.21176470588235294, 0.2235294117647059]], shape=[257, 257], strides=[-257, 1], layout=c (0x4), const ndim=2&gt;
</code></pre>
<p>These are small things but very important in practice when exploring the library. Although you’ll probably want to load your data from big files stored on your hard drive somehow (I still need to add support for that), having simple ways of loading basic data goes a long way when trying things out.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="368368" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/sciex-scientific-programming-for-elixir-based-on-bindings-to-rusts-ndarray/70058/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-368368" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="368368"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="370749" data-post-id="370749">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="tmbb" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  tmbb
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<h2><a name="p-370749-request-for-help-1" class="anchor" href="#p-370749-request-for-help-1" aria-label="Heading link" rel="nofollow"></a>Request for help</h2>
<p>I’d like to request help (or maybe even tips) regarding the following topics:</p>
<h3><a name="p-370749-code-quality-and-compile-times-2" class="anchor" href="#p-370749-code-quality-and-compile-times-2" aria-label="Heading link" rel="nofollow"></a>Code quality and compile times</h3>
<p>Currently, I implement arrays of complex numbers (32 or 64 bits) for 1 to 6 dimensions as separate rust types (2x2x6=24 different types of arrays). I think this makes it easy to improve communication between Elixir and rust because it makes almost everything “monomorphic” on the rust side and allows me to deal with non-unique types only on the Elixir side. However, I wonder if this doesn’t make it a bit harder to implement functions with multile arguments on the rust side. It also requires creating a huge number of rust functions (usually at least one per array type). Is there a way to leverage traits or something like that to decrease compile-times and improve the code size?</p>
<h3><a name="p-370749-array-dimensions-3" class="anchor" href="#p-370749-array-dimensions-3" aria-label="Heading link" rel="nofollow"></a>Array dimensions</h3>
<p>Currently, I have hardcoded the array dimensions as being between 1 and 6. Is this a problem? Has anyone ever used dense arrays of dim &gt; 6 in practice? A dense array of dimension &gt; 6 is usully so huge I can’t think of a use for it…</p>
<h3><a name="p-370749-rustlerprecompiled-4" class="anchor" href="#p-370749-rustlerprecompiled-4" aria-label="Heading link" rel="nofollow"></a>RustlerPrecompiled</h3>
<p>Is anyone interested in integrating rustler precompiled in this package?</p>
<p>Thanks in advance to all interested people</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="370749" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/sciex-scientific-programming-for-elixir-based-on-bindings-to-rusts-ndarray/70058/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-370749" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="370749"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #15"></div>
  </section>
</div>
    <div class="postbit" id="384741" data-post-id="384741">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="tmbb" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  tmbb
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>A release has been published on hex, because I’ve found myself using parts of it in other projects. A lot of the functionality still doesn’t have proper tests, and if you’re not willing to read the code yourself, everything that isn’t simple arithmetic or function calls is probably not yet good enough for production use.</p>
<p>I have added new features, including support for hierarchical clustering based on dissimilarity matrices. This can be useful to perform clustering on sentence embeddings from LLMs.</p>
<p>I have added support for 1D kernel density estimates and 2D contouring (isolines). This package doesn’t plot them and never will, but it can be integrated with packages for plotting.</p>
<p>I have also added utilities to generate multidimensional arrays from Elixir lists. Thus is not very useful for practical use but it is useful for testing and exploring small examples.</p>
<p>In case engine wants to contribute, I suggest the following topics:</p>
<ul>
<li>Adding tests to the clustering logic (maybe take them from SciPy, who’s API I’m trying to emulate)</li>
<li>Adding support for RustlerPrecompiled</li>
<li>Find a way of integrating tests suites from other languages for the functions (even though the functions are taken from the rust stdlib or some other rust package, testing the implementation over the arrays seems like a good idea)</li>
</ul> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="384741" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/sciex-scientific-programming-for-elixir-based-on-bindings-to-rusts-ndarray/70058/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-384741" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="384741"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>