JordiPolo

JordiPolo

Performant data structure for a table (think 2D array or list of lists)

I am writing a library to deal with dataframes. Dataframes can be thought as excel tables. The main data is a 2D table and any operation can be row based or column based.
Operations include filtering by column, row, adding or deleting rows, columns, etc.

Right now, in a very naive way, I’ve implemented the main data as a list of lists, row based. But of course this makes column operations expensive (add a new element in a certain position to each list for instance).

If I switch to list of list, column based, then row operations would be expensive.
Tuples would not make it I believe because they may get huge and there may be lots of modifications
and Maps are not ordered, and I need order.

Any advise? Are out there any comparison of performance of each of the Elixir data structures?

Most Liked

Qqwy

Qqwy

TypeCheck Core Team

I think you might want to look at the Tensor library that I made a while back, which allows for vectors, matrices and any higher-order tensors as well. It works with sparse maps, making it more efficient when it is only partially filled. It probably provides the functionality you want (a multitude of different ways of filtering, adding/removing rows or colums, flipping or rotating, fast access to a single element, row or column, easy ways to sort rows or columns, etc).

It is of course also available on Hex.pm.

NobbZ

NobbZ

Extracting a single row or column I think is most efficient by using Enum.filter/2. This would make extracting a single column or row O(row * colums) though.

If you really need O(1) access to a complete row or column randomly, I do think, that a doubled DS would be the only thing to go, but it does double the amount of memory needed to hold the structure, also it makes writes expensive since they had to be done twice.

Basically you need to two nested maps %{row_id => %{col_id => data_sample}} and %{col_id => %{row_id => data_sample}}.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Yeah the duplication is definitely a possibility. Fortunately this is less duplication than it may look at first glance because cell values are just pointed to by each map location, not actually duplicated.

:ets could be another possibility because it does support a {_, 1} style match operation.

FWIW even traditional mutable 2d arrays suffer from this question. If you make row access primary then column access requires large jumps in memory which eats away at any cache locality you may have gotten with row access.

Where Next?

Popular in Discussions Top

blackode
Elixir Upgrading is so Simple in Ubuntu and It worked for me Ubuntu 16.04 git clone https://github.com/elixir-lang/elixir.git cd elixir...
New
Rustixir
Hi everyone, im working on find best language/framework/system for high concurrency, high performance and stable performance after wor...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
lorenzo
Hey everone! I created a prototype for my app using Nodejs for the api. But the framework I chose wasnt great (in general theresnt any g...
New
New
tomekowal
Hey guys! I want to create a toy project that shows a chart of temperature over time and updates every 5 seconds. I feel LiveView is per...
New
scouten
I’m looking for a host for the server part of a small (personal) side project that I’m working on. It’s currently written in Node.js and ...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
fireproofsocks
This is more of a general question, but I’m wondering how other people in the community think about the pattern matching in function sign...
New
Markusxmr
Since Drab has been developed for a while in the open, introducing the Liveview functionality in a way it happend appears to undermine th...
New

Other popular topics Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 31013 112
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36352 110
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement