How would you mutate Explorer Series data like an Enum reduce

"Hello everyone,

I’m currently diving into Elixir’s data libraries, exploring ways to apply stock market technical analysis logic using Explorer and data frames.

During my attempt to calculate the RSI (Relative Strength Index) for a stock, I encountered a requirement to compute series elements based on a formula that involves the previous row in a data frame. I initially tried using the shift function, but it appears not to track changes effectively, possibly due to lazy evaluation.

Let me simplify the problem. Imagine having a list of 1’s, and you want to update each element by adding it to the previous one.

%{a: [1, 1, 1, 1, 1]}
|> DF.new()
|> DF.mutate(c: S.add(a, S.shift(a, 1)))

What I expect is [1, 2, 3, 4, 5].

I’m wondering if there’s a more straightforward approach, perhaps involving a reduction function, where I can easily track the updated values. It’s possible I’m making it more complex than necessary. :thinking:"

Here is the actual livebook file I’m testing this out.

Javascript equivalent code that I’m trying out in elixir dataframe.

2 Likes

Hi @pkrawat1,

The short answer is that there are a few open issues related to this functionality:

A version of it is definitely on the roadmap. But we need to structure the API in such a way that Polars can do the heavy lifting, so it’s tricky.

For the time being, the only way I know how to do this is to convert your series to an Elixir data structure, do this part in Elixir, then convert back.

3 Likes

Hey @billylanchantin

Thanks for the heads up! I see those Explorer issues, and it’s good to know they’re on the roadmap. Wrapping my head around the Polars challenges – tricky but exciting! :sweat_smile:

The workaround tip with converting to Elixir data structure, doing the magic, and then converting back is handy for now. Much appreciated! :raised_hands: