Hello,

and thank you for the excellent libraries like Nx, Scholar, Axon..

We are in the process of transitioning some of our work from Python to Elixir. It’s heartening to see that many of our tasks can be accomplished in Elixir, thanks to Nx and the surrounding ecosystem. However, we have not yet found a solution for handling partial/batches to conserve memory when performing tasks like PCA.

PCA in Scholar

Given our dataset, we would need to construct a tensor of s64[50000][10000] as input, which poses challenges for several reasons.

In Python, we might use something like IncrementalPCA from scikit-learn to handle this. However, we’re unsure if we’ve overlooked a way to achieve this with Nx/Scholar.

We would appreciate any guidance or directions on this matter.
Thank you.