Importing large python npy data sets and doing post processing

A couple of months ago I started learning Erlang for fun after watching a
talk about Joe Armstrong and from there went on to further learn Elixir
to some extent overall liking it a lot. At work I mostly deal with Python and
have to deal with large data sets acquired by measurement hardware.
The data is 3 dimensional and typically saved to hard drive in the .npy format.
I wondered if it is possible with Nx to perform least squared optimization (numerical fits)
on the GPU? This would allow me to speed up the process a lot maybe even allowing
real time analysis of the data.

Hi @Geegee, welcome to the forum :slight_smile:

In principle, yes. Nx is designed to work on the GPU and in fact has a least squares function built in:

The trouble will be getting the data out of the .npy format. A quick search turned up this library:

That appears to be exactly what you want. I have not tried that library though so I can’t vouch for it.

2 Likes

Awesome. I will try that out sometime with a livebook to see how it goes.

1 Like