Geegee
Importing large python npy data sets and doing post processing
A couple of months ago I started learning Erlang for fun after watching a
talk about Joe Armstrong and from there went on to further learn Elixir
to some extent overall liking it a lot. At work I mostly deal with Python and
have to deal with large data sets acquired by measurement hardware.
The data is 3 dimensional and typically saved to hard drive in the .npy format.
I wondered if it is possible with Nx to perform least squared optimization (numerical fits)
on the GPU? This would allow me to speed up the process a lot maybe even allowing
real time analysis of the data.
Most Liked
billylanchantin
Hi @Geegee, welcome to the forum 
In principle, yes. Nx is designed to work on the GPU and in fact has a least squares function built in:
The trouble will be getting the data out of the .npy format. A quick search turned up this library:
That appears to be exactly what you want. I have not tried that library though so I can’t vouch for it.
Geegee
Awesome. I will try that out sometime with a livebook to see how it goes.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"