Elixir Nx - any plans for something like Pandas for data analysis + could the pluggable compiler be used for purposes outside Nx?

Taufeeq_Mowzer · February 20, 2021, 11:33pm

Hi, I’m a senior medstudent interested in programming and the concurrency/fault-tolerance of the Beam vm. I have two questions:

Is Elixir planning on having something like pandas for data analysis, or is the focus mainly on ML?
Having watched the Lambda days talk of J Valim. Wanted to know if the pluggable compiler feature could be used for purposes outside Nx, such as utilizing LLVM for webassembly or performance purposes?

My understanding is quite limited atm and I only learn in my free time (rotating through pediatric surgery atm).

I feel that ML, though seemingly interesting, is not something i see myself spending time with anytime soon, but general data analysis seems useful and feel it might assist me in some ideas i might have in future. With respect to pluggable compilers, the computational performance with XLA compiler made me feel that we could possibly use it with other compilers for performance (Only when needed) outside numerical computation?

I do wish to contribute to/develope some library of my own in time. Apologies forbthe lengthy message

As for why Elixir: Concurrency/fault-tolerance/very friendly community/hardworking-passionate J Valim/not looking for employment/feel its ideal for a indie dev or startup that wants to scale-up using in-house hosting.

seanmor5 · February 21, 2021, 9:41am

Hi,

To answer your questions:

I think in general our goal is to support something like Pandas for Data analysis, but within the scope of the Nx project. We have some ideas, but nothing concrete yet. One of the ways you can help is maybe providing your use case to our mailing list. We’d be interested to hear how you would do data analysis in Nx/Elixir, as well as what you think something of a data analysis tool would look like!
The pluggable compiler is a reference to Tensor Compilers. So if a Tensor Compiler targets WebAssembly then it certainly could work. The compilers are pluggable just for Nx expressions, which are a small subset of the language, not for Elixir as a whole. If you’re interested in WebAssembly on the BEAM, there is the Lumen project. For the most part, the incredible performance gains from Tensor compilers come from specialization rather than generalization.

Taufeeq_Mowzer · February 21, 2021, 12:21am

Thanks, I would be sure to to give my input at some time.

For the time being what comes to mind is having dedicated scraping tools (AFAIK the available scrapers in Elixir only work on structured datasets) as well as keeping integration with phoenix in mind.

Even though Microsoft Azure is not as popular as other implementations of data science; however, their ecosystem seems to be more structured in terms of the tools they have all in one place; which Elixir/Nx could perhaps try to emulate their structure in its own extent.

I heard of Lumen yes, i also remember J Valim mentioning that the Beamjit is leveraging asmjit for its compilation as well.

My use case (not anytime soon) would be taking patient data from different communities and finding trends among them…eventually ML would probably come into play down the line. (you could say epidemiology, but with its(my) own spin).

Something that comes to mind is where i live, during lockdown we’ve had several ‘alcohol ban’ where the sale of alcohol was prohibited over a specific time-frame(s) all over the country…at the time of the bans, various doctors claimed that their trauma units were virtually empty… A women i attended to suffered from wernickes encephalopathy (severe thiamine deficiency from acute alcohol withdrawal in an alcoholic), injected with thiamine, on follow up, no changes…had to rush her to an ER as she developed Korsakoff syndrome (a more severe form); i saw another male who’s breathe reeked of alcohol complaining of muscle aches of unknown cause (it was alcohol induced, the ban was then lifted for weekends so it checked out according to the time frame etc.)…this wouldn’t be what I’d necessarily use it for per say…but it is an interesting example of what you could use it for… A large time-frame where dates stipulating ban of alcohol sales/changes made to the ban/lifting of the ban/restarting the ban, and comparing it to illnesses/causes for hospital admission. One could find major relationships related to alcohol spanning over around a whopping 10 month period!

Btw, thanks for your influence on Nx and the Elixir community!