Huge CSV processing

This conference talk by José Valim sounds like it may be of interest to you: https://www.youtube.com/watch?v=XPlXNUXmcgE

It goes from processing data in an eager way (reading everything up front), to a lazy way (leveraging Streams), to concurrent processing. Depending on how complex your data+processing is it sounds like Streams would be more than sufficient.

If you provide more context my opinion could change though

You may be aware of it already, but Plataformatec created a library called NimbleCSV that I would look it. I’ve never used it on anything close to a 10 GB file, but it was pretty easy to use, and it supports streaming pretty well. I would think that as long as you handled it in appropriately-sized chunks you should be fine.

As far as scheduling the task goes, the only app-level crons I’ve done in Elixir used Quantum. I don’t remember it well enough to endorse, but I also don’t remember hating it.

I process large CSVs also… In addition to nimble_csv when I need to do the reading/writing in Elixir I also shell out from my Elixir app to xsv extensively for pre-processing… it’s written in Rust and super fast. Not sure what processing you need to do with the hourly files but it could speed you along.

https://github.com/BurntSushi/xsv

Thank you very much @brettbeatty @tme_317 and @BurntSushi.
You helped me a lot.

I know this is an old thread, but it still appears in searches for processing large CSV files. I maintain RustyCSV, a NimbleCSV-compatible parser and encoder using precompiled Rust NIFs. It supports bounded-memory streaming for files like this.

More details here: