I didn’t mean to be rude and sorry if it read that way – my point is that without some level of understanding we cannot help you because you might not understand and not be able to apply the suggestions given to you. So then how can we help you?
What I suggested to you is: integrate all processing of the CSV file together in one place so you can reap the full benefits of streaming and save CPU time and memory. If you need an even more detailed example, I’ll be happy to give it to you – can even make a GitHub repo for you if you provide some sample CSV data we can work with. (E.g. how much columns does your CSV file have?)
One of my first tasks with Elixir – 5.5 years ago now – was exactly to process hundreds of megabytes of CSV and XML files per hour. On a virtual hosting with a 4-core vCPU and using full parallelism (using Task.async_stream or Flow) I wasn’t able to load the CPU to more than 60% while still ingesting several CSV / XML files with sizes 500MB+, per hour. There are ways and we here are trying to show you those ways.
I don’t have a horse in this race – just don’t be too quick to discount Elixir as unsuitable. There’s a lot that can be done. And you won’t regret sticking with Elixir. It saves you from a ton of problems that no other language has solved yet (like transparent parallelism and concurrency).
I never did that before.
