How to generate a big CSV file concurrently?

There is no such possibility, as in the end you will be locked on the writing anyway (writes to the single file must be sequential). So if these “thousand entries” do not fit in RAM then your only option is to write it to separate files and then join these files. If it fits in the memory then you can build iodata in memory and then dump it into a file. However I think that speedup will be marginal, as this operation is probably IO bound, not CPU bound, so parallelisation will have marginal impact there.

You can stream into a file without having it all in memory. “thousand entries” really isn’t that many though.

stream_of_data
|> NimbleCSV.RFC4180.dump_to_stream
|> Stream.into(File.stream!("output.csv"))

But without parallelism. If sequentially then of course that you do not need to keep everything in memory at once.

Ah yes good point. As you note though parallelism within the same file is unlikely to be all that helpful.

A general query: Isn’t this problem similar to logging from multiple process to a single log file? How does the elixir logger solves this problem ?

It isn’t really a problem. A single process is perfectly capable of writing to a file as fast as the hard drive will allow as long as it can get data fast enough. In Logger there is a single process that owns the file, and then other processes that need to write logs send that log process lines (short version). There are optimizations there where ets can be used to communicate or manage back pressure but that’s the rough idea.

Ah that makes sense, Thanks