sabri
How to generate a big CSV file concurrently?
Hello
I need to generate a big CSV file, that might has thousands of entries, how can I run concurrent processes to handle this task and all write to the resulted file at the same time?
Is there an advice? what is the recommendation here?
Marked As Solved
benwilson512
You can stream into a file without having it all in memory. “thousand entries” really isn’t that many though.
stream_of_data
|> NimbleCSV.RFC4180.dump_to_stream
|> Stream.into(File.stream!("output.csv"))
Also Liked
benwilson512
It isn’t really a problem. A single process is perfectly capable of writing to a file as fast as the hard drive will allow as long as it can get data fast enough. In Logger there is a single process that owns the file, and then other processes that need to write logs send that log process lines (short version). There are optimizations there where ets can be used to communicate or manage back pressure but that’s the rough idea.
hauleth
But without parallelism. If sequentially then of course that you do not need to keep everything in memory at once.
hauleth
There is no such possibility, as in the end you will be locked on the writing anyway (writes to the single file must be sequential). So if these “thousand entries” do not fit in RAM then your only option is to write it to separate files and then join these files. If it fits in the memory then you can build iodata in memory and then dump it into a file. However I think that speedup will be marginal, as this operation is probably IO bound, not CPU bound, so parallelisation will have marginal impact there.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance








