@ LostKobrakai Initially we tried this
|> File.read!()
|> CSV.decode()
It was even worse.
I followed the suggestion to use streams, the data being large.
quda
0
LostKobrakai
Streaming doesn’t make things faster. It trades less memory for more cpu cycles. Switching from CSV to NimbleCSV is what makes things faster.
0
quda
LostKobrakai
to
is being faster in my book. Now if you can handle having all the data in memory (reading from the fs once instead of streaming) you’ll likely again see a speed improvement.
0
quda
Right. Slightly faster but still not usable/performant:
21.4MB → 7.2min
Why the timing expands exponentially to the size of the file ?!?
0
LostKobrakai
Sure, but also you’re going back to the file system for each individual line. File.stream! by default works on a per line basis. Unless you’re memory constraint it’s likely a lot faster to load more data into memory at a time.
1
krasenyp
I was wondering the same thing. @quda is still loading all the data into memory. Maybe something like the following would work.
result_stream =
path
|> Path.expand(__DIR__)
|> File.stream!()
|> NimbleCSV.RFC4180.parse_stream(skip_headers: false)
|> Stream.map(&Map.new/1)
# in your controller
conn = send_chunked(conn)
chunk(conn, "[")
stream_result
|> Stream.transform(true, fn
m, true -> {Jason.encode_to_iodata!(m), false}
m, false -> {["," | Jason.encode_to_iodata!(m)], false}
end)
|> Stream.each(&chunk/1)
|> Stream.run()
chunk(conn, "]")
1
kokolegorille
LostKobrakai
I really doubt the memory is the issue here given it worked with loading all data into memory for the CSV backed implementation.
0
