AWS - How to stream a CSV file

Have you checked that ExAws.stream!() gives you anything?

Afaik streaming data from AWS gives you chunks of same size in bytes. NimbleCSV expects a stream of (individual) lines. So you need to convert between those two, which NimbleCSV has a helper for.

I just had a more detailed read through the ExAws.S3 docs - there is also an example there of converting the stream into a stream by line - see ExAws.S3 — ExAws.S3 v2.5.9 , but the NimbleCSV one (NimbleCSV — NimbleCSV v1.3.0) would be the way to go.

They should be mostly the same code since Update example for streaming lines by fastjames · Pull Request #170 · ex-aws/ex_aws_s3 · GitHub, but yeah, using the available API over copy pasting the implementation is certainly simpler.

Thank you, the CSV.parse_stream worked!

Final code:

alias NimbleCSV.RFC4180, as: CSV

ExAws.S3.download_file("testfiles", "file_upload.csv", :memory) 
|> ExAws.stream!() 
|> CSV.to_line_stream()
|> CSV.parse_stream ()
|> Stream.chunk_every(10_000)
|> Stream.map(& process_stream(&1)
|> Enum.to_list()