shirawebill
Hi,
I am trying to stream a file from AWS.
It is a very large file, so I thought it would be better to use Stream and , once that works, to chunk the data.
I am using NimbleCSV - the parse_stream function.
I am getting back an empty list after running Enum.to_list
This is my code so far:
alias NimbleCSV.RFC4180, as: CSV
ExAws.S3.download_file("testfiles", "file_upload.csv", :memory)
|> ExAws.stream!()
|> CSV.parse_stream
|> Enum.to_list
I would like to do something like:
alias NimbleCSV.RFC4180, as: CSV
ExAws.S3.download_file("testfiles", "file_upload.csv", :memory)
|> ExAws.stream!()
|> CSV.parse_stream
|> Stream.chunk_every(10_000)
|> Stream.map(& process_stream(&1)
|> Enum.to_list
Thanks! 
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #ai
- #graphql
- #genstage
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 5- Show Best Posts
- Show All Posts (oldest first)
- Show All Posts (newest first)
mindok
Have you checked that
ExAws.stream!()gives you anything?LostKobrakai
Afaik streaming data from AWS gives you chunks of same size in bytes. NimbleCSV expects a stream of (individual) lines. So you need to convert between those two, which NimbleCSV has a helper for.
mindok
I just had a more detailed read through the
ExAws.S3docs - there is also an example there of converting the stream into a stream by line - see ExAws.S3 — ExAws.S3 v2.5.9 , but the NimbleCSV one (NimbleCSV — NimbleCSV v1.3.0) would be the way to go.LostKobrakai
They should be mostly the same code since Update example for streaming lines by fastjames · Pull Request #170 · ex-aws/ex_aws_s3 · GitHub, but yeah, using the available API over copy pasting the implementation is certainly simpler.
shirawebill
Thank you, the CSV.parse_stream worked!
Final code: