frumos
Simple way to stream by line from compressed .gz file
Hello,
I have a text file which has lines with new line separation “\n”
I need to stream this file by lines so I can apply my own functions per line and I am doing like this:
iex(30)> "file.txt" |> File.stream!() |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50)
Now the same file becomes gzip compressed data and I would like to stream it by lines and get the same result as from above code.
I am doing it like:
iex(38)> "file.gz" |> File.stream!([], 2024*2024) |> StreamGzip.gunzip() |> WHAT_I_NEED_TO_PUT_HERE |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50)
and question is what function(s) i need to put instead of “WHAT_I_NEED_TO_PUT_HERE” placeholder ?
I tried lots of things , splitting by “\n” applying flat_map but I can not get result that I am getting new line after StreamGzip.gunzip()
Seems this simple requirement is not trivial to implement.
Thank you for your time.
Most Liked
frumos
One of my colleague suggest this:
"file.gz" |> File.stream!([:compressed]) |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50)
I tried with that test file and it seems it works out of the box. Need to do more testing but if it is that simple I can not believe, I struggled so much with gz files in Elixir in past …
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









