frumos

frumos

Simple way to stream by line from compressed .gz file

Hello,

I have a text file which has lines with new line separation “\n”

I need to stream this file by lines so I can apply my own functions per line and I am doing like this:

iex(30)> "file.txt" |> File.stream!() |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50)

Now the same file becomes gzip compressed data and I would like to stream it by lines and get the same result as from above code.

I am doing it like:

iex(38)> "file.gz" |> File.stream!([], 2024*2024) |> StreamGzip.gunzip() |> WHAT_I_NEED_TO_PUT_HERE |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50) 

and question is what function(s) i need to put instead of “WHAT_I_NEED_TO_PUT_HERE” placeholder ?

I tried lots of things , splitting by “\n” applying flat_map but I can not get result that I am getting new line after StreamGzip.gunzip()

Seems this simple requirement is not trivial to implement.

Thank you for your time.

Most Liked

frumos

frumos

One of my colleague suggest this:

"file.gz" |> File.stream!([:compressed]) |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50)

I tried with that test file and it seems it works out of the box. Need to do more testing but if it is that simple I can not believe, I struggled so much with gz files in Elixir in past …

Where Next?

Popular in Questions Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

Other popular topics Top

WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36352 110
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54120 245
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement