ImNotAVirus

ImNotAVirus

`Stream.chunk_every/4` equivalent for Flow

Context:
While practicing optimizing some exercises, I got stuck this morning trying to use Flow on a piece of code using Stream.

Here is the code :

counter = fn x, acc ->
  Map.update(acc, x, 1, &(&1 + 1))
end

# Create a stream for large files processing
"your_filename.ext"
|> File.stream!()
# Normalize every words
|> Stream.map(&String.downcase/1)
|> Stream.map(&String.replace(&1, ~r"[^a-z0-9]", " "))
# Split with spaces and remove empty words
|> Stream.flat_map(&String.split(&1, " ", trim: true))
# Get every sequences of 3 words in the text
|> Stream.chunk_every(3, 1, :discard)
# Join the sequences
|> Stream.map(&Enum.join(&1, " "))
# Count occurences for 3 words sequences
|> Enum.reduce(%{}, counter)
# Just displays the 10 most used sequences
|> Map.to_list()
|> Enum.sort_by(&elem(&1, 1), :desc)
|> Enum.take(10)
|> IO.inspect()

# Example of output with http://www.gutenberg.org/cache/epub/2009/pg2009.txt :
#    [
#      {"of the same", 320},
#      {"the same species", 130},
#      {"conditions of life", 125},
#      {"in the same", 117},
#      {"of natural selection", 111},
#      {"from each other", 104},
#      {"species of the", 102},
#      {"on the other", 89},
#      {"the other hand", 81},
#      {"the case of", 78}
#    ]

In short, this piece of code retrieves all the 3-word sequences of a text and displays the 10 that appear the most.

Problem:
I can’t find the equivalent of the line Stream.chunk_every(3, 1, :discard) for Flow.

Here is my current code:

Mix.install([{:flow, "~> 1.2"}])

counter = fn x, acc ->
  Map.update(acc, x, 1, &(&1 + 1))
end

"your_filename.ext"
|> File.stream!(read_ahead: 100_000)
|> Flow.from_enumerable()
|> Flow.map(&String.downcase/1)
|> Flow.map(&String.replace(&1, ~r"[^a-z0-9]", " "))
|> Flow.flat_map(&String.split(&1, " ", trim: true))
# |> Stream.chunk_every(3, 1, :discard)      <= ????????
|> Flow.map(&Enum.join(&1, " "))
|> Flow.partition()
|> Flow.reduce(&Map.new/0, counter)
|> Enum.to_list()
|> Enum.sort_by(&elem(&1, 1), :desc)
|> Enum.take(10)
|> IO.inspect()

Where Next?

Popular in Questions Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44265 214
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

We're in Beta

About us Mission Statement