ImNotAVirus
`Stream.chunk_every/4` equivalent for Flow
Context:
While practicing optimizing some exercises, I got stuck this morning trying to use Flow on a piece of code using Stream.
Here is the code :
counter = fn x, acc ->
Map.update(acc, x, 1, &(&1 + 1))
end
# Create a stream for large files processing
"your_filename.ext"
|> File.stream!()
# Normalize every words
|> Stream.map(&String.downcase/1)
|> Stream.map(&String.replace(&1, ~r"[^a-z0-9]", " "))
# Split with spaces and remove empty words
|> Stream.flat_map(&String.split(&1, " ", trim: true))
# Get every sequences of 3 words in the text
|> Stream.chunk_every(3, 1, :discard)
# Join the sequences
|> Stream.map(&Enum.join(&1, " "))
# Count occurences for 3 words sequences
|> Enum.reduce(%{}, counter)
# Just displays the 10 most used sequences
|> Map.to_list()
|> Enum.sort_by(&elem(&1, 1), :desc)
|> Enum.take(10)
|> IO.inspect()
# Example of output with http://www.gutenberg.org/cache/epub/2009/pg2009.txt :
# [
# {"of the same", 320},
# {"the same species", 130},
# {"conditions of life", 125},
# {"in the same", 117},
# {"of natural selection", 111},
# {"from each other", 104},
# {"species of the", 102},
# {"on the other", 89},
# {"the other hand", 81},
# {"the case of", 78}
# ]
In short, this piece of code retrieves all the 3-word sequences of a text and displays the 10 that appear the most.
Problem:
I can’t find the equivalent of the line Stream.chunk_every(3, 1, :discard) for Flow.
Here is my current code:
Mix.install([{:flow, "~> 1.2"}])
counter = fn x, acc ->
Map.update(acc, x, 1, &(&1 + 1))
end
"your_filename.ext"
|> File.stream!(read_ahead: 100_000)
|> Flow.from_enumerable()
|> Flow.map(&String.downcase/1)
|> Flow.map(&String.replace(&1, ~r"[^a-z0-9]", " "))
|> Flow.flat_map(&String.split(&1, " ", trim: true))
# |> Stream.chunk_every(3, 1, :discard) <= ????????
|> Flow.map(&Enum.join(&1, " "))
|> Flow.partition()
|> Flow.reduce(&Map.new/0, counter)
|> Enum.to_list()
|> Enum.sort_by(&elem(&1, 1), :desc)
|> Enum.take(10)
|> IO.inspect()
Popular in Questions
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: )
Hello all, this is ...
New
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar.
I p...
New
I am trying to run a deploy with docker and I successfully runned with this command:
docker build -t romenigld/blog-prod .
but when I t...
New
Other popular topics
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
New
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]
15:22:35.803 [error] gen_event {lager_file_backend...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex









