I have 16 core here, what needs to be tested? ^.^
OvermindDL1
JEG2
Author of Designing Elixir Systems with OTP
My “System Report” says I have 8 cores, but the BEAM starts 16 schedulers when I run it.
Another advantage of my approach though is that memory usage should stay pretty reasonable, since it’s still working line-by-line.
1
sb8244
Author of Real-Time Phoenix
I ran @JEG2 locally and it came up with 9.5s. Here’s the comparison:
➜ time elixir wp_parallel.exs < ./words.txt > /dev/null
elixir wp_parallel.exs < ./words.txt > /dev/null 27.14s user 1.79s system 302% cpu 9.550 total
➜ time elixir wp_singlel.exs < ./words.txt > /dev/null
elixir wp_single.exs < ./words.txt > /dev/null 10.28s user 2.91s system 103% cpu 12.739 total
You can see that the parallel version uses much more CPU, but it brings the time down by ~20%. It is a cool approach to read in parallel. I tried to read at once and then spawn out the processes, which was slower than I expected.
0
jola
I guess you have 8 physical cores and 16 virtual?
@darinwilson running with > /dev/null gets it down lower
➜ elixirconf time elixir lib/direction3/async_stream.ex < ../words.txt > /dev/null
elixir lib/direction3/async_stream.ex < ../words.txt > /dev/null 17.79s user 3.08s system 380% cpu 5.484 total
and then you can cheat a bit
➜ elixirconf time elixir --erl "+hms 500000000" lib/direction3/async_stream.ex < ../words.txt > /dev/null
elixir --erl "+hms 500000000" lib/direction3/async_stream.ex < ../words.txt > 16.26s user 2.82s system 420% cpu 4.543 total
For those keeping score at home, 4.5 is almost as fast as C 
(admittedly it uses 4x CPU)
Here’s my execution time for @JEG2’s version
➜ elixirconf time elixir lib/extra/jeg2.exs < ../words.txt > /dev/null
elixir lib/extra/jeg2.exs < ../words.txt > /dev/null 27.02s user 1.77s system 276% cpu 10.417 total
Using the +hms cheat brings it down to like 7 seconds.
I really like this solution. You don’t have to stitch together prefix+suffix and memory usage is about 1GB on my machine. Speed is totally reasonable 
4
dimitarvp
evadne
My count-stream solution was sorting by string content instead of by count. This has been fixed and gives a modest speed-up.
Old:
$ elixir ./count.ex ./words.txt --sort 1> out-sorted.txt
Started
Using 4 Workers
Processing: 4.665098s
Load Results: 0.819044s
Order Results: 2.111665s
Print Results: 3.020239s
Total Runtime: 10.620714s
New:
$ elixir ./count.ex ./words.txt --sort 1> out-sorted.txt
Started
Using 4 Workers
Processing: 4.474982s
Load Results: 0.717677s
Order Results: 1.109028s
Print Results: 2.277602s
Total Runtime: 8.583487s
The stream based solution is faster though. Additionally, with the sort function replaced with List.keysort/2 there was another 500ms or so recovered.
0
jola
You weren’t sorting by strings, you were sorting by count (they second value in the tuple). The output was correct. At least in the version I got from your gist.
defp report_sort(entries), do: Enum.sort(entries, fn {_, lhs}, {_, rhs} -> lhs > rhs end)
With List.keysort they get sorted in reverse order though (asc instead of desc). There doesn’t seem to be an option to choose order? But yeah, it’s faster 
nice find! Something like 200-400ms after a few runs.
And a message to everyone in the thread: I just want to point out that all my measurements are with the output printed to terminal, unless otherwise specified. Just to be clear. Running with > /dev/null or > outputfile is much faster.
0
alvises
Hey @jola, just re-watched your talk at ElixirConf 2019 
To close the “hole” in the pipeline between the Enum.each and :ets.tab2list, what do you think about Enum.reduce with the ets table as the accumulator?
...
|> Enum.reduce(:ets.new(:words, []), fn word, table ->
:ets.update_counter(table, word, {2, 1}, {word, 0})
table
end)
|> :ets.tab2list()
...
Performance wise I get same results. But maybe we could argue that returning the table in the reduction is not aesthetically that pleasant 
1
jola
tycoles
Hey @jola! Really enjoyed your talk. Learned a lot from it
I jacked @JEG2’s code and wrote it in a slightly different way. For whatever reason it seems to run slightly faster for me (by like half a second or so). Maybe someone smarter than me knows the reason 
It uses await_many, which was introduced in Elixir 1.11. Nothing like resurrecting a thread for some code golf!
table = :ets.new(:words, [:public, write_concurrency: true])
pattern = :binary.compile_pattern([" ", "\n"])
for _ <- 1..System.schedulers_online() do
Task.async(fn ->
Stream.take_while(IO.binstream(:stdio, :line), fn line ->
line
|> String.split(pattern)
|> Enum.each(fn word ->
:ets.update_counter(table, word, {2, 1}, {word, 0})
end)
end)
|> Stream.run()
end)
end
|> Task.await_many(:infinity)
:ets.tab2list(table)
|> Enum.sort(fn {_, a}, {_, b} -> b < a end)
|> Enum.map(fn {word, count} ->
[String.pad_leading(Integer.to_string(count), 8), " ", word, "\n"]
end)
|> IO.binwrite()
2