<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="123830" data-post-id="123830">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jola" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jola/120/18010_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jola
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Nice! That’s a really cool solution, and probably scales well! It fails the first requirement though, the input should be piped in. Reading from a file is faster than reading from stdin in general (eg replace <code>IO.binstream</code> with <code>File.read</code> and it runs faster) and gives you much more control, like you show in your code.</p>
<p>You can speed up your version quite a bit if you switch to iolists, instead of printing line by line.</p>
<p>I actually ended up writing an article about this script, with my final version single threaded (13 lines of code) running in 13 seconds <a href="https://blog.jola.dev/elixir-string-processing-optimization" class="inline-onebox" rel="noopener nofollow ugc">Elixir String Processing Optimization | jola.dev</a></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="123830" data-batch-url="/posts/batch_likers">
                        7
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/23">Post #22</a>
	                </div>
	            </div>
              <div id="likers-container-123830" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="123830"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #22"></div>
  </section>
</div>
    <div class="postbit" id="123927" data-post-id="123927">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="alvises" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/alvises/120/2247_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  alvises
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Hi <a class="mention" href="/u/jola" rel="nofollow">@jola</a>, did you try getting rid of the <code>:line</code> option and processing chunk by chunk? What I’ve seen so far (more details here: <a href="https://forum.elixirforum.com/t/streaming-lines-from-an-enum-of-chunks/21244" class="inline-onebox" rel="nofollow">Streaming lines from an enum of chunks</a>) is that this</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">File.stream!("numbers.txt",[],2048) #chunks instead of lines
|&gt; Stream.transform("",fn chunk, acc -&gt;
  [last_line | lines] = 
      acc &lt;&gt; chunk
      |&gt; String.split("\n")
      |&gt; Enum.reverse()
	{Enum.reverse(lines),last_line}
end)
</code></pre>
<p>seems to be 2x faster than using the <code>:line</code> option in <code>File.stream!</code>. I still have to try it in your code and see if there is any difference.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="123927" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/24">Post #23</a>
	                </div>
	            </div>
              <div id="likers-container-123927" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="123927"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #23"></div>
  </section>
</div>
    <div class="postbit" id="123928" data-post-id="123928">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jola" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jola/120/18010_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jola
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I did do that in the article <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> and compared performance as well as some words on the downsides</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="123928" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/25">Post #24</a>
	                </div>
	            </div>
              <div id="likers-container-123928" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="123928"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #24"></div>
  </section>
</div>
    <div class="postbit" id="123929" data-post-id="123929">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="alvises" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/alvises/120/2247_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  alvises
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>thx <a class="mention" href="/u/jola" rel="nofollow">@jola</a>, do you mean this part?</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">IO.binstream(:stdio, 102400)
|&gt; Enum.to_list()
|&gt; :binary.list_to_bin()
|&gt; String.split(pattern)
</code></pre>
<p>Does it load everything into memory and then splits?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="123929" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/26">Post #25</a>
	                </div>
	            </div>
              <div id="likers-container-123929" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="123929"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #25"></div>
  </section>
</div>
    <div class="postbit" id="123934" data-post-id="123934">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jola" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jola/120/18010_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jola
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>It does <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> like I said in the article, I didn’t spend too much time messing around with the chunk size, but that number gave decent performance improvement compared to reading line by line with that input on my machine.</p>
<p>Also, switching to reading the file directly would be faster, but the original article used stdin in all examples, so it would be not be a fair comparison.</p>
<p>Take a look at <a class="mention" href="/u/evadne" rel="nofollow">@evadne</a>’s solution for one that optimizes reading over 4 workers if you’re curious how fast it  can be reading directly from file.</p>
<aside class="quote quote-modified" data-post="22" data-topic="21091">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/evadne/48/13949_2.png" class="avatar">
    <div class="quote-title__text-content">
      <a href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/22" rel="nofollow">Erlang/Elixir string performance - can this be improved?</a> <a class="badge-category__wrapper " href="/c/questions-help/questions/53" rel="nofollow"><span data-category-id="53" style="--category-badge-color: #C14BFB; --category-badge-text-color: #000000; --parent-category-badge-color: #C14BFB;" data-parent-category-id="171" data-drop-close="true" class="badge-category --style-square --has-parent" title="Elixir Questions / Help"><span class="badge-category__name">Questions</span></span></a>
    </div>
  </div>
  <blockquote>
    Hi, a friend of mine sent this over, so I gave it a try: <a href="https://gist.github.com/evadne/43e79dd51c068dc19e42086cdbecc8a7" class="inline-onebox-loading" rel="noopener nofollow ugc">https://gist.github.com/evadne/43e79dd51c068dc19e42086cdbecc8a7</a> 
FWIW 
$ curl http://teralink.net/\~serpent/words.txt &gt; words.txt

$ md5 words.txt 
MD5 (words.txt) = 640ef9e082ef75ef07f0e9869e9d8ae2

$du -h words.txt
217M	words.txt
Execution

$ elixir ./count.ex ./words.txt --sort &gt; out-sorted.txt
Started
Using 4 Workers
Processing: 4.748574s
Load Results: 0.820397s
Order Results: 1.99405s
Print Results: 3.136065s
Total Runtime: 10.710908s…
  </blockquote>
</aside>
 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="123934" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/27">Post #26</a>
	                </div>
	            </div>
              <div id="likers-container-123934" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="123934"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #26"></div>
  </section>
</div>
    <div class="postbit" id="123949" data-post-id="123949">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="evadne" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/evadne/120/13949_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  evadne
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><a href="https://gist.github.com/evadne/43e79dd51c068dc19e42086cdbecc8a7#file-count-stream-ex" rel="noopener nofollow ugc">New version</a> is up. It reads from STDIN and somehow manages to beat the file-based approach. <img src="https://forum.elixirforum.com/images/emoji/apple/wink.png?v=15" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<pre data-code-wrap="elixir"><code class="lang-elixir">elixir ./count-stream.ex &lt; words.txt &gt; out.txt
Started
Processing: 4.627264s
Reporting: 0.771289s
Total Runtime: 5.408457s
</code></pre>
<pre data-code-wrap="elixir"><code class="lang-elixir">elixir ./count-stream.ex --sort &lt; words.txt &gt; out.txt
Started
Processing: 4.761361s
Reporting: 1.936912s
Total Runtime: 6.707307s
</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="123949" data-batch-url="/posts/batch_likers">
                        11
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/28">Post #27</a>
	                </div>
	            </div>
              <div id="likers-container-123949" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="123949"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-most-liked cat-most-liked" title="One of the top 3 liked posts in this thread!"></div>
  </section>
</div>
    <div class="postbit" id="142432" data-post-id="142432">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="JEG2" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/JEG2/120/936_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  JEG2
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Designing Elixir Systems with OTP</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I love this problem.  I tried several things that I just knew you be faster, only to learn the hard way that they were not.</p>
<p>Here’s the faster thing I was able to come up with:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">defmodule WordCounter do
  def run(table, pattern, parent) do
    spawn(__MODULE__, :read_and_count, [table, pattern, parent])
  end

  def read_and_count(table, pattern, parent) do
    case IO.binread(:line) do
      :eof -&gt;
        send(parent, {:done, self()})

      line -&gt;
        line
        |&gt; String.split(pattern)
        |&gt; Enum.each(fn word -&gt;
          :ets.update_counter(table, word, {2, 1}, {word, 0})
        end)
        read_and_count(table, pattern, parent)
    end
  end

  def wait_on(pid) do
    receive do
      {:done, ^pid} -&gt;
        :ok
    end
  end
end

table = :ets.new(:words, [:public, write_concurrency: true])
pattern = :binary.compile_pattern([" ", "\n"])

Stream.repeatedly(fn -&gt; WordCounter.run(table, pattern, self()) end)
|&gt; Enum.take(System.schedulers_online)
|&gt; Enum.each(fn pid -&gt; WordCounter.wait_on(pid) end)

:ets.tab2list(table)
|&gt; Enum.sort(fn {_, a}, {_, b} -&gt; b &lt; a end)
|&gt; Enum.map(fn {word, count} -&gt;
  [String.pad_leading(Integer.to_string(count), 8), " ", word, "\n"]
end)
|&gt; IO.binwrite
</code></pre>
<p>I did like that I was able to get some decent speed with a still pretty straight forward approach.  (It just reads lines in multiple processes.)</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="142432" data-batch-url="/posts/batch_likers">
                        10
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/29">Post #28</a>
	                </div>
	            </div>
              <div id="likers-container-142432" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="142432"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-most-liked cat-most-liked" title="One of the top 3 liked posts in this thread!"></div>
  </section>
</div>
    <div class="postbit" id="142442" data-post-id="142442">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="darinwilson" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/darinwilson/120/15104_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  darinwilson
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Programming Ecto</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>What was the runtime of this implementation?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="142442" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/30">Post #29</a>
	                </div>
	            </div>
              <div id="likers-container-142442" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="142442"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #29"></div>
  </section>
</div>
    <div class="postbit" id="142479" data-post-id="142479">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="JEG2" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/JEG2/120/936_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  JEG2
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Designing Elixir Systems with OTP</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>It runs in about 8.7 seconds on my laptop.  I expect that it’s super dependent on how many cores a machine has.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="142479" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/31">Post #30</a>
	                </div>
	            </div>
              <div id="likers-container-142479" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="142479"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #30"></div>
  </section>
</div>
    <div class="postbit" id="142502" data-post-id="142502">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="darinwilson" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/darinwilson/120/15104_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  darinwilson
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Programming Ecto</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>That’s awesome! I think <a class="mention" href="/u/jola" rel="nofollow">@jola</a> got hers down to 7-ish seconds in her talk (if I’m remembering correctly)? But it would be neat to throw yours onto a 16-core machine and see how it does <img src="https://forum.elixirforum.com/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=15" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="142502" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/erlang-elixir-string-performance-can-this-be-improved/21091/32">Post #31</a>
	                </div>
	            </div>
              <div id="likers-container-142502" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="142502"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #31"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/21091/load_more?page=4">Load more posts (10 remaining)</a>
</div></template></turbo-stream>