<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="5671" data-post-id="5671">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="karolsluszniak" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/karolsluszniak/120/1627_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  karolsluszniak
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<blockquote>
<p>Just to be clear: there’s no process communication happening here. Everything happens in the same process, since you’re working with both files in the raw mode (default for file streams). For clarification, see “Processes and raw files” in File doc1.</p>
</blockquote>
<p>This is interesting. Initially I also thought there’s no process involved, but I kept that statement from the original article assuming that even if we don’t spawn an extra process (which happens with <code>File.open</code> in its default not-raw config) then there is still some low-level process within Erlang VM itself that’s responsible for serving file operations in a non-blocking fashion.</p>
<p>But then again, even if that would be true, I can’t really be sure if that could even be classified as “process”. It may be a thread instead and use other means for delivering data to our process than what we used to call “sending messages”. That seems to be the case according to the top part of <a href="http://erlang.org/doc/man/file.html" rel="nofollow">this doc</a>. So ultimately you’re right <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> It was a leftover from the original article where I’ve clearly mixed up the concepts of interprocess message passing with a tight stream loop that runs on the same process. I’ve fixed it.</p>
<blockquote>
<p>Also, I couldn’t sleep over the fact that streamed version was about 3x slower on my machine than the read version (1.8s vs 0.6s). I played with it a bit more and discovered that streaming bytes works much faster than streaming lines.</p>
</blockquote>
<p>This is amazing. I’ve tried taking it further, with <a href="https://github.com/karolsluszniak/process-csv/commit/8ae1d0d6daeb709ad4586cf63b463392924777c0" rel="noopener nofollow ugc">this</a> and then <a href="https://github.com/karolsluszniak/process-csv/commit/fa4ef6cd79f6ef1e5eea9b5e8290bf5fb909d03d" rel="noopener nofollow ugc">this</a>, but failed to beat your version. I wonder if it could be made even faster, perhaps by adding some clever concurrency.</p>
<blockquote>
<p>Note that I’m splitting input per bytes, so I’m not sure whether this will work correctly with unicode files.</p>
</blockquote>
<p>As far as I understood the docs, <code>String.split</code> is UTF-8 aware. Also, you’re using <code>utf8</code> qualifiers on every pattern match so it looks like it should work with UTF-8 just fine to me. The <code>IO.binwrite</code> doc includes a warning about using it with devices in unicode mode, but that’s not the case here. I’ve added some emoticons to input CSV and they are still there in the output, but I’m not sure that’s an ultimate proof of being UTF-8 compilant <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="5671" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/help-with-performance-file-io/802/32">Post #31</a>
	                </div>
	            </div>
              <div id="likers-container-5671" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="5671"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #31"></div>
  </section>
</div>
    <div class="postbit" id="5678" data-post-id="5678">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sasajuric" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sasajuric/120/991_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sasajuric
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Elixir In Action</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="karolsluszniak" data-post="32" data-topic="802">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/karolsluszniak/48/1627_2.png" class="avatar"> karolsluszniak:</div>
<blockquote>
<p>This is amazing. I’ve tried taking it further, with this and then this, but failed to beat your version. I wonder if it could be made even faster, perhaps by adding some clever concurrency.</p>
</blockquote>
</aside>
<p>I actually tried some variants of your approaches first, and they failed <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I think this one is the fastest because <code>String.split</code> will just forward to <code>:binary.split</code> if the split pattern is a binary, and then the split will be done in C code. But it’s just a guess on why I think <code>String.split</code> works better here.</p>
<aside class="quote no-group" data-username="karolsluszniak" data-post="32" data-topic="802">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/karolsluszniak/48/1627_2.png" class="avatar"> karolsluszniak:</div>
<blockquote>
<p>As far as I understood the docs, String.split is UTF-8 aware. Also, you’re using utf8 qualifiers on every pattern match so it looks like it should work with UTF-8 just fine to me.</p>
</blockquote>
</aside>
<p>One thing that I wonder, but didn’t verify is what happens if a 2 byte codepoint ends up being split over two chunks. I suspect it might still work, but didn’t really check it.</p>
<p>In any case, this was a nice exercise. Thanks for writing the post, which motivated me to do some playing on my own. I think together both of us learned a lot in the process <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="5678" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/help-with-performance-file-io/802/33">Post #32</a>
	                </div>
	            </div>
              <div id="likers-container-5678" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="5678"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #32"></div>
  </section>
</div>
    <div class="postbit" id="5681" data-post-id="5681">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sasajuric" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sasajuric/120/991_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sasajuric
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Elixir In Action</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="karolsluszniak" data-post="32" data-topic="802">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/karolsluszniak/48/1627_2.png" class="avatar"> karolsluszniak:</div>
<blockquote>
<p>I wonder if it could be made even faster, perhaps by adding some clever concurrency.</p>
</blockquote>
</aside>
<p>I’m not gonna say no since I didn’t try it, but I’m not sure we can get a lot from concurrency here since there’s not a lot of processing on each element, so I think the cost of message passing would shadow any possible benefit of concurrency.</p>
<p>But to be honest, I’m fairly pleased with results. On my machine I can read, filter, and write ~500k rows of CSV in about 1 sec, using about 64kb of memory. That seems decent to me <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="5681" data-batch-url="/posts/batch_likers">
                        5
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/help-with-performance-file-io/802/34">Post #33</a>
	                </div>
	            </div>
              <div id="likers-container-5681" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="5681"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #33"></div>
  </section>
</div>
    <div class="postbit" id="5688" data-post-id="5688">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="benwilson512" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/benwilson512/120/1457_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  benwilson512
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Craft GraphQL APIs in Elixir with Absinthe</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>One very relevant area with concurrency is when you need to process multiple files. We have to do something a lot like this at cargosense as part of parsing binary data files that get, and I’m able to do several hundred MB/sec of IO on my MBP by leveraging 4-8 files processing concurrently. With streams, this only uses about 10mb of memory.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="5688" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/help-with-performance-file-io/802/35">Post #34</a>
	                </div>
	            </div>
              <div id="likers-container-5688" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="5688"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #34"></div>
  </section>
</div>
    <div class="postbit" id="26931" data-post-id="26931">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sunaku" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sunaku/120/13206_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sunaku
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><a href="https://sunaku.github.io/elixir-fileio-speedup.html" rel="noopener nofollow ugc">I encountered a similar slowdown</a> while reading a 1.5 GiB log file (which contained enormous NUL byte sequences due to corruption) line by line in Elixir 1.4.2 and Erlang 19.2 under Linux 3.16, which took <strong>8 hours and 7 minutes</strong> to complete!</p>
<p>However, I was able to bring the execution time <strong>down to 90 seconds</strong> (thereby achieving a massive <strong>320x speedup</strong>) by reading the file as a raw byte stream (instead of UTF-8)  and with generous caching:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">file = "path/to/very/large/file"
io = file |&gt; File.open!(read_ahead: 128 * 1024) # 128 KiB cache
lines = io |&gt; IO.binstream(:line)
</code></pre>
<p><a href="https://sunaku.github.io/elixir-fileio-speedup.html" rel="noopener nofollow ugc">See my blog post</a> for the details on the major contributing sources of this solution.  Cheers!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="26931" data-batch-url="/posts/batch_likers">
                        6
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/help-with-performance-file-io/802/36">Post #35</a>
	                </div>
	            </div>
              <div id="likers-container-26931" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="26931"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #35"></div>
  </section>
</div>
    <div class="postbit" id="314903" data-post-id="314903">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="stevensonmt" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/stevensonmt/120/20503_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  stevensonmt
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="cjbottaro" data-post="17" data-topic="802">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/cjbottaro/48/1469_2.png" class="avatar"> cjbottaro:</div>
<blockquote>
<p><a href="http://blog.alainodea.com/en/article/398/blazing-fast-concurrent-text-i-o-in-erlang" rel="noopener nofollow ugc">http://blog.alainodea.com/en/article/398/blazing-fast-concurrent-text-i-o-in-erlang</a></p>
</blockquote>
</aside>
<p>This link appears to be broken. Does anyone know if that blog moved or is archived somewhere?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="314903" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/help-with-performance-file-io/802/37">Post #36</a>
	                </div>
	            </div>
              <div id="likers-container-314903" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="314903"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>