<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="194484" data-post-id="194484">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sasajuric" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sasajuric/120/991_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sasajuric
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Elixir In Action</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="solanav" data-post="1" data-topic="35504">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/solanav/48/20748_2.png" class="avatar"> solanav:</div>
<blockquote>
<p>it seems like the scheduler gets to 100% utilization which is bad</p>
</blockquote>
</aside>
<p>While a nif is running on a scheduler its utilization will be plotted as 100%. This is normal.</p>
<aside class="quote no-group" data-username="solanav" data-post="1" data-topic="35504">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/solanav/48/20748_2.png" class="avatar"> solanav:</div>
<blockquote>
<p>it becomes impossible to do things like launch the observer</p>
</blockquote>
</aside>
<p>If the NIF is running on a dirty scheduler it should not block regular schedulers. Here’s how I tested it locally:</p>
<ol>
<li>In the C code of the dirty NIF function I added <code>sleep(10);</code></li>
<li>I started the shell with <code>ELIXIR_ERL_OPTIONS="+S 1" iex -S mix</code> (to make sure there’s just one scheduler and one dirty CPU scheduler)</li>
<li>From the shell I started the nif as <code>spawn(&amp;my_nif/0)</code></li>
<li>As soon as process was spawned the iex shell was responsive and I was able to start the observer.</li>
</ol>
<aside class="quote no-group" data-username="solanav" data-post="1" data-topic="35504">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/solanav/48/20748_2.png" class="avatar"> solanav:</div>
<blockquote>
<p>downloads a lot of files in parallel</p>
</blockquote>
</aside>
<p>If you invoke a lot of longer-running dirty nifs concurrently, you may end up occupying all the dirty scheduler threads, which might block some other operations. For example, <code>length</code> will switch to dirty schedulers for larger list, and if all schedulers are busy for a couple of seconds, <code>length</code> will be blocked. I was able to reproduce this by spawning a long-running NIF, and then invoking <code>length(Enum.to_list(1..1_000_000))</code> which blocked until NIF finished.</p>
<p>Based on the description of the scenario this looks like a case where port might be a better fit.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="194484" data-batch-url="/posts/batch_likers">
                        9
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/dirty-cpu-nif-blocking-scheduler/35504/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-194484" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="194484"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-solved cat-solved" title="Marked as solution"></div>
  </section>
</div>
    <div class="postbit" id="194513" data-post-id="194513">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="mpope" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  mpope
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Good point.  I should have also mentioned that it should be ran in parallel to the nif execution.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="194513" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/dirty-cpu-nif-blocking-scheduler/35504/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-194513" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="194513"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="194566" data-post-id="194566">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="solanav" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/solanav/120/20748_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  solanav
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Thank you so much for the explanation, I will rewrite the NIF as a port to see if it’s a better choice.</p>
<p>I’m a big fan by the way, keep up the good work!</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="194566" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/dirty-cpu-nif-blocking-scheduler/35504/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-194566" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="194566"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="194568" data-post-id="194568">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sasajuric" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sasajuric/120/991_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sasajuric
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Author of Elixir In Action</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Based on your screenshot it seems that a single <code>extract</code> call takes a few seconds, right? If that’s the case, I recommend limiting the amount of concurrency, i.e. make sure you’re not running more than <code>x</code> simultaneous extracts, where <code>x</code> should probably be smaller than the number of cpu cores.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="194568" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/dirty-cpu-nif-blocking-scheduler/35504/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-194568" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="194568"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="194805" data-post-id="194805">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dch" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dch/120/2682_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dch
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>For NIF related code I usually refer to Paul Davis, Jesper Andersen, &amp; Steve Vinoski, in no particular order. These are all in erlang, but that’s not really important for NIFs anyway.</p>
<p>If people have other examples in Elixir – I expect Frank Hunleth has a bunch of amazing Nerves ones. I’d hope they can chime in with some examples.</p>
<p>Paul’s authored jiffy, probably the most widely used NIF of all, Steve was instrumental in documenting and authoring the first working dirty scheduler code to my knowledge, and Jesper’s enacl NIF comes with excellent explanatory notes in addition.</p>
<ul>
<li><a href="https://github.com/davisp/jiffy" class="inline-onebox" rel="noopener nofollow ugc">GitHub - davisp/jiffy: JSON NIFs for Erlang · GitHub</a> &amp; <a href="https://github.com/apache/couchdb-erlfdb" class="inline-onebox" rel="noopener nofollow ugc">GitHub - apache/couchdb-erlfdb: Erlang API for FoundationDB · GitHub</a></li>
<li><a href="https://github.com/jlouis/enacl" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jlouis/enacl: Erlang bindings for NaCl / libsodium · GitHub</a> &amp; <a href="https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7" rel="noopener nofollow ugc">https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7</a></li>
<li><a href="https://github.com/basho/enm" class="inline-onebox" rel="noopener nofollow ugc">GitHub - basho/enm: Erlang driver for nanomsg · GitHub</a></li>
</ul>
<p>The first two are both used extensively and show IMO canonical examples of building cross-platform compilation solutions. The middle is really well documented, and the latter shows how you can ship C source as part of the code and have it compiled locally.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="194805" data-batch-url="/posts/batch_likers">
                        9
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/dirty-cpu-nif-blocking-scheduler/35504/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-194805" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="194805"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>