<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="172623" data-post-id="172623">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dimitarvp" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/120/38664_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dimitarvp
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="akash-akya" data-post="10" data-topic="30157">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/akash-akya/48/18688_2.png" class="avatar"> akash-akya:</div>
<blockquote>
<p>Yes. I’m more inclined towards just adding additional <code>no_stdin</code> , <code>no_stdout</code> options and keep the same <code>stream!</code> interface.</p>
</blockquote>
</aside>
<p>That actually sounds much better than what I came up with.</p>
<aside class="quote no-group" data-username="akash-akya" data-post="10" data-topic="30157">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/akash-akya/48/18688_2.png" class="avatar"> akash-akya:</div>
<blockquote>
<p>Just to avoid adding more functions, which can be confusing. But we can have separate functions if that makes more sense.</p>
</blockquote>
</aside>
<p>Nope, no need IMO. You are right. Options are enough. Just make sure to document them well and all is good.</p>
<aside class="quote no-group" data-username="akash-akya" data-post="10" data-topic="30157">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/akash-akya/48/18688_2.png" class="avatar"> akash-akya:</div>
<blockquote>
<p>Agree, I want to ditch the odu and have everything in a single library. When I started odu, I was mostly experimenting, and keeping it separate seemed simpler. Rustler looks interesting, I’ll look into it as soon as I get some time <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">.</p>
</blockquote>
</aside>
<p>Have in mind that documentation is pretty sparse for <code>0.22-rc</code>, and that all Rustler guides out there use the older, much more verbose syntax. To get a good understanding on how to use the new and terser format, <a class="mention" href="/u/scrogson" rel="nofollow">@scrogson</a> (one of Rustler’s maintainers) recommended me to inspect his own projects for inspiration. <a href="https://github.com/scrogson/franz" rel="noopener nofollow ugc">His franz GitHub repo</a> got the job done for me and you can see the result in the sqlite3 DB adapter library that I am currently writing here: <a href="https://github.com/dimitarvp/xqlite/blob/master/native/xqlitenif/src/lib.rs" class="inline-onebox" rel="noopener nofollow ugc">xqlite/native/xqlitenif/src/lib.rs at main · dimitarvp/xqlite · GitHub</a></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="172623" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/excmd-communicate-with-external-programs-with-back-pressure/30157/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-172623" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="172623"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="173113" data-post-id="173113">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="akash-akya" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/akash-akya/120/18688_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  akash-akya
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group quote-modified" data-username="dimitarvp" data-post="9" data-topic="30157">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/48/38664_2.png" class="avatar"> dimitarvp:</div>
<blockquote>
<p>I am also interested in your opinion on:</p>
<ul>
<li><a href="https://github.com/saleyn/erlexec" rel="noopener nofollow ugc">https://github.com/saleyn/erlexec </a> (Erlang)</li>
<li><a href="https://github.com/ericentin/exexec" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ericentin/exexec: Execute and control OS processes from Elixir. · GitHub</a> (Elixir wrapper around <code>erlexec</code> )</li>
<li><a href="https://github.com/alco/porcelain" rel="noopener nofollow ugc">https://github.com/alco/porcelain </a> (Elixir)</li>
<li><a href="https://forum.elixirforum.com/t/rambo-run-your-command-send-eof-get-output/25052" rel="nofollow">Rambo: Run your command. Send EOF. Get output </a> (Elixir)</li>
</ul>
</blockquote>
</aside>
<p>The main issue which ExCmd tries to solve, which none of these libraries solve is having proper stream with back-pressure. In these libraries progress of the external command is uncontrolled.</p>
<p>Internally they all use the port and some sort middleware program for <strong>IO</strong>. We can never have back-pressure (or limit) using ports as the process mailbox is unbounded. ExCmd also uses port and a middleware program (<code>odu</code>) but it only uses port for controlling the external program <strong>not for IO</strong>. For IO it uses named pipe (FIFO) which is demand-driven.</p>
<p>One can just write the output to a file and read that from to “solve” this. But,</p>
<ul>
<li>This involves disk IO for writing and reading which add latency</li>
<li>We can not control external program speed</li>
<li>we have to cleanup these files properly which might not be trivial</li>
<li><em>and IMHO ergonomics is better with ExCmd approach as the stream is more composable</em></li>
</ul>
<p>Apart from these issues,</p>
<ol>
<li><code>erlexec</code>: It is more focused on orchestrating and linking the external command than on communicating with it. If erlexec accepts os pid for its functionality <em>maybe</em> we can use erlexec with ExCmd</li>
<li><code>porcelain</code>: There are few important issues like zombie process and not having the ability to forcefully kill</li>
<li><code>rambo</code>: similar to porcelain, but rambo does not allow streaming input to stdin so all input must be kept in memory or it has to be passed to the command by writing to a file. It collects output binary in memory by appending it which is not efficient</li>
</ol>
<p>I think when these libraries were created, they were not trying to solve the issues I mentioned, so I think its not correct to compare them. Please let me if anything is incorrect.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="173113" data-batch-url="/posts/batch_likers">
                        4
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/excmd-communicate-with-external-programs-with-back-pressure/30157/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-173113" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="173113"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-most-liked cat-most-liked" title="One of the top 3 liked posts in this thread!"></div>
  </section>
</div>
    <div class="postbit" id="176354" data-post-id="176354">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jayjun" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jayjun/120/7100_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jayjun
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<blockquote>
<p>It collects output binary in memory by appending it which is not efficient</p>
</blockquote>
<p>Fixed in Rambo 0.3.2. <img src="https://forum.elixirforum.com/images/emoji/apple/slightly_smiling_face.png?v=15" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="176354" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/excmd-communicate-with-external-programs-with-back-pressure/30157/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-176354" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="176354"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="176362" data-post-id="176362">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dimitarvp" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/120/38664_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dimitarvp
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="jayjun" data-post="14" data-topic="30157">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jayjun/48/7100_2.png" class="avatar"> jayjun:</div>
<blockquote>
<p>Fixed in Rambo 0.3.2. <img src="https://forum.elixirforum.com/images/emoji/apple/slightly_smiling_face.png?v=15" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Replaced by what behaviour?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="176362" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/excmd-communicate-with-external-programs-with-back-pressure/30157/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-176362" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="176362"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="176406" data-post-id="176406">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jayjun" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jayjun/120/7100_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jayjun
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Output is collected in IO lists now. I’m considering a <code>Stream</code> API too, but without backpressure since ports are limited as mentioned.</p>
<p>Anyway, don’t wish to hijack this thread. Continue Rambo discussions <a href="https://forum.elixirforum.com/t/rambo-run-your-command-send-eof-get-output/25052" rel="nofollow">here</a>.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="176406" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/excmd-communicate-with-external-programs-with-back-pressure/30157/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-176406" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="176406"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #15"></div>
  </section>
</div>
    <div class="postbit" id="181920" data-post-id="181920">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="akash-akya" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/akash-akya/120/18688_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  akash-akya
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<h2><a name="p-181920-v030httpsgithubcomakash-akyaex_cmd-1" class="anchor" href="#p-181920-v030httpsgithubcomakash-akyaex_cmd-1" aria-label="Heading link" rel="nofollow"></a><a href="https://github.com/akash-akya/ex_cmd" rel="noopener nofollow ugc">v0.3.0</a></h2>
<p>This is a major release with completely different approach to solve the back-pressure. Ditched named pipes in favor of slightly complicated protocol. And with that ExCmd no longer has <a href="https://github.com/akash-akya/ex_cmd/tree/cb8a5798d2e806640c7719637c42385e2d7757a8#please-read-this-before-proceeding" rel="noopener nofollow ugc">scheduler issuers</a>.</p>
<p>I did consider this approach before, but dropped after I hit a blocker. Thanks to <a class="mention" href="/u/ananthakumaran" rel="nofollow">@ananthakumaran</a> for clearing it up <img src="https://forum.elixirforum.com/images/emoji/apple/+1.png?v=15" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
<h3><a name="p-181920-changes-2" class="anchor" href="#p-181920-changes-2" aria-label="Heading link" rel="nofollow"></a>Changes</h3>
<ul>
<li>fixed beam <a href="https://github.com/akash-akya/ex_cmd/tree/cb8a5798d2e806640c7719637c42385e2d7757a8#please-read-this-before-proceeding" rel="noopener nofollow ugc">scheduler issues</a></li>
<li>demand-driven protocol for back-pressure instead of named pipes</li>
<li>few breaking changes to options</li>
<li>many internal changes for maintainability. Such as, <code>ExCmd.Process</code> now uses <code>GenSteteMachine</code> instead of <code>GenServer</code></li>
</ul> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="181920" data-batch-url="/posts/batch_likers">
                        6
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/excmd-communicate-with-external-programs-with-back-pressure/30157/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-181920" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="181920"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-most-liked cat-most-liked" title="One of the top 3 liked posts in this thread!"></div>
  </section>
</div>
    <div class="postbit" id="181951" data-post-id="181951">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jayjun" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jayjun/120/7100_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jayjun
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Nice update <a class="mention" href="/u/akash-akya" rel="nofollow">@akash-akya</a>. I’ve decided on a protocol approach for my next update because I need a cross-platform solution, but you got there first. <img src="https://forum.elixirforum.com/images/emoji/apple/slightly_smiling_face.png?v=15" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="181951" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/excmd-communicate-with-external-programs-with-back-pressure/30157/18">Post #17</a>
	                </div>
	            </div>
              <div id="likers-container-181951" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="181951"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #17"></div>
  </section>
</div>
    <div class="postbit" id="185603" data-post-id="185603">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="akash-akya" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/akash-akya/120/18688_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  akash-akya
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Nice side effect removing fifo dependency is windows support. Finally, got time to test and confirm after a minor fix.</p>
<h2><a name="p-185603-v041httpsgithubcomakash-akyaex_cmd-1" class="anchor" href="#p-185603-v041httpsgithubcomakash-akyaex_cmd-1" aria-label="Heading link" rel="nofollow"></a><a href="https://github.com/akash-akya/ex_cmd" rel="noopener nofollow ugc">v0.4.1</a></h2> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="185603" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/excmd-communicate-with-external-programs-with-back-pressure/30157/19">Post #18</a>
	                </div>
	            </div>
              <div id="likers-container-185603" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="185603"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>