<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="48648" data-post-id="48648">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Trevoke" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Trevoke/120/1160_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Trevoke
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Okay, so my <em>actual</em> problem is that I am receiving webhooks. Each of those webhooks needs to go through a series of steps, a couple of HTTP requests, etc. Right now I just have those webhooks dumped to a directory, a producer produces them, a consumersupervisor spawns a task for each webhook it receives.</p>
<p>Unfortunately some of it is a little dirty because I get more files I have to manage, and it seemed like one new “workflow step” per new file might make sense and keep the code as readable as possible.</p>
<p>And this is where I am.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="48648" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/genstage-producerconsumersupervisor/7356/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-48648" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="48648"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="48693" data-post-id="48693">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="peerreynders" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/120/5826_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  peerreynders
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Hopefully you’ll bear with me because I’m not entirely sure I’m grasping all the essential details from your description -  so I’m just going to describe what I think you are telling me - with the full expectation that you are going have to correct me.</p>
<h3><a name="p-48693-high-level-goal-1" class="anchor" href="#p-48693-high-level-goal-1" aria-label="Heading link" rel="nofollow"></a>High level goal</h3>
<p>Some kind of “domain event” occurs and zero to many URIs need to be notified (my understanding of webhooks).</p>
<h3><a name="p-48693-complication-2" class="anchor" href="#p-48693-complication-2" aria-label="Heading link" rel="nofollow"></a>Complication</h3>
<p>The domain event doesn’t actually include all the information necessary for the notifications to be posted. The supplementary information needs to be fetched from multiple sources - preferably concurrently. (I’m assuming that the supplementary information depends on information from the domain event rather the URI the notification is posted to).</p>
<h3><a name="p-48693-observations-3" class="anchor" href="#p-48693-observations-3" aria-label="Heading link" rel="nofollow"></a>Observations</h3>
<ul>
<li>I suspect that currently the producer essentially just combines the “domain event” with the collection of URIs that need to be notified - i.e. the “event” emitted by the producer is “domain event + URIs that need to be notified”.</li>
<li>From the description it doesn’t seem like GenStage’s back pressure regulation is an essential element in solving the problem - the primary motivation for using GenStage is 1) to cleanly separate/segregate all the steps necessary to turn “Domain Event + notification URIs” into “notifications being sent to all the URIs”, 2) have these steps in the pipeline potentially working progressively but concurrently on distinct instances of domain events (each with their own set of URIs).</li>
</ul>
<p>Now at this point I suspect that you are trying to insert a stage that is responsible for “concurrently fetching supplementary information for one single domain event (on behalf of all specified notification URIs)” - and this is where the disconnect happens for me for attempting to employ a ConsumerSupervisor for this responsibility:</p>
<ul>
<li>The ConsumerSupervisor is <a href="https://hexdocs.pm/gen_stage/ConsumerSupervisor.html#moduledoc" rel="noopener nofollow ugc">designed to</a> “<em>start a (i.e. <strong>one</strong>) new child process <strong>per event</strong></em>” and it seems to me that for concurrent fetches you need to “<em>start multiple new child processes <strong>per event</strong></em>”.</li>
<li>By design the ConsumerSupervisor never had to combine the results from the various child processes - however the results of the concurrent fetches need to be aggregated back into the original domain event data.</li>
</ul>
<p>So where are my misunderstandings?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="48693" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/genstage-producerconsumersupervisor/7356/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-48693" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="48693"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="48772" data-post-id="48772">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Trevoke" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Trevoke/120/1160_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Trevoke
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Hi there, thanks for taking the time!</p>
<p>My high-level goal is: “I receive notifications all the time and I need to do work on each of those notifications before I can ingest them. This work requires a number of HTTP calls to be made”.</p>
<p>Currently as you have stated I do not need backpressure. In fact, most of the time, I do not want to process these events as batches, I want to process them as they arrive, hence the appeal of the consumer supervisor: it automatically creates one worker per event.</p>
<p>Frankly, as I chat with you, I realize that I’m trying to make the problem more complex than it needs to be right now, I just need to make a small change to what I currently have to be much happier with the way it works.</p>
<p>We’ll probably want some backpressure for some things as the influx grows, but there’s a number of things that we’ll just want to be able to send through very quickly anyway.</p>
<p>Well, thank you for helping me think about things and convincing me to not try and bend GenStage out of shape <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="48772" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/genstage-producerconsumersupervisor/7356/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-48772" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="48772"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="48774" data-post-id="48774">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="peerreynders" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/peerreynders/120/5826_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  peerreynders
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<blockquote>
<p>##<span class="hashtag-raw">#Rubber</span> Ducking<br>
A very simple but particularly useful technique for finding the cause of a problem is simply to explain it to someone else. The other person should look over your shoulder at the screen, and nod his or her head constantly (like a rubber duck bobbing up and down in a bathtub). They do not need to say a word; the simple act of explaining, step by step, what the code is supposed to do often causes the problem to leap off the screen and announce itself.‡</p>
</blockquote>
<blockquote>
<p>It sounds simple, but in explaining the problem to another person you must explicitly state things that you may take for granted when going through the code yourself. By having to verbalize some of these assumptions, you may suddenly gain new insight into the problem.</p>
</blockquote>
<blockquote>
<p>‡. Why “rubber ducking”? While an undergraduate at Imperial College in London, Dave did a lot of work with a research assistant named Greg Pugh, one of the best developers Dave has known. For several months Greg carried around a small yellow rubber duck, which he’d place on his terminal while coding. It was a while before Dave had the courage to ask…</p>
</blockquote>
<p>p. 95 - <a href="https://pragprog.com/book/tpp/the-pragmatic-programmer" rel="noopener nofollow ugc">The Pragmatic Programmer</a></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="48774" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/genstage-producerconsumersupervisor/7356/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-48774" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="48774"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>