<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="117673" data-post-id="117673">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="mikemccall" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  mikemccall
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>This was also just announced today</p>
<p><a href="https://github.com/plataformatec/broadway" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/plataformatec/broadway</a></p>
<p>I think if you look at the amount of <strong>good</strong> available options in the ecosystem as you have listed, it’s clear there is a need for background job queues with a lot of different ways to tackle it. I typically lean towards SQS because it’s cheap, durable, at least once deliverability, widely supported, and it’s managed. (… it can also trigger lambdas.. <img src="https://forum.elixirforum.com/images/emoji/apple/sunglasses.png?v=15" title=":sunglasses:" class="emoji" alt=":sunglasses:" loading="lazy" width="20" height="20">). My interest in elixir has made me take a deeper look at RabbitMQ. In addition to the obvious purpose as a queue, I like to think of a background job queue as message passing in a language/application agnostic way. i.e.</p>
<aside class="quote no-group" data-username="PragTob" data-post="1" data-topic="20436">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/pragtob/48/2401_2.png" class="avatar"> PragTob:</div>
<blockquote>
<p>I want to <strong>notify a partner system</strong> of something</p>
</blockquote>
</aside>
<p>It makes it easy to introduce new services/technology such as migrating to elixir or use a more fitting technology based on the queue.</p>
<aside class="quote no-group" data-username="PragTob" data-post="1" data-topic="20436">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/pragtob/48/2401_2.png" class="avatar"> PragTob:</div>
<blockquote>
<p>What are other scenarios where we should reach for a background job processing system?</p>
</blockquote>
</aside>
<p>90% of our needs for a background job processing system is image processing.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="117673" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-117673" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="117673"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="117881" data-post-id="117881">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="PragTob" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/PragTob/120/2401_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  PragTob
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I think we’re mixing background job systems and message queues in this thread a bit. If I remember correctly Mike Perham (author of sidekiq, the ruby background job system that is excellent <img src="https://forum.elixirforum.com/images/emoji/apple/star.png?v=15" title=":star:" class="emoji" alt=":star:" loading="lazy" width="20" height="20">) defined the difference in <a href="https://devchat.tv/ruby-rogues/221-rr-sidekiq-with-mike-perham/" rel="noopener nofollow ugc">(Ithink) this ruby rogues episode</a> as follows: a baclground job system  usually is part of the application code (and is hence also written in the same language) while with message queues the consumer is often another application and also potentially in another language.</p>
<p>I know there’s exceptions (as we see with libraries that consume/produce the sidekiq format into redis) or some applications that I know which go through kafka or Message Queues but consume their own messages (for reasons of easy extraction and decoupling).</p>
<aside class="quote no-group" data-username="mikemccall" data-post="12" data-topic="20436">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/a587f6/48.png" class="avatar"> mikemccall:</div>
<blockquote>
<p>I think if you look at the amount of <strong>good</strong> available options in the ecosystem as you have listed, it’s clear there is a need for background job queues with a lot of different ways to tackle it.</p>
</blockquote>
</aside>
<p>There definitely is! However, honestly we have so many redis backed queues with a sidekiq format I wouldn’t know which one to pick. I haven’t really investigated them though. I’d personally be happy to have a clear first/main choice (maybe per data store) to concentrate efforts, have more community help etc.</p>
<p>In Ruby you just use Sidekiq and you’re usually fine.</p>
<aside class="quote no-group" data-username="mikemccall" data-post="12" data-topic="20436">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/a587f6/48.png" class="avatar"> mikemccall:</div>
<blockquote>
<p>It makes it easy to introduce new services/technology such as migrating to elixir or use a more fitting technology based on the queue.</p>
</blockquote>
</aside>
<p>That’s good when the system is yours, my concrete example was meant differently: the system belong to customers or companies we collaborate with (we call both partners usually). A system I maintain usually won’t be done for hours, not the same for some of their old legacy systems <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="mikemccall" data-post="12" data-topic="20436">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/m/a587f6/48.png" class="avatar"> mikemccall:</div>
<blockquote>
<p>90% of our needs for a background job processing system is image processing.</p>
</blockquote>
</aside>
<p>This is very interesting to me - why is that? I assumed you could do image processing just in elixir through processes. Is it that you sometimes get so many of these requests at once so that it’d create too much load on the system? Limited I/O? So that something with a pool of workers where work can pile up more directly is better? Do you need to absolutely make sure none of the jobs gets lost? Or do you have a separate service for image processing (and you use a message queue) so that it’s done in a more efficient language?</p>
<p>edit: haven’t listened to it yet but found <a href="https://dev.to/changelog/274-faktory-and-the-future-of-background-jobs-with-mike-perham" rel="noopener nofollow ugc">Mike talking about Faktory</a></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="117881" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-117881" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="117881"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="118071" data-post-id="118071">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="sorentwo" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/sorentwo/120/37360_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  sorentwo
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Oban Core Team</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I’m the author of Kiq and though I could shed some light into why I wrote it when there were already some other redis backed sidekiq formatted job queues out there.</p>
<aside class="quote no-group" data-username="PragTob" data-post="13" data-topic="20436">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/pragtob/48/2401_2.png" class="avatar"> PragTob:</div>
<blockquote>
<p>There definitely is! However, honestly we have so many redis backed queues with a sidekiq format I wouldn’t know which one to pick. I haven’t really investigated them though. I’d personally be happy to have a clear first/main choice (maybe per data store) to concentrate efforts, have more community help etc.</p>
<p>In Ruby you just use Sidekiq and you’re usually fine.</p>
</blockquote>
</aside>
<p>We have been using Sidekiq and Sidekiq Enterprise in an historic Rails app for many years. Once we started most new development in Elixir it became clear that we wanted to run our background jobs in Elixir as well. There was no way that we’d stop development and rewrite nearly a hundred workers in Elixir, so interop with Sidekiq (and features from Pro/Enterprise) was essential.</p>
<p>Some of the older job queues like Exq and Verk would have worked in isolation, but they didn’t fully integrate with Sidekiq. Some particular features that we required:</p>
<ol>
<li>Full integration with the Sidekiq Enterprise UI (running jobs, running worker nodes, stats)</li>
<li>Expiring jobs</li>
<li>Periodic jobs</li>
<li>Unique jobs</li>
<li>Reliable queues (particularly difficult due to inconsistencies between Ruby and Elixir JSON encoding)</li>
<li>Ad-hoc error reporting</li>
<li>Structured logging</li>
</ol>
<p>With all of those features in place we’ve been able to run queues in both platforms and slowly migrate workers over one at a time. Typically jobs execute faster and the workers perform more reliably, largely because each queue is managed in isolation. That prevents a bunch of slow jobs in one queue from backing up processing in all of the others.</p>
<hr>
<p>The internals of Sidekiq are rather scary and the miscellaneous use of data structures makes the system very complex. After Redis Streams were released I started on a fresh approach fully based on streams, which eventually led me to begin <em>another</em> job queue that is stream oriented, but backed by Postgres (it has similarities to Rhianna and EctoJob, but with some key differences). I have some writing to do about that one and I look forward to sharing the details soon.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="118071" data-batch-url="/posts/batch_likers">
                        4
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-118071" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="118071"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="118207" data-post-id="118207">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="supernova32" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/supernova32/120/6576_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  supernova32
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>While I was developing the very first version of <a href="https://alloy-ci.com" rel="noopener nofollow ugc">AlloyCI</a> I went with <code>exq</code> as it provided something familiar. Coming from the Ruby world, and just starting to learn Elixir, I thought it would be a good idea to use something similar to Sidekiq with similar functionality.</p>
<p>After further learning Elixir, I realized that it was a complete overkill for my use case to have Redis as a dependency for my system, so I ditched <code>exq</code> and replaced it with <code>que</code>.</p>
<p>I was extremely happy with it. It had no external dependencies and provided some persistence for the jobs. It worked perfectly, until Elixir 1.7 was released, then it just crashed every time. It was due to the <a href="https://github.com/meh/amnesia" rel="noopener nofollow ugc">Amnesia</a> wrapper around <code>mnesia</code>, it used private Elixir functions that had no guarantee of being supported in future releases, so it broke, and stayed broken for months.</p>
<p>At that point I decided to roll out my own background processor just for AlloyCI. It is very simple, and dumb, but does the trick, and is more than enough for my use case. You can check the code <a href="https://github.com/AlloyCI/alloy_ci/blob/master/lib/alloy_ci/lib/background_scheduler.ex" rel="noopener nofollow ugc">here</a>.</p>
<p>The author of <code>Que</code> eventually wrote his own wrapper around <code>mnesia</code> to make it compatible with Elixir 1.7, but at that point I did not require to use it anymore.</p>
<p>Now that I am thinking of adding more functionality to AlloyCI, I might need a background scheduler that can give me better assurances about my jobs. For that I would prefer something with no extra dependencies than the ones I already have, so having a PostgreSQL backed processor would be the ideal scenario for me. From the list provided, I think <code>ecto_job</code> might be the clear winner.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="118207" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-118207" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="118207"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="118211" data-post-id="118211">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="mikemccall" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  mikemccall
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="PragTob" data-post="13" data-topic="20436">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/pragtob/48/2401_2.png" class="avatar"> PragTob:</div>
<blockquote>
<p>This is very interesting to me - why is that? I assumed you could do image processing just in elixir through processes.</p>
</blockquote>
</aside>
<p>I should have mentioned 90% of our <strong>current</strong> background jobs processing is image processing. This process is using sidekiq and ruby on internal worker machines. The image processing is cpu intensive and some machines can handle more than others. They come in batches. Sometimes it’s 600 and sometimes it’s 60,000. So going through them all can take some time. I haven’t thought too deeply about how to handle it in elixir. But, I don’t see why we couldn’t remove the external dependencies.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="118211" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-118211" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="118211"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #15"></div>
  </section>
</div>
    <div class="postbit" id="118216" data-post-id="118216">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="jeremyjh" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/jeremyjh/120/5995_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  jeremyjh
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I thought I would just mention that we’ve been happy users of honeydew for many months in production now. One of the nice things about honeydew’s Ecto queues is that they can be implemented as just a couple of columns added to an existing table. This way you can be sure a given entity has only one job scheduled to run on it at at time, and you can do things like have the default of a new record to be schedule a job related to it. It is also helpful that job schedule is part of the same transaction as other operations you are doing in your application; so a job is only actually scheduled if the transaction it is a part of goes through.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="118216" data-batch-url="/posts/batch_likers">
                        7
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-118216" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="118216"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #16"></div>
  </section>
</div>
    <div class="postbit" id="118811" data-post-id="118811">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lucaong" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lucaong/120/21794_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lucaong
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I personally very much agree with <a class="mention" href="/u/pragtob" rel="nofollow">@PragTob</a>’s analysis of the reasons for/against using a background job queue.</p>
<p>In my company we use RabbitMQ exactly for the aforementioned use-cases: retry with exponential backoff, durable queues in presence of restarts, scheduling jobs into the future (with the <code>delayed-message-exchange</code> plugin). It is true that RabbitMQ is a messaging queue rather than a background job system, but I think it’s fair to consider the two concepts overlapping a bit, at least because a background job system is built upon some sort of message queue.</p>
<p>We find RabbitMQ very versatile: it might not be the fastest kid in the block (although it’s <em>very fast</em>), but its semantics allow for a wide variety of solutions and topologies. I think of it as a great general-purpose tool (we even use it as an MQTT broker for IoT telemetry), as opposed to specialized background job processors like Sidekiq-inspired tools, or more “use-case-optimized” tools like Kafka. The versatility also comes with a slightly steeper learning curve though, as one needs to learn how to use the provided building blocks, rather than a turn-key solution.</p>
<p>In addition to reasons dealing with job completion guarantees, another valid reason to introduce a queue could be as a decoupling device: if there is benefit in the producer of a job not having to know details about the consumer, and the two being developed and/or scaled independently, then a background queue can be a good solution.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="118811" data-batch-url="/posts/batch_likers">
                        3
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/18">Post #17</a>
	                </div>
	            </div>
              <div id="likers-container-118811" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="118811"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #17"></div>
  </section>
</div>
    <div class="postbit" id="126182" data-post-id="126182">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="bamorim" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/bamorim/120/11162_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  bamorim
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Really nice discussion here.<br>
Where I work we do care a lot about consistency and not losing any jobs, for that reason we use ecto_job (rihanna would work too).</p>
<p>One thing about consistency is that there is some problems other than losing messages. One of them is the fact that if you are in a database transaction and send a message, your message may be sent but your transaction may still fail. In that scenario, you’ve sent the message too early.<br>
To solve that there is a well-known pattern, the <a href="http://gistlabs.com/2014/05/the-outbox/" rel="noopener nofollow ugc">Outbox pattern</a>. In order to achieve that, I needed something that uses the same Ecto.Repo as my application and therefore I can run inside a transaction (when scheduling a new job).</p>
<p>If you need more throughput you may want to have your jobs do not handle database connection. So you may still want to do the heavy lifting using some SQS or RabbitMQ based job, so you can just have your “PSQL backed jobs” just schedule the job in other system.</p>
<p>Up until now, we are fine with PSQL bakcked job queue.</p>
<p>We also use Kafka to publish events (as our pubsub infrastructure), and we use EctoJob as a gateway to publish kafka events, so we avoid publishing events too early or losing events.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="126182" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/19">Post #18</a>
	                </div>
	            </div>
              <div id="likers-container-126182" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="126182"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #18"></div>
  </section>
</div>
    <div class="postbit" id="127102" data-post-id="127102">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="koudelka" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/koudelka/120/16420_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  koudelka
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><a href="https://github.com/koudelka/honeydew/tree/master/examples/ecto_poll_queue" rel="noopener nofollow ugc">Honeydew’s Ecto Queue</a> might be of interest to you.</p>
<p>It acts as a “follower” and keeps completely out of your insertion transaction. It only comes along later to look for rows that were successfully inserted, but haven’t yet executed a job.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="127102" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/20">Post #19</a>
	                </div>
	            </div>
              <div id="likers-container-127102" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="127102"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #19"></div>
  </section>
</div>
    <div class="postbit" id="127166" data-post-id="127166">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="bamorim" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/bamorim/120/11162_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  bamorim
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><a class="mention" href="/u/koudelka" rel="nofollow">@koudelka</a>, honeydew looks amazing. I really like the way it just “merge into another table”. That makes async reaction to events a breeze. But I think that other job queues have some other advantages, such as a built in web interface (which is normally underrated)</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="127166" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/background-job-queues-when-to-use-when-not-to-use-which-one-to-use/20436/21">Post #20</a>
	                </div>
	            </div>
              <div id="likers-container-127166" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="127166"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #20"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/20436/load_more?page=3">Load more posts (5 remaining)</a>
</div></template></turbo-stream>