@outlog indeed, currently there’s no guarantee when it comes to job order. Nice suggestion for a future release, thanks!
mspanc
Membrane Core Team
mspanc
Membrane Core Team
@fishcakez thank you for great and deep review. Wow!
I am aware of these limitations. I had to write this library because I quickly needed some replacement for exq/toniq due to some fu***up in one project where we needed to reprocess tens of thousands of files and none of the alternatives were reliable enough to at least reach half of the queue…
So while I agree that there are some corner cases that might affect queue behaviour, they are, well, 1% of cases that I wasn’t intending to cover in the very first release, which does not mean they won’t be improved in the future releases. In real life, in most of the apps jobs are not things that affect the process tree, they are rather simple tasks, such as sending mails, processing some data etc. It is highly unlikely that someone will call exit(:normal) in a job. Perfect Library should obviously handle such cases but I was time constrained so I wanted to make Good Enough For 99% Cases Library. At least for 1.0.0.
According to my own experience even now Jumbo is way more stable and faster than Exq/Toniq so it fits into Good Enough principle 
Callbacks are on the TODO list.
0
fishcakez
Ecto Core Team
I think you should handle the first issue even if you ignore the rest. That is the assumption that an exit of the form {_, _} is an exception. This is an invalid assumption and will catch many exit/1 calls that are far from edge cases. Formatting errors incorrectly when we have the Exception is inexcusable.
0
mspanc
Membrane Core Team
@fishcakez let me clarify, so I am sure that I understand you correctly.
When you write about {_,_} you mean improper callback handler in jumbo/lib/jumbo/queue.ex at bfed4e62b2dc432e7c150f886ecf1565bb51991c · mspanc/jumbo · GitHub, right?
When you write about formatting, you mean using http://elixir-lang.org/docs/v1.3/elixir/Exception.html#format/3 and similar functions, right?
0
fishcakez
Ecto Core Team
I meant specifically the pattern on this line: jumbo/lib/jumbo/queue.ex at bfed4e62b2dc432e7c150f886ecf1565bb51991c · mspanc/jumbo · GitHub and the other :DOWN patterns (but they are being handled ok). We have Exception.format_exit(reason) or Exception.format({:EXIT, pid}, reason) that you should use to format the reason from the :DOWN tuple.
0
fishcakez
Ecto Core Team
mspanc
Membrane Core Team
sasajuric
Author of Elixir In Action
I believe you might be able to delegate persistency, as well as restarting of failed jobs, to library users by supporting callbacks. If you invoke a custom callback function when the job is about to be started, and after it finishes or crashes, it would give users the chance to store job state into an arbitrary database, and/or requeue jobs if they crash.
That would keep the job queue logic pretty simple, and yet provide a lot of flexibility.
6