nathanl
I’ve got some jobs that are “stuck”. Eg, one is scheduled 20 hours ago and is in available on attempt 0 of 15. It’s part of a workflow and has deps, but searching meta.workflow_id:0198ec97-ad63-783d-9c6b-12f1c6a15728 in all states finds only this stuck job in available.
The queue has config global_limit: [allowed: 1, partition: [fields: [:args], keys: [:regression_model_id]]], local_limit: 5. Yesterday I got some of these jobs to run by temporarily bumping that to 2, but that’s not very safe. This is on oban_pro 1.6.3, oban 2.20.1, oban_met 1.0.3, oban_web 2.11.4. For config we’ve got notifier: Oban.Notifiers.PG.
Any suggestions to debug?
Where Next?
Trending in Questions
Other Trending Topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
sorentwo
The Oban version looks good, but a fix in Pro v1.6.4 may help with the stuck processing.
The fact that there’s a job in a workflow with deps that is marked
availablemeans all the deps have alreadycompleted(orcancelled/discardeddepending on your config).adamtharani
I just ran into this, where jobs were queued yesterday and they sat in the queue untouched, the root problem looked like the
partition_keyon the job records werenil, not sure how it could get into this scenario, running Oban Pro 1.6.2sorentwo
Was this following an upgrade from v1.5, and if so, did you upgrade to v1.5.4 first to have it pre-generate the partition keys as suggested in the upgrade guide?
Is the situation resolved now?
adamtharani
No I had done the upgrade a while ago actually. I resolved it by turning off the global limit to allow it to process the jobs with no partition_key(wasn’t sure how else to get through it). I did have it happen locally too with one job, but I have yet to figure out what scenario triggered it.
adamtharani
Hey @sorentwo,
Had this happen again, currently have about 6k Jobs sitting in a queue with no partition key, would appreciate some guidance on how to debug this.
Running: Oban v2.20.1 Oban.Web v2.11.4 Oban.Met v1.0.3 Oban.Pro v1.6.5
Thanks
sorentwo
There was a change made in Pro v1.6.7 to address this specific issue. Will you upgrade and give that a shot?
adamtharani
Ahh I missed that, upgrading now. Will report back
adamtharani
@sorentwo actually before I upgrade, even new jobs added to the queue have a null
partition_key, is there anything I should debug before making any changes?sorentwo
Ah, that shouldn’t be happening then. Where are you inserting the jobs from? What does the oban config look like in that app?
adamtharani
So these records in particular are actually created via the AshOban extension
I just tried creating a new job in that queue without using the AshOban extension and it does has a
partition_key,coincidentally creating one via the
AshObanextension usingAshOban.run_triggers(record, trigger)now has a partition_keySo it seems like it’s working now and I’ll try the upgrade, but wondering why it would even get into this scenario?Actually that only worked from one instance