kwando

kwando

I would have a “Setup / Start”-task in my supervision three that just query the database to see if there is a clear_session job or not and queue a new one if there is none.

But in this particular case I would say, keep it in a GenServer.. much less complexity.

0
Post #112
sorentwo

sorentwo OP

Oban Core Team

Regardless of whether you should do it this way, you should be able to do ti this way. In this case you need to use a unique argument, say the nearest minute as unix time. Something like this would round off the edges:

DateTime.utc_now()
|> DateTime.add(60)
|> Map.put(:second, 0)
|> DateTime.to_unix()

Until the next minute you’ll keep getting the same value, so uniqueness is enforced within the current minute.

I tend to agree with you on that. Quantum in particular is a large library.

Yes, I would do exactly that.

For something that is clearing expired sessions and doesn’t need to be coordinated to ensure it only runs on a single host/node, putting it in a GenServer is fine. If you are running multiple nodes and you only want to execute a job once within a window of time it is pretty sensible to use background jobs to do it.

  1. No matter how many nodes you are running the job will only execute on a single node
  2. If your node crashes, or you restart after a deploy the task will run again off-schedule. With unique scheduled jobs you can maintain uniqueness between nodes and restarts.
1
Post #113
sorentwo

sorentwo OP

Oban Core Team

Oban v0.9.0 has been released with a few important bug fixes and a couple of minor feature additions. Definitely upgrade if you’ve ran into trouble in CI with multiple unscoped Oban.Migration calls.

Thanks to everybody that has been using the library and reporting issues! :yellow_heart:

From the CHANGELOG

Added

  • [Oban] Add insert_all/2 and insert_all/4, corresponding to Ecto.Repo.insert_all/3 and Ecto.Multi.insert_all/5, respectively. @halostatue

  • [Oban.Job] Add to_map/1 for converting a changeset into a map suitable for database insertion. This is used by Oban.insert_all/2,4 internally and is exposed for convenience.

Changed

  • [Oban.Config] Remove the default queue value of [default: 10], which was overriden by Oban.start_link/1 anyhow.

  • [Oban.Telemetry] Allow the log level to be customized when attaching the default logger. The default level is :info, the same as it was before.

Fixed

  • [Oban.Migrations] Prevent invalid up and down targets when attempting to run migrations that have already been ran. This was primarily an issue in CI, where the initial migration was unscoped and would migrate to the current version while a subsequent migration would attempt to migrate to a lower version. @jc00ke

  • [Oban.Job] Prevent a queue comparison with nil by retaining the default queue (default) when building uniqueness checks.

  • [Oban.Job] Set state to scheduled for jobs created with a scheduled_at timestamp. Previously the state was only set when schedule_in was used.

8
Post #114
jc00ke

jc00ke

:heart: Thanks for the quick fix + release of the migration issue! :heart:

1
Post #115
sorentwo

sorentwo OP

Oban Core Team

Oban v0.10.0 is published with a huge performance improvement, a change to the default logging behavior and a huge restructuring of notifications. The result is much faster (258,000 faster with extremely large queues, millions of unprocessed jobs) and more fault tolerant (able to run without a database connection and recover safely when the database comes back up).

This release does involve a migration and I highly recommend it.

From the CHANGELOG

Migration Optional (V5)

Tables with a lot of available jobs (hundreds of thousands to several million) are prone to time outs when fetching new jobs. The planner fails to optimize using the index available on queue, state and scheduled_at, forcing both a slow sort pass and an expensive bitmap heap scan.

This migration drops the separate indexes in favor of a a single composite index. The resulting query is up to 258,757x faster on large tables while still usable for all of the other maintenance queries.

History of the EXPLAIN ANALYZE output as the query was optimized is available here: 9Vh7 : Optimization for: Optimization for: Optimization for: plan #IpcK; plan #CEg; plan #xInd | explain.depesz.com

Changed

  • [Oban.Config] Change the default for verbose from true to false. Also, :verbose now accepts only false and standard logger levels. This change aims to prevent crashes due to conflicting levels when the repo’s log level is set to false.

Fixed

  • [Oban.Notifier] Restructure the notifier in order to to isolate producers from connection failures. Errors or loss of connectivity in the notification connection no longer kills the notifier and has no effect on the producers. @axelson
10
Post #116
mindreframer

mindreframer

Awesome! Huge thanks for such outstanding package and the continuing effort to improve it!

6
Post #117
cnck1387

cnck1387

I took a look through the readme file on github and didn’t see this but is it possible to have scheduled tasks that get repeated on a set interval? For example, every Monday at 2pm run X job (or any cron-like interval variant)?

1
Post #118
benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

I’ve used a combination of Quantum and Oban for this. Quantum takes cron config, and if the action is super fast, I just do it. If it may fail or can take a while, I have Quantum simply enqueue a job.

4
Post #119
cnck1387

cnck1387

Thanks. I am using Quantum too. I guess repeated scheduling is better off being kept as a third party dependency? Oban is so well put together that not supporting repeated scheduling seems like it’s probably by design?

1
Post #120
sorentwo

sorentwo OP

Oban Core Team

It is kind of by design. I have a separate package called “oban_cron” that I haven’t published yet, but it does exactly what you’re asking about. I’ve debated making it part of oban itself, but it works nicely as a separate package.

Quantum is really powerful, but it doesn’t make sense in every situation IMO. There are a couple of downsides that I hoped to address with a focused package:

  1. Work without connected nodes. Quantum uses node communication for leadership, to prevent double-enqueuing a job.
  2. Have a smaller footprint. Quantum is a lot of code when all you want to specify is {"0 14 * * MON", MyApp.SomeWorker} in some configuration.
6
Post #121