benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Ah I see now, if the node that first tries the job goes down while the job is in progress, other nodes can determine that the job is abandoned by observing that there is a job in the "executing" state that lacks a corresponding lock.

I wonder if this could be achieved by a dedicated table that tracks cluster membership. Each node, when started, inserts a new row into a "nodes" table that contains basically just an id, the node name for debugging, and a alive_at timestamp. The node needs a process that updates the timestamp at some regular interval. When you cut a job, record the id of the node row. Abandoned jobs are any job rows where the timestamp on the corresponding node row alive_at is older than now() - the interval by some appreciable amount. You can garbage collect the "nodes" table when the alive_at value is old and there are no associated jobs in the running or queued states.

Thoughts?

P.S.

To be clear, I don’t have any concerns about the correctness of the current implementation. However I’m just in the unfortunate position of already using advisory locks on 64bit auto incrementing ids. I think advisory locks can be reasonably used by people’s application logic, but in a library it’s tough because it is easy to become incompatible with any other library that also uses them.

3
Post #31
sorentwo

sorentwo OP

Oban Core Team

Precisely :+1:

I’m sure it could, this is essentially how Redis based tools work. They use two lists (sidekiq, verk) or a list and a hash (kiq) to hold a backup of each job while it executes.

What you’ve described is exactly how sidekiq and kiq handle tracking live nodes. It definitely works in that environment. I eschewed that in favor of advisory locks and pubsub to simply bookkeeping.

This is an excellent point. Based on your points and some of the other discussion in this thread I’m moving to the “namespaced” double int version.

Thanks for all the thought and feedback.

3
Post #32
benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

This seems like it would work, just make sure to let people pick the namespace so that they can ensure it doesn’t happen to be the same namespace they’ve picked for some other thing. This is also important if you had multiple copies of Oban running on the same database for some reason

4
Post #33
LostKobrakai

LostKobrakai

I just saw the docs for implementing migrations for oban:

defmodule MyApp.Repo.Migrations.AddObanJobsTable do
  use Ecto.Migration

  defdelegate up, to: Oban.Migrations
  defdelegate down, to: Oban.Migrations
end

Generally I’m an advocate for immutable migrations, so I’m wondering what would happen if you release a new version with different database needs. Is everything in there idempotent so I can at least add another migration doing the same? But really I’d prefer something, where I can lock my migration to a certain version of your migration script.

8
Post #34
sorentwo

sorentwo OP

Oban Core Team

Excellent point. While it was pre-1.0 I hadn’t expected the migrations to be immutable, but I can see how that is a concern.

I’m modifying the migration mechanism to support versions for the next release. There won’t be any breaking migrations though you may need to update some names in the older migrations.

5
Post #35
sorentwo

sorentwo OP

Oban Core Team

I’ve made some changes to namespace the advisory locks. The namespace is based on the oid of the oban_jobs table, which is unique per database (and actually changes if you create/drop a table repeatedly). This may not be the final solution as I’m working to make the prefix and possibly the table name more flexible, but it eliminates the likely intersection with application level advisory locks.

Here is the commit for the curious. There is a bit more detail in the comments and CHANGELOG: Use the two integer variant of advisory locks · oban-bg/oban@461060f · GitHub

3
Post #36
sorentwo

sorentwo OP

Oban Core Team

Oban v0.3.0 has been released. It includes a number of fixes and improvements that came directly from the conversations in this thread, so thank you all! :heart:

Directly from the CHANGELOG:

Added

  • [Oban] Allow setting queues: false or queues: nil to disable queue
    dispatching altogether. This makes it possible to override the default
    configuration within each environment, i.e. when testing.

    The docs have been updated to promote this mechanism, as well as noting that
    pruning must be disabled for testing. (@yogodoshi)

  • [Oban.Testing] The new testing module provides a set of helpers to make
    asserting and refuting enqueued jobs within tests much easier. (@bamorim)

Changed

  • [Oban.Migrations] Explicitly set id as a bigserial to avoid mistakenly
    generating a uuid primary key. (@arfl)

  • [Oban.Migrations] Use versioned migrations that are immutable. As database
    changes are required a new migration module is defined, but the interface of
    Oban.Migrations.up/0 and Oban.Migrations.down/0 will be maintained.

    From here on all releases with database changes will indicate that a new
    migration is necessary in this CHANGELOG. (@LostKobrakai)

  • [Oban.Query] Replace use of (bigint) with (int, int) for advisory locks.
    The first int acts as a namespace and is derived from the unique oid value
    for the oban_jobs table. The oid is unique within a database and even
    changes on repeat table definitions.

    This change aims to prevent lock collision with application level advisory
    lock usage and other libraries. Now there is a 1 in 2,147,483,647 chance of
    colliding with other locks. (@benwilson512)

  • [Oban.Job] Automatically remove leading “Elixir.” namespace from stringified
    worker name. The prefix complicates full text searching and reduces the score
    for trigram matches.

Note: When upgrading a migration is required.

v0.3.0 Docs
Testing Changes

12
Post #37
jc00ke

jc00ke

I’d love to use the UI as well, and contribute where I can.

@sorentwo any plans on releasing a preview soon?

Thanks for your work on this: I just switched some stuff over to Oban in a couple of hours. :+1:

Any plans on adding batches, like Sidekiq’s batches? I think this is one of the most powerful abstractions I’ve used for years.

2
Post #38
sorentwo

sorentwo OP

Oban Core Team

That’s great to hear! I’m glad it was a smooth process.

Yes, I hope to have a preview version of the UI ready by mid June. There are some essential features that are lacking currently. Once those are implemented and a few bugs are worked out it will be ready to try.

There are a few other features that I plan on tackling first (most of which overlap with the Sidekiq Enterprise feature set):

  • Expiring Jobs
  • Periodic Jobs (like cron jobs)
  • Rate Limiting
  • Dampeners (automatic queue scaling based on mem/cpu usage)

Batches are a great addition to the list!

3
Post #39
jaimeiniesta

jaimeiniesta

Oban looks great! There are 2 features that I miss in Exq and I’m not sure if they can be achieved in Oban:

1.- Avoiding duplicate jobs. That is, being able to check if a job has already been enqueued with the same arguments so I can avoid enqueueing it twice.

2.- Tagging jobs so they can be found quickly. My main use would be tagging the related jobs to some entity - if that is destroyed then I want to kill all the pending jobs for it easily.

3
Post #40