benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

I mean advisory locks are ultimately rows too. Managing a job ownership table isn’t gonna produce a whole lot more garbage than dead advisory lock rows. Both will produce some but ultimately we need to just trust VACUUM to do its job properly. It may also be possible to simply put the node information in the job row itself, I need to think through the details some.

EDIT: One option that would help with this specific issue is to just have a single dedicated process per node hold the locks. This probably isn’t performant enough however. I can get away with it in Fable because locks are only acquired on boot and then held for the whole time the node is up. The locks here are per job right?

2
Post #82
sorentwo

sorentwo OP

Oban Core Team

Right, they are taken whenever a job is moved to the executing stage.

An important additional note/requirement is that the solution works across languages. I will implement Ruby and Python versions (and possibly others) and they need to have compatibility.

0
Post #83
benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Ruby / python job runners? Or do they simply need to be able to enqueue jobs.

EDIT: I guess it doesn’t really matter, all of this is just bare postgres anyway. Honestly a set of node tracking tables is probably easier to manage than the advisory locks in other languages cause it’s hard to deal with the locks when you start using something like pg bouncer.

1
Post #84
sorentwo

sorentwo OP

Oban Core Team

Full job runners, meaning they would take locks, gossip, everything.

Enqueuing jobs is really straight forward, but it only works when all the jobs are in Elixir.

0
Post #85
sorentwo

sorentwo OP

Oban Core Team

I’ve released Oban v0.8.0, AKA “Ben Wilson Edition”. This is a pretty big release that contains one critical change—the introduction of an table to track heartbeats and dropping the use of advisory locks entirely. This will reduce the amount of log noise (rampant warnings) that we all see, and it improves the reliability of job recovery as well.

If all goes well this is the last big set of changes before 1.0.0 :sunny:

From the CHANGELOG

Added

  • [Oban.Job] Add an attempted_by column used to track the node, queue and producer nonce that attempted to execute a job.

  • [Oban.Beat] Add a new oban_beats table, used by producers to publish “pulse” information including the node, queue, running jobs and other information
    previously published by gossip notifications.

    Beat records older than one hour are pruned automatically. Beat pruning respects the :disabled setting, but it ignores length and age configuration. The goal is to prevent bloating the table with useless historic information—each queue generates one beat a second, 3,600 beat records per hour even when the queue is idle.

Changed

  • [Oban.Executor] Don’t swallow an ArgumentError when raised by a worker’s backoff function. @balexand

  • [Oban.Notifier] Remove gossip notifications entirely, superseded by pulse activity written to oban_beats.

  • [Oban.Query] Remove all use of advisory locks!

  • [Oban.Producer] Periodically attempt to rescue orphans, not just at startup. By default a rescue is attempted once a minute and it checks for any executing jobs belonging to a producer that hasn’t had a pulse for more than a minute.

Fixed

  • [Oban.Worker] Validate worker options after the module is compiled. This allows dynamic configuration of compile time settings via module attributes, functions, Application.get_env/3, etc.

  • [Oban.Query] Remove scheduled_at check from job fetching query. This could prevent available jobs from dispatching when the database’s time differed from the system time. @seanwash

  • [Oban.Migrations] Fix off-by-one error when detecting the version to migrate up from.

8
Post #86
cnck1387

cnck1387

How different is this compared to using something like broadway?

0
Post #87
benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

:heart_eyes:

Oban in general? I mean it’s a persistent job queue in the classical sense, whereas Broadway is more a data processing pipeline. The difference is that each item in a job queue is usually a discrete piece of work, whereas individual items in a data processing pipeline are usually part of some data stream. This has a lot of implications with respect to API, implementation, performance, etc.

3
Post #88
smaximov

smaximov

Is this (heartbeat/pulse tracking) an established/well known technique? I tried to google it but didn’t find relevant results (btw, the search query “postgres pulse tracking” yields the Oban Github repo as the first hit :slight_smile: )

2
Post #89
sorentwo

sorentwo OP

Oban Core Team

Many systems use heartbeats to indicate that they are still alive (i.e. Phoenix Channels), but they aren’t typically persistent. The technique used in Oban is derived from how Kiq/Sidekiq use heartbeats to track activity. Those libraries are built on Redis and use a complex system of hash objects and timeouts, while Oban is able to use a single table for tracking.

2
Post #90
AndyL

AndyL

Since using Oban in a couple apps, my Postgres logs grow by tens of gigabytes per day. I suppose it may be due to the heartbeat. The logs actually consumed all disk and crashed one of my servers. Now that I’m aware it’s easy to prevent. Oban works great, but pay attention to the size of your Postgres logs!

1
Post #91