Oban Web/Met counting jobs twice in a scenario with separate Web and Worker nodes

Special Sections Oban
, ,
1

Hello! I need help setting up Oban Web with separate Web and Worker nodes.

My scenario is:

  • The Worker app runs the ObanRunner instance with all our queues.
  • The Web app has a separate ObanWatcher instance that I’m passing as the :oban_name option to Oban.Web.Router.oban_dashboard/2; This instance is configured with queues: false since it’s not a worker node.
  • Both the ObanWatcher and ObanRunner instances have Oban.Met in the plugins list and auto-start is disabled globally.
  • The Worker and Web apps are released/deployed separately.

The problem is that:

  • If the Worker and Web nodes are clustered together, the job counts in the ‘States’ sidebar component are shown doubled (i.e. if there are 10k completed jobs in the database, it will show 20k in the dashboard).
  • If the Worker and Web nodes are not clustered together, the counts are accurate but the ‘Nodes’ and ‘Queues’ sidebar components won’t show (I presume because of the queues: false option).

This seems to be an issue in the Oban.Met package, as a similar scenario using the previous Oban.Web.Plugins.Stats plugin would work.

Is there anything I’m doing wrong? I’m aware that turning Worker into a Phoenix app so the dashboard can talk directly to the ObanRunner instance should work, but I would like to avoid that since each app serves a different purpose.

PS: I already tried switching to the PG notifier (with the two nodes in the same cluster) and I also confirmed it’s not related to the estimate function; I didn’t find any other possible reasons in the documentation for oban_web or oban_met.

2

You aren’t doing anything wrong, but you’ve hit a quirk of having two instances with different names pointed at the same table and on the same network. The solution is to set peer: false on the Web nodes to prevent them from having leadership.

Internally, Met will report counts from each leader. The other instances will record those values and sum them together. If Web and Worker used the same instance name, e.g. Oban, only one would be leader and this wouldn’t be an issue.

It’s kind of a feature that instances with different names can share metrics, but it can be confusing and Met shouldn’t be summing the values regardless.

3

Thanks for the quick and thorough response! It works perfectly now. :purple_heart:

1 Like