Oban add_workflow/4 with put_context/2 in a sub-workflow

Special Sections Oban
, ,
1

Environment

  • Oban Pro: 1.6.2
  • Oban: 2.19.4

The Issue

I have some workflows IngestionWorkflow and MapperWorkflow, each of which looks something like this:

defmodule MapperWorkflow do
  def build(source_name, opts \\ []) do

    # ...

    Workflow.new()
    |> Workflow.put_context(%{
      source_name: to_string(source_name),
      batch_size: batch_size,
      since: since
    })
    |> Workflow.add_graft(:mappers, &graft_mappers/1)
    |> Workflow.add_cascade(:summarize, &summarize/1,
      deps: :mappers,
      queue: :external_content_ingestion,
      recorded: true
    )
  end
end

I also have an orchestration workflow which works like this:

Workflow.new()
|> Workflow.add_workflow(
  :ingestion_workflow,
  IngestionWorkflow.build(source_name, opts)
)
|> Workflow.add_workflow(
  :mapper_workflow,
  MapperWorkflow.build(source_name, opts),
  deps: [:ingestion_workflow]
)
|> Oban.insert_all()

Each of the sub-workflows work perfectly fine on their own, but when run from the orchestration, I get this error:

** (RuntimeError) module is not a worker: Oban.Pro.Workers.Context

I assume this has something to do with the fact that Oban.Pro.Workers.Context is a virtual module registered like: Job{worker: Oban.Pro.Workers.Context}

I can kinda work around this by grafting in the workflow, but that shouldn’t be necessary, and it would be better to be able to use the sub-workflows via add_workflow/4

2

You’re correct. The Context job is virtual and shouldn’t ever run.

Are you seeing this issue in testing, or when running the jobs normally? We’ll take a look into it either way, but there are semantic differences :slightly_smiling_face:

3

I am seeing this issue in testing. I actually do not believe it appears when running the job normally, but I have refactored the child workflows to do something like this:

parent_workflow_id = Keyword.get(opts, :parent_workflow_id, nil)

workflow =
  case parent_workflow_id do
    id when is_binary(id) ->
      Workflow.new()

    nil ->
      Workflow.new()
      |> Workflow.put_context(%{
        source_name: to_string(source_name),
        batch_size: batch_size
      })
  end

And just make sure I have the orchestration workflow set the correct context. I can un-refactor it if you want me to check the difference between test/dev environments, but I do recall it works fine outside of test.

4

Are you testing with inline mode? If so, that would cause a problem. Workflows aren’t really compatible with inline mode because the jobs never hit the database. They are executed in memory in insert order, which doesn’t exercise the workflow dependencies at all.

Testing workflows should be done with the run_workflow/1 Pro testing helper, or manual draining from manual testing mode if needed.

1 Like
5

I am testing in :manual mode. An example test setup looks something like this:

Orchestration.kickoff(source_name, pipeline: :daily)
assert %{completed: 8} = drain_jobs()

Sub-workflows have been tested and work correctly like so:

MapperWorkflow.start(source_name)
assert %{completed: 2} = drain_jobs()

I’ll give run_workflow/1 a shot real quick and see if it solves the issue

1 Like
6

Okay I tried run_workflow/1 and also re-checked in dev mode with my workaround removed. Both cases actually act identical to testing in :manual mode with drain_jobs/1.

I.E.: ** (RuntimeError) module is not a worker: Oban.Pro.Workers.Context

When actually trying this out at runtime in dev mode, the workflow actually runs correctly, but the MapperWorkflow’s Oban.Pro.Workers.Context fails with the following meta:

%{
...
}

The MapperWorkflow then continues on to work correctly, I assume with the parent’s context (though I haven’t validated whether that is true)

7

Thanks for investigating and confirming it’s the same result with run_workflow. We’ll get it fixed :+1:

1 Like
8

Update on this—we were able to reproduce the issue and there’s a fix on main ready to go out with the next patch. Thanks again for such detailed investigation work!

1 Like
9

Great stuff thanks for looking into it!

1 Like