Workflow dependency may not be respected in some cases if grafted workflow dynamically appends job

Special Sections Oban
, ,
1

Hi! Our team ran into an issue where a job would not respect a dependency on a grafted workflow, if this workflow dynamically appends job to itself. Here a short reproduction script:

defmodule WaitJob do
  use Oban.Pro.Worker, queue: :not_default
  
  @impl true
  def process(%{args: %{"i" => i}}) do
    IO.inspect(i, label: "START WAIT")
    Process.sleep(30_000)
    IO.inspect(i, label: "END WAIT")
    
    :ok
  end
end

defmodule SubWorkflow do
  use Oban.Pro.Worker

  alias Oban.Pro.Workflow

  def workflow() do
    Workflow.new()
    |> Workflow.add(:sub_a, new(%{}))
  end

  @impl true
  def process(job) do
    waits = Enum.map(1..5, &WaitJob.new(%{i: &1}))

    job
    |> Workflow.all_jobs()
    |> Workflow.append()
    |> Workflow.add_many(:long_wait, waits)
    |> Oban.insert_all()

    :ok
  end
end

defmodule MainWorkflow do
  use Oban.Pro.Worker

  alias Oban.Pro.Workflow

  def workflow() do
    Workflow.new()
    |> Workflow.add_graft(:a, &graft_a/1)
    |> Workflow.add(:b, new(%{}), deps: :a)
  end
  
  def graft_a(_) do
    SubWorkflow.workflow()
    |> Workflow.apply_graft()
    |> Oban.insert_all()
  end

  @impl true
  def process(_job) do
    IO.inspect("PARENT WORKFLOW COMPLETED")

    :ok
  end
end

iex(46)> MainWorkflow.workflow() |> Oban.insert_all()
"PARENT WORKFLOW COMPLETED"
START WAIT: 3
START WAIT: 1
START WAIT: 4
START WAIT: 2
START WAIT: 5
END WAIT: 3
END WAIT: 1
END WAIT: 4
END WAIT: 2
END WAIT: 5

From my testing, this only seems to happen if appended jobs use a different queue.

1 Like
2

Looking at 1.6.10 changelog, this one should be fixed. But after removing the workaround in our app I don’t see any difference in behaviour :thinking: I tried the reproduction script from the initial post and it behaves the same way.

@sorentwo could you please double check if I’m missing something here?

3

@martosaur Strange :thinking:. We based the fix on a minimal reproduction of your reproduction, which did initially fail before the fixes. Taking another look for v1.6.11

1 Like