felix-starman

felix-starman

Bug or incorrect usage w/ sub-workflows from a graft with a `cascade_capture`

I have a reproduction below that I asked Claude to extract from our implementation, and seems to suffer the same issue.

The background is we have a workflow that loads data from our DB (populated from a previous job), and then for each item (for us it’s insurance plans) we need to fetch n child items (policies for the plan) from an API. There are ~200 plans and per plan there’s 300-2000 policies. Due to the API’s pagination, and the variability of the number of policies, we use a sub-workflow per plan.

What I’m seeing is that the sub-workflows never run, or insert into the database.

From the documentation, it seems like I should not be using Oban.insert_all in the sub-workflow defining function, as that would result in orphaned sub-workflows.

Am I misusing apply_graft?

defmodule NestedWorkflowRepro do
  @moduledoc """
  Minimal reproduction of multi-level nested workflow pattern using Oban.Pro.Workflow.

  This emulates the structure in PolicyBotPolicySync:
  - Level 1: Main workflow fetches items
  - Level 2: Graft per-item workflows
  - Level 3: Graft per-chunk workflows within each item workflow

  Each step returns simple concatenated strings for easy debugging.
  """

  use Oban.Pro.Worker, queue: :default, max_attempts: 1

  alias Oban.Pro.Workflow

  require Logger

  # Level 1: Main workflow functions

  @doc """
  Fetches the initial list of items to process.
  Returns %{items: ["A", "B", "C"]}
  """
  def fetch_items(_context) do
    items = ["A", "B", "C"]
    Logger.info("fetch_items: #{inspect(items)}")
    %{items: items, result: "fetched_items"}
  end

  @doc """
  Grafts a sub-workflow for each item.
  This is Level 2 grafting - creates per-item workflows.
  """
  def graft_per_item(%{fetch_items: %{items: items}}) do
    Logger.info("graft_per_item: creating workflows for #{length(items)} items")

    {items, &process_item_workflow/2}
    |> Workflow.apply_graft(context: %{batch_size: 2})
    |> Oban.insert_all()
  end

  # Level 2: Per-item workflow functions

  @doc """
  Creates a workflow to process a single item.
  This workflow will:
  1. Process initial batch
  2. Determine if more batches are needed
  3. If needed, graft sub-workflows for remaining batches (Level 3)
  """
  def process_item_workflow(item, %{batch_size: batch_size}) do
    workflow_name = "nested_workflow_repro_item_#{item}"

    Workflow.new(workflow_name: workflow_name)
    |> Workflow.put_context(%{
      item: item,
      batch_size: batch_size
    })
    |> Workflow.add_cascade(:process_initial_batch, &process_initial_batch/1)
    |> Workflow.add_cascade(:determine_remaining, &determine_remaining/1, deps: :process_initial_batch)
    |> Workflow.add_graft(:process_remaining_batches, &graft_remaining_batches/1,
      deps: [:process_initial_batch, :determine_remaining]
    )
  end

  @doc """
  Processes the initial batch for an item.
  Simulates work that determines total work needed.
  """
  def process_initial_batch(%{item: item, batch_size: batch_size}) do
    # Simulate: item "A" has 5 total chunks, "B" has 3, "C" has 1
    total_chunks =
      case item do
        "A" -> 5
        "B" -> 3
        "C" -> 1
      end

    processed = min(batch_size, total_chunks)

    result = "item_#{item}_initial_batch_#{processed}"
    Logger.info("process_initial_batch: #{result}")

    %{
      result: result,
      total_chunks: total_chunks,
      processed: processed,
      needs_more: processed < total_chunks
    }
  end

  @doc """
  Determines what remaining work is needed.
  """
  def determine_remaining(%{
        process_initial_batch: %{total_chunks: total, processed: processed, needs_more: needs_more},
        item: item
      }) do
    result = "item_#{item}_determined_#{total - processed}_remaining"
    Logger.info("determine_remaining: #{result}")

    %{
      result: result,
      remaining_chunks: total - processed,
      needs_more: needs_more
    }
  end

  @doc """
  Grafts sub-workflows for remaining batches if needed.
  This is Level 3 grafting - creates per-chunk workflows.
  """
  def graft_remaining_batches(%{
        process_initial_batch: %{processed: processed, total_chunks: total, needs_more: true},
        item: item,
        batch_size: batch_size
      }) do
    # Calculate chunks needed (similar to calculate_policy_chunks)
    chunks =
      Range.new(processed, total - 1, batch_size)
      |> Enum.map(fn idx -> %{chunk_index: idx, size: batch_size} end)

    Logger.info("graft_remaining_batches: item=#{item}, chunks=#{length(chunks)}")

    # Level 3: Graft per-chunk workflows
    {chunks, &process_chunk/2}
    |> Workflow.apply_graft()
    |> Oban.insert_all()
  end

  # No additional chunks needed
  def graft_remaining_batches(%{process_initial_batch: %{needs_more: false}, item: item}) do
    Logger.info("graft_remaining_batches: item=#{item}, no additional chunks needed")

    # Return empty workflow
    Workflow.new()
    |> Oban.insert_all()
  end

  # Level 3: Per-chunk workflow functions

  @doc """
  Processes a single chunk within an item's workflow.
  """
  def process_chunk(%{chunk_index: idx, size: size}, %{item: item}) do
    result = "item_#{item}_chunk_#{idx}_size_#{size}"
    Logger.info("process_chunk: #{result}")

    %{result: result}
  end

  @doc """
  Builds the main workflow.

  This creates a 3-level nested workflow structure:
  - Level 1: Main workflow (fetch items -> graft per-item)
  - Level 2: Per-item workflows (process initial -> graft per-chunk)
  - Level 3: Per-chunk workflows (process individual chunks)
  """
  def workflow(opts \\ []) do
    batch_size = Keyword.get(opts, :batch_size, 2)

    Workflow.new(workflow_name: "nested_workflow_repro_main")
    |> Workflow.put_context(%{batch_size: batch_size})
    |> Workflow.add_cascade(:fetch_items, &fetch_items/1)
    |> Workflow.add_graft(:process_items, &graft_per_item/1, deps: [:fetch_items])
  end

  @impl Oban.Pro.Worker
  def process(%Oban.Job{args: args}) do
    batch_size = Map.get(args, "batch_size", 2)

    workflow(batch_size: batch_size)
    |> Oban.insert_all()

    :ok
  end
end

Most Liked

sorentwo

sorentwo

Oban Core Team

You need to use insert_all after apply_graft or the jobs won’t be inserted at all.

This is because the term (whatever the struct name is) isn’t available on the Web node and it’s an unsafe action to render it. You can override that with a custom resolver ( Oban.Web.Resolver — Oban Web v2.12.5 ). It can also happen in development mode because modules are loaded lazily.

That effectively grafts multiple workflows like they are one, which causes that MultipleResultsError as you saw.

We’re working through your top example to see if there’s an alternate version that will work, or if there’s a bug to fix.

Last Post!

felix-starman

felix-starman

@sorentwo I saw 1.6.8 came out recently and mentions fixes for nested graphs :tada:

I haven’t had a chance to test it yet, but I will try to poke this soon w/ 1.6.8

Where Next?

Popular in Questions Top

minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" =&gt; #BSON.ObjectId&lt;58eb1a7a9ad169198c3dXXXX&gt;, "email" =&gt; ...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
Lily
In templates/appointment/index.html.eex: &lt;%= for appointment &lt;- @appointments do %&gt; &lt;tr&gt; &lt;td&gt;&lt;%= appoi...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Other popular topics Top

minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 31494 112
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 40042 209
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

We're in Beta

About us Mission Statement