martosaur

martosaur

Can't fan out workflows with cascade functions if they have a context

Here’s an example workflow that has a container sub-workflow that fans out multiple sub-workflows each of which contains a cascade function (pls don’t judge me)

defmodule Single do
alias Oban.Pro.Workflow
  
  def workflow() do
    Workflow.new()
    |> Workflow.put_context(%{hello: "world"})
    |> Workflow.add_cascade(:foo, &cascade/1)
  end
  
  def cascade(_context) do
    IO.inspect("Single fan-out cascade complete")
  end
end

defmodule Mothership do
  alias Oban.Pro.Workflow
	
  def workflow() do
    fan_out_wf =
      Enum.reduce(1..2, Workflow.new(), fn i, wf ->
        Workflow.add_workflow(wf, to_string(i), Single.workflow())
      end)
  
    Workflow.new()
    |> Workflow.add_workflow(:fan_out, fan_out_wf)
  end
end

When I run it:

Mothership.workflow() |> Oban.insert_all()

Jobs fail with this error:

** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:

from j0 in Oban.Job,
  where: j0.state in [
  "scheduled",
  "available",
  "executing",
  "retryable",
  "completed",
  "discarded",
  "cancelled"
],
  where: fragment(
  "? ? 'workflow_id' AND ?->>'workflow_id' = ?",
  j0.meta,
  j0.meta,
  ^"019b1e4a-fb19-76d5-b41c-b4e83d2e79e0"
),
  where: j0.meta["name"] in ^["context"],
  order_by: [asc: j0.id],
  select: j0.meta

    (ecto 3.13.5) lib/ecto/repo/queryable.ex:166: Ecto.Repo.Queryable.one/3
    (oban_pro 1.6.9) lib/oban/pro/workflow.ex:2228: Oban.Pro.Workflow.get_recorded/3
    (oban_pro 1.6.9) lib/oban/pro/workflow/cascade.ex:39: Oban.Pro.Workflow.Cascade.all_context/2
    (oban_pro 1.6.9) lib/oban/pro/workflow/cascade.ex:23: Oban.Pro.Workflow.Cascade.process/1
    (oban_pro 1.6.9) lib/oban/pro/worker.ex:1156: Oban.Pro.Worker.process/3
    (oban 2.20.2) lib/oban/queue/executor.ex:143: Oban.Queue.Executor.perform/1
    (oban 2.20.2) lib/oban/queue/executor.ex:75: Oban.Queue.Executor.call/1
    (elixir 1.19.3) lib/task/supervised.ex:105: Task.Supervised.invoke_mfa/2
    (elixir 1.19.3) lib/task/supervised.ex:40: Task.Supervised.reply/4

Without put_context it’s working fine. Seems like a bug?

Marked As Solved

martosaur

martosaur

Just checked this against 1.6.10 and this seems to be fixed

Also Liked

sorentwo

sorentwo

Oban Core Team

It’s likely that we can put in a fix to prevent the error, but based on the example, you’re using it slightly wrong. There’s no need to add a workflow to a workflow without any other jobs in it, just return the workflow:

  def workflow do
    Enum.reduce(1..2, Workflow.new(), fn i, wf ->
      Workflow.add_workflow(wf, to_string(i), Single.workflow())
    end)
  end

Fewer nested workflows are better when possible. We’ll work on a fix for the query issue regardless.

martosaur

martosaur

Thanks! I purposefully removed from the example as much as possible. The full version contains more stages, and looks more like this:

  def workflow(task_ids) do
    fan_out_wf =
      Enum.reduce(task_ids, Workflow.new(), fn task_id, wf ->
        Workflow.add_workflow(wf, task_id, Single.workflow(task_id))
      end)
  
    Workflow.new()
    |> Workflow.add(:prepare_fan_out, ...)
    |> Workflow.add_workflow(:fan_out, fan_out_wf, deps: :prepare_fan_out)
    |> Workflow.add(:process_results, ..., deps: :fan_out)
  end

The reason I wrap fan-out in a workflow is so that I could depend on the fan-out without explicitly collecting every fan out job id at workflow creation time, if this makes sense!

sorentwo

sorentwo

Oban Core Team

Thanks for the reminder! It’s out now as v1.6.13

Where Next?

Popular in Questions Top

9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New

Other popular topics Top

malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31265 143
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 48342 226
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 127089 1222
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54120 245
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement