tovarchristian21
Using Oban on functions that return
Hello guys, I’ve been checking out Oban for implementing some processing queues for my application. Something I’ve been wondering is that enqueueing does not allow to use the return of whatever business logic you are using on the worker module. For example, when the following code is executed:
%{id: 1, in_the: "business", of_doing: "business"}
|> MyApp.Business.new()
|> Oban.insert()
The perform/2 function from the MyApp.Bussiness module will return something I need, however the insertion from the pipeline will return the Oban Structure, with some arguments and metadata that I do not need for the moment. Is there a way for enqueueing and still being able to use the return from the worker module when executing the perform/2 function ?
Marked As Solved
sorentwo
Thank you for the well reasoned and insightful response. You bring up some great points and I’m inclined to agree, I have similar concerns.
This is entirely true. While some type of await is possible you couldn’t do yield because the value isn’t stored anywhere—and it absolutely shouldn’t be.
This seems like the correct approach. There is already a lot of pubsub/notification handling in the library but it is private and lacks a few conveniences. Making that public with the addition of a few convenience functions would make all of these async workflow use-cases possible.
With a minor code change and some documentation this would be easy to do within your own app:
defmodule MyApp.ObanUtils do
import Oban.Notifier, only: [signal: 0]
alias Oban.Notifier
def insert_await(changeset, timeout \\ 5_000) do
Notifier.listen([:signal])
job = Oban.insert!(changeset)
receive do
{:notification, signal(), %{"action" => "complete", "job_id" => ^job.id} = payload} ->
{:ok, payload["result"]}
after
timeout ->
{:error, :timeout, job}
end
end
def notify(job, result) do
Notifier.notify(signal(), %{action: :complete, job_id: job.id, result: result})
end
end
defmodule MyApp.Worker do
use Oban.Worker
alias MyApp.ObanUtils
@impl Oban.Worker
def perform(_args, job) do
result = do_some_work()
ObanUtils.notify(job, result)
:ok
end
end
You’re then explicitly controlling when and how you broadcast and you’re aware of the values that are being returned.
Also Liked
NobbZ
No.
In general what you want is not possible in asynchronous background processors, by design.
The job does not need to run immediately. it can happen in a couple of seconds or take minutes or even hours before the job actually gets run.
A common approach is to let the job write its result into the filesystem or a database. You might need to also put additional information about current state into your FS/DB.
benwilson512
@tovarchristian21 if you can block until the stuff is done, can you just ditch oban entirely and simply call whatever is inside perform/2?
benwilson512
@chulkilee @sorentwo Those are completely sensible use cases, but I suppose the bit I was questioning was whether a first class bespoke solution is better in the long run vs finding a way to utilize existing community tools. Conceptually, the client isn’t waiting on a job to finish, the client is waiting on a specific domain related thing to happen. By publishing, you gain complete control over the value sent out, you can even send out multiple events within the same job as various sub parts complete. All of these are completely orthogonal to the fact that the domain related activity is occurring within a job. Coupling those two feels problematic to me.
I also have library scope concerns. Obviously it’s @sorentwo’s library and he can do with it he wants, but I worry about scope creep when I see the library looking to duplicate work already found within popular and well tested libraries within the community. I just know that the moment you deliver this feature, folks will want to be able to easily add job identities, and then await on the status of those jobs on reconnect to so that you could populate eg a lists of image resize operations that are in progress. That kind of information though is extremely domain specific, you don’t want to push those attributes into the job row.
Perhaps there is a middle ground here. Maybe in lieu of building a built in pubsub, Oban could provide some useful hooks or easy to follow guide on broadcasting stuff from within a job to interested subscribers? If the concern is that people don’t want to add Erlang distribution, maybe the missing piece is a postgres based Phoenix PubSub implementation? I guess I’m pushing for a more modular solution here.
Popular in Questions
Other popular topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"