Thanks for the suggestions @benwilson512 and @sb8244 ! I do think those would be good options, but I’d like to keep the queue services I’m using pretty thin and “dumb”. I have a main Phoenix service that handles api, business logic, etc. and smaller plain elixir services that do some heavier processing with ports etc. They just do what the job they get from the db queue tells them without any networking and report success or failure.
Thanks for the idea and example code @sorentwo ! I’ll give it a try and let you know how it goes. That makes sense about telemetry, and I do think something along the lines of Oban.insert_await/3 would be quite useful in situations like mine. Thanks again!
@sorentwo Your solution of using Oban.Notifier has been working great in my testing so far, thanks!
I also like the ability to send a json payload because it allows me to send a message with some useful information, which works great in my current project. So if you decide on an official implementation, I would vote for that as a good feature to consider keeping.
I’d even say using notifications directly is a useful enough feature to consider improving the interface of as an alternative or in addition to an await feature. Now I’m actually considering using notifications to improve the progress reporting on another service you previously helped me with. It would send a number of small json notifications over the course of a job, maybe up to a couple a second at times. It would allow me to move some logic to my primary app where I would prefer it to live if possible. Does that seem like a reasonable use for Notifier or should I exercise more caution?
Author of Craft GraphQL APIs in Elixir with Absinthe
What would Oban be able to send about your job? It doesn’t know anything about its internal structure. I’m increasingly thinking @sb8244 has the right idea here: you are looking to library level introspection to solve what is a business level concern. If your users have a concept of job progress, you need to model that and communicate about that, flat out. I understand you’re trying to limit the complexity of the worker nodes, but they need to be sufficiently nuanced to provide the level of functionality you require of them.
Great, that’s nice to hear. I neglected to mention that pg_notify messages aren’t delivered until after a transaction commits, so it can be difficult to test with them.
Embarrassingly, I forgot that there is a notify/3 function in Oban.Query that makes this a little easier. You can see it here. As with the notifier module it is internal and undocumented, which makes it private, technically.
That level of activity for PG notifications (and the notifier) is perfectly fine. You could do the same thing with redis pubsub or phoenix channels, it’s up to you. As @benwilson512 mentioned, this is ultimately an app/business level decision, you can use whichever models the data flow best for you.
I’m not using Oban for anything besides enqueuing/retrieving jobs, and now passing some simple messages back. This would just be using Oban.Notifier as a vehicle for sending simple status messages from the service doing work to the service that requested the work. For the progress notifications for instance, it would allow me to move some logic to the service that maintains overall system state and communicates with the front-end etc. which is where I would prefer to keep it.
@sorentwo
With the Oban.Query.notify/3 function you mentioned, I’m able to reduce the notify code to:
import Oban.Notifier, only: [signal: 0]
defp notify(jid) do
payload = %{job_complete: jid}
Oban.Query.notify(Oban.config(), signal(), payload)
end
It seems to me that if you decide to officially support this feature, you could make a public Query.notify/1 that just takes a payload map and just calls Query.notify/3 with the default values for the first two arguments.
Not sure if this question is more appropriate here or on GH. I’ll start here, but let me know if you prefer to move it to issue tracker.
We have the following scenario. We need to enqueue some jobs for a distant future, but then through other interactions, we might decide that some scheduled jobs don’t need to be executed. The question is how can we remove those? From what I can tell, there’s no special API, except maybe kill_job (but I’m not clear if that one would work, and even if it did would require unscheduling jobs one by one). Currently, we’re just deleting oban_jobs entries. Perhaps this approach is fine, but I’d like to double check if this is ok, or is there something else we’re missing?
That’s exactly what I would do (and what the UI does). Using kill_job wouldn’t work in this situation because it only works when a job is executing.
You aren’t missing anything! This has come up fairly often. The fact that this is ambiguous leads me to think that there is a case for an Oban.delete/1 function. Not that it is difficult to implement, but it would be the “blessed” way to delete a job.
