I am currently trying @koudelka 's Honeydew library hooked to Ecto models to handle background job processing in a project. So far so good, but I am a bit puzzled on how to best set it up in a way that we run a background job on each record update, not just after a record has been created.
I have opened an issue on Honeydew Github because this is something we may want to document for others in docs: https://github.com/koudelka/honeydew/issues/81
but basically my thinking is if I need to run a job after record has been created, I need to insert new record that represents this update. So, let’s say I have a Photo record, and I want to run ClassifyPhoto job after user updated the Photo. The only good solution I can think of is to create PhotoUpdate record, that belongs_to Photo, and it triggers ClassifyPhoto job.
Is this the correct solution or I am missing something?
I haven’t used Honeydew. But can’t you fire off a process that regularly checks if all photos have been updated after the last check and then fire off jobs for each that have been updated?
In looking over honeydew, the premise is that it doesn’t have a dedicated jobs table, rather it just treats regular rows in a give table as part of the job queue. This seems like an idea with some serious limitations, including the one the OP is running into right now.
Yep. At this point you’d be better off just firing your own processes under a dynamic supervisor, IMO of course. But I understand it’s created with a rather specific use case in mind and I respect that.
Indeed, this is not a scenario that Honeydew’s Ecto Queue was designed to solve. It’s intended for use cases where the creation of an entity implies some fixed number of different background jobs.
2 Likes
