“ALWAYS serial” requirement sounds like an ideal case for a GenServer.
TwistingTwists
kevinschweikert
No, sorry, that isn’t what I meant. I’ll try it with an example:
A user wants to schedule an event from 9 AM to 10 AM and then another one from 10 AM to 11 AM. Totally fine. But I don’t want to create an overlapping event from lets say 9:30 AM to 10:30 AM.
But the more I think about it, I am convinced, that this isn’t the responsibility of Oban.
0
gregvaughn
I haven’t implemented this, but I ran across this blog recently that demonstrates postgres functions to find overlapping meeting situations. Might be useful to you.
3
mgibowski
You could enforce it in Postgres using range type and constraint.
In that case you would model your event with an events table that would have a duration column of tsrange or timerange type - the values would be '[09:30, 10:30]'::timerange or '[2022-06-20 09:30, 2022-06-20 10:30]'::tsrange depending on your needs.
And then you could create a non-overlapping constraint on that column, like in the example:
1
percej
You can probably use the :unique option, but it could wind up being kludgy. I found :unique very handy for emails - to avoid duplicates. I think I would use ecto to model it and create a function that checks for overlapping events and schedules the oban job only if you get a valid changeset. Then you can encapsulate all that validation into the model. Probably want the check and update in a multi.
0
cjbottaro
Well, the whole idea spawned from making a serverless background job system that could work for my day job. The notion being that if it works for our use case, it could work for anyone.
I’m a huge fan of Faktory. It’s a polyglot background job system that is totally free (including a nice UI). And it’s based on Redis, which has an extremely high performance ceiling.
The downside to Faktory (or Resque or Sidekiq) is hosting your own instance, monitoring that instance, setting up alerting, etc etc.
I’ve personally longed for a serverless version of these products by the 5th time our Redis instance blew out and our product came to a screeching halt. I think it’s possible, depending on how motivated someone is to make it.
0
brightball
PostgreSQL can only do this effectively due to a special type of locking system (Advisory Locks) that is dramatically more efficient for this particular use case.
The first place I ever remember seeing the technique was a Ruby Queue called Que. GitHub - que-rb/que: A Ruby job queue that uses PostgreSQL's advisory locks for speed and reliability. · GitHub
0
cjbottaro
I’m a huge fan of advisory locks! We use them frequently with Sidekiq/Resque/Faktory, as well as in web requests… 
We also make use of other awesome Postgres features like jsonb columns and logical decoding of replication for CDC purposes. Postgres is an amazing piece of tech. Our product has 19 Postgres instances and not wimpy ones either; we love Postgres!
I’ve never tested Postgres’ advisory locks for the “high contention consumer” problem; I just assumed it would break down because it’s a very difficult problem… and Postgres has broken down for us in other ways.
We tried both Etcd and Redis for distributed locks and even Etcd went into completely unrecoverable states. Hell, even Redis Cluster has issues, hence why we just went with propriety AWS MemoryDB.
Distributed computing is hard.
1
brightball
I always assume that if I have system that manages to outgrow Oban, the next move to optimize would be Broadway.
0
sorentwo
Oban Core Team
The feature that makes Postgres an efficient queue processor is FOR UPDATE SKIP LOCKED, which eliminates SELECT contention and doesn’t make use of advisory locks.
Long-held advisory locks don’t play well with Ecto because of how it uses pools. Advisory locks are owned by the connection that starts them, and they can only be released by the same connection. Add the fact that advisory locks don’t work with pg_bouncer in transaction or session pooling modes, and you’ll find that only the xact (transactional) variant is broadly useful for concurrency controls. That’s why Oban uses an unlogged table for centralized leadership.
4