The feature that makes Postgres an efficient queue processor is FOR UPDATE SKIP LOCKED, which eliminates SELECT contention and doesn’t make use of advisory locks.
Long-held advisory locks don’t play well with Ecto because of how it uses pools. Advisory locks are owned by the connection that starts them, and they can only be released by the same connection. Add the fact that advisory locks don’t work with pg_bouncer in transaction or session pooling modes, and you’ll find that only the xact (transactional) variant is broadly useful for concurrency controls. That’s why Oban uses an unlogged table for centralized leadership.
On the non overlapping topic… You could use range as mentionned by other
Here is an example
def change do
execute "CREATE EXTENSION btree_gist", "DROP btree_gist"
create table(:examples) do
add :any, :string
# Using a tstzrange should allow exclusion constraint.
# https://www.postgresql.org/docs/13/rangetypes.html#RANGETYPES-CONSTRAINT
add :period, :tstzrange
timestamps(type: :utc_datetime)
end
# Create a custom constraint enforcing period do not overlap for the same resource.
# This request btree_gist extension!
# https://hexdocs.pm/ecto_sql/Ecto.Migration.html#constraint/3
create constraint(:examples, :no_overlap,
exclude: ~s|gist ("any" WITH =, tstzrange("period") WITH &&)|)
end
This use another field for non-overlapping.
Then You can check the constraint in the schema
...
|> check_constraint(:period, name: :no_overlap, message: "Any overlap in schedule")
I haven’t looked at things like AWS Lambda in a long time, but I feel like they’ve gotten to the point of being able to just run Docker images. Seems like you could make an image with an Elixir release and entrypoint that takes args specifying which function you want to run.
Oh wow, TIL! Really interesting.
I thought Broadway was more about making complex job topologies similar to Apache Spark. With things like multiple datasources and even infinitely running datasources.
