sir, please do not encourage me. I ended up discussing medieval literature and Don Quixote in particular the last time. On this forum.

As much as I love fault-tolerant consensus (and you know I do), I don’t think this is really necessary here. If you want exactly-once semantics on the queue you would have to store all of your data in Ra (which will certainly never scale), and process registries are a use-case where I’m not entirely convinced you actually want strong consistency guarantees - because where you really want those guarantees is on the database the processes are writing to. Once the database guarantees correctness the process registry and such only needs good performance (it should be right most of the time), so you don’t really need consensus anymore.

(I have had the misfortune of thinking about this a lot lately.)

But here the data is already stored in Postgres! So if you just put your work queue in Postgres you get transactions over both and you are totally safe correctness-wise (read committed footguns notwithstanding).

Like you could literally just do this:

def insert_data(text) do
  %Data{id: id} = Repo.insert!(%Data{text: text})
  Repo.insert! %EmbedJob{data_id: id, lease: nil}
end

def pop_job do
  expired = DateTime.utc_now() |> DateTime.add(-1, :minute)
  Repo.transaction(fn ->
    job = Repo.one!(from j in EmbedJob,
      where: is_nil(j.lease) or j.lease < ^expired,
      order_by: :inserted_at, limit: 1, lock: "for update", preload: :data)
    Repo.update! change(job, %{lease: DateTime.utc_now()})
  end)
end

def worker do
  {:ok, job} = pop_job()
  vector = embed_text(job.data.text)
  Repo.transaction(fn ->
    %Job{} = Repo.one!(from j in EmbedJob, where: j.id == ^job.id, lock: "for update")
    Repo.update! change(job.data, %{vector: vector})
    Repo.delete! job
  end)
end

And now you don’t need a process registry at all!

I’d default to Oban, and run some queues only on selected nodes.