kreiling.io

kreiling.io

I’m also considering an asynchronous approach, which might break some of the OP’s constraints.

My application already generates UUIDs for persisted documents, but we now want to additionally generate sequential IDs in the form of monotonically incrementing integers. Occasional gaps are permissible but collisions are not. I’m not using postgres as a storage backend and would prefer not to introduce it for just this purpose.

So I’m considering 2 different approaches:

  1. Use Swarm to manage Counter processes, dumping state to e.g. S3 on shutdown
  2. Enqueue an event to asynchronously generate the integer ID after document creation

The tradeoff to option 1 is Counters scale linearly with document types, and S3 failures (though unlikely) would be borderline unrecoverable. Option 2’s big downfall, IMO, is that we’d have to accept eventually consistent IDs.

:ets interests me as a possibility, but I don’t have much experience with it. I’m wary of any gotchas in a distributed setting

0
Post #11
derek-zhou

derek-zhou

If the OP cannot tolerate occasional gaps then it is a hard problem indeed.

0
Post #12
emoragaf

emoragaf OP

we actually went that exact same route, we have a process per product on a libcluster+swarm cluster
we can ask for the next id in the sequence to those processes

1
Post #13
kreiling.io

kreiling.io

I did the same thing, it’s good-enough and actually quite simple. I’ll elaborate in case anyone’s curious :slight_smile:

I have a function SequentialID.get_and_increment(tag) that roughly:

  1. Ensures a SequentialID.Counter GenServer process is started under a DynamicSupervisor
  2. The Counter process fetches the integer ID value in init/1 from a pluggable storage backend
  3. The Counter process is call’d to increment the counter, put it in the storage backend, and reply with it

The Counter process is named via Swarm, e.g. {:via, :swarm, {SequentialID.Counter, "my-tag"}} to ensure only one instance of the process is started in the cluster, per tag. Forming a dynamic cluster is easy to do with libcluster.

1
Post #14
— All posts loaded —