Stamp - fast and flexible Snowflake-flavored ID generator

Stamp is a fast and flexible Snowflake-flavored ID generator based on 8-byte integers with optional encoding.

The original Snowflake supported BigTech-scale numbers (e.g. 1024 workers each inserting up to 4,096,000 records per second). With BEAM favoring vertical (rather than horizontal) scaling and hardware getting more powerful every year it is reasonable to assume that we can leverage the bits in the ID more effectively than Twitter did back in 2010, especially if the application never reaches Twitter’s scale.

Features

• works both with and without Ecto
• BlazingFast™ sequences on top of process-free atomic counters
• no global configuration - uses per-field configuration and runtime values
• user-configurable number of bits for every part of the ID
• some parts can be disabled, freeing up the bits for other parts
• optional encoding of the integers to string versions
• optional Stripe-style prefix to easily distinguish IDs of different models/schemas
• optional range-based partitioning and sharding (experimental)

How stamps look

Stamp vs Alternatives

UUID fields have become more popular recently, but they are expensive for use in indexes and especially primary keys. UUIDv7 improves upon v4 and other versions in some ways, but the size hasn’t changed. There are also exotic alternatives like KSUID and ULID, but they share some issues of UUID.

	Stamp	Serial/Identity	UUID
generated in the application
globally unique with distributed generation
compact in terms of storage and RAM
composite PK/indexes without bloat
looks good and is short both raw/encoded
efficient BTree index operations			in v7
may remove the need for extra time field			in v7
bulk inserts	sequence nuances[1]
range partitioning/sharding (not by time)
keeps creation time secret		guessable[2]	not in v7
keeps number of records secret		barely[3]
next id is unpredictable	depends[4]
easy to set up and use	planning, setup[5]

[1] If an insert overflows the sequence, time goes into the future. This does not limit the insertion,
but forces timestamp accuracy trade-offs. In the worst case scenario you can have more accurate version of creation time in a separate field. See Important details and caveats section in the docs for more info.

[2] Because there is only 1 sequence correlated with creation, one could roughly estimate the
creation time of a record by comparing the number with other records, where the time may be known.

[3] Under normal conditions 1 used sequence number corresponds to 1 record. If we can observe
sequence growth, we can estimate or even know precisely how many records are created in a
period of time.
With total number of records it’s a bit different. Some applications set initial sequence
number high to help with the issue, but it’s only a half measure. If it’s possible to observe random ids inside the application, one can notice the gap and guess that it’s empty.

[4] Stamp is monotonic for a combination of partition, node number, sequence id. If you can generate ids for this combination, you can predict next id. Some randomness can be added by randomly picking from a pool of node numbers on each generation. In general, Stamp is definitely not as unpredictable as UUID simply because of the size difference.

[5] The library provides good defaults tuned for “average project scale” rather than “Twitter
scale”, but learning about the available configuration parameters to get the most benefits is still encouraged.

Project on Hex
Docs