emoragaf

emoragaf

Best way to generate sequential ids

Hello, we have this situation right now:

Every time we make a sale we need to emit a document with a unique and sequential document number.

These numbers belong to a sequence based on the product sold (each document is for one product), we have a reserved range of numbers for each product, the ranges are big enough that we do not need to worry about running out of numbers right now.

We are looking at some options to generate these numbers (postgres sequences, Kafka events, Zookeeper sequence nodes), and I thought that maybe this is a good fit for Elixir.

I’m thinking of spawning a GenServer for each product configured with the sequence range and current value, so I can use it to generate the subsequent numbers.

I would persist the updated values to a DB to restore the processes in case of failure, and in case of needing some redundancy I could use something like libcluster + swarm to spread my GenServers across a cluster

Does this look right? Maybe I’m missing an obvious alternative, any pitfalls I might not be taking into account?

Most Liked

lalo2302

lalo2302

In my company we had the exact problem you have and we solved it using Postgres advisory locks:
PostgreSQL: Documentation: 9.4: Explicit Locking Point 13.3.5

So you don’t deal with the concurrency on your elixir application, but on the database level.

So let’s say you want to have a sequential number for product “A”, and 2 concurrent requests come

Request 1 and 2 concurrently:

    1. Comes in
    1. Comes in
    1. Sets advisory lock with id :erlang.crc32("A")
    1. Sets advisory lock with id :erlang.crc32("A")
    1. Lock is being used, I’ll wait until it is free
    1. Query count “A” => returns 2
    1. Inserts new record with 3
    1. Finishes transaction and releases lock
    1. Lock is available, I can proceed
    1. Query count “A” => returns 3
    1. Inserts new record with 4

I hope I was explicit enough. Using postgres takes away you a lot of headaches. You don’t need to worry about if your new GenServer is a bottle neck, or if you deploy on a cluster how should you manage your processes.

shanesveller

shanesveller

Do the counters / sequences need to be durable across BEAM restarts, or internally consistent across multiple BEAM nodes? Both of those are challenging with any of the ideas above. Some other questions that come to mind:

Do they need to be monotonic across all machines?
Are gaps permissible?
Are collisions disastrous?
Can you overflow a given product’s range?

Most of these point me towards solutions on the persistent storage rather than application code, i.e. a specialized use of Postgres sequences or similar.

I’d also be curious what real-world or business constraints are at play here since the described technical limitations are unusual.

kokolegorille

kokolegorille

ksuid generate unique, sortable id.

You could (post/)prepend with your product id if You want.

Now your solution with a Genserver per product could work… it depends how much do You sell, but if You sell too much You might have a bottleneck. It will not work that well in a distributed mode.

It depends also if You just want an increasing counter, or uuid style id.

Where Next?

Popular in Questions Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New

Other popular topics Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52341 488
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 126479 1222
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement