acrolink

acrolink

How to ensure a gapless auto-incremental id field when using PostgreSQL

I have noticed that the primary_key field (id field) of tables in PostgreSQL doesn’t guarantee auto-incremental values. The rows keys can be for example 1, 2, 3, 5, 6, 9, 10. This is related to the way PostgreSQL works and due to for example validation errors when creating new entities.

My question, how to avoid that? How to make sure that there would be no gaps in the id field? Thank you.

Marked As Solved

slashdotdash

slashdotdash

You can use a separate counter table and either a Postgres function or a Common Table Expression (CTE) to update the counter value during insert into your target table.

Using a Postgres function

  1. Create a “counter” table containing a single “last value” row:

    CREATE TABLE counter (last_value integer NOT NULL);
    INSERT INTO counter (last_value) VALUES (0);
    
  2. Create a get_next_id function:

    CREATE OR REPLACE FUNCTION get_next_id(countertable regclass, countercolumn text) RETURNS integer AS $$
    DECLARE
        next_value integer;
    BEGIN
        EXECUTE format('UPDATE %s SET %I = %I + 1 RETURNING %I', countertable, countercolumn, countercolumn, countercolumn) INTO next_value;
        RETURN next_value;
    END;
    $$ LANGUAGE plpgsql;
    
    COMMENT ON get_next_id(countername regclass) IS 'Increment and return value from integer column $2 in table $1';
    
  3. Create your desired table to use the gapless sequence:

    CREATE TABLE example (id integer NOT NULL, value text NOT NULL);
    CREATE UNIQUE INDEX events_pkey ON example(id);
    

Usage:

INSERT INTO example(id, value)
VALUES (get_next_id('counter','last_value'), 'example');

Using a CTE

Use a separate “counter” table and a Common Table Expression (CTE) to update the counter value during insert into your target table.

  1. Create a “counter” table containing a single “last value” row:

    CREATE TABLE counter (last_value integer NOT NULL);
    INSERT INTO counter (last_value) VALUES (0);
    
  2. Create your desired table to use the gapless sequence:

    CREATE TABLE example (id integer NOT NULL, value text NOT NULL);
    CREATE UNIQUE INDEX events_pkey ON example(id);
    

Usage:

WITH
  counter AS (
    UPDATE counter SET last_value = last_value + 1
    RETURNING last_value
  )
INSERT INTO example (id, value)
SELECT counter.last_value, 'example';

With both approaches the UPDATERETURNING query will block any other update to the counter table, thus guaranteeing a gapless sequence. You can test this by attempting to run the query concurrently using BEGIN; but not committing. You’ll notice that the second query is blocked until the first commits (or aborts).

This also means that if the first query’s transaction is aborted (using ROLLBACK) the second query can continue and will be assigned the next value, no gaps. Unlike using a traditional Postgres sequence which is not transactional.

A caveat is that if you delete a row from the table you will then have gaps. This can be prevented with a rule to prevent deletion from the table:

CREATE RULE no_delete_example AS ON DELETE TO example DO INSTEAD NOTHING;

Also Liked

al2o3cr

al2o3cr

What if I do the insert that gets ID 4 in a transaction that doesn’t commit for several minutes? What ID should inserts in other transactions taking place at the same time get? What if the transaction where I inserted ID 4 eventually rolls back?

“Max ID” is not a straightforward concept once MVCC and transactions are involved. See the notes in the Postgres docs for CREATE SEQUENCE for additional thoughts. Big takeaway:

Because nextval and setval calls are never rolled back, sequence objects cannot be used if “gapless” assignment of sequence numbers is needed. It is possible to build gapless assignment by using exclusive locking of a table containing a counter; but this solution is much more expensive than sequence objects, especially if many transactions need sequence numbers concurrently.

(emphasis mine)

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Again, what is your actual use case here?

Kurisu

Kurisu

I don’t really know how MySQL is different in this case, but it seems to me that it won’t fill ever at least the gaps due to records deletion. I think the way those SQL engines work is optimal to make quick inserts instead of checking first, gaps of freed ids.

One tricky way I can think about, If the behaviour you want is really important, would be generating the ids yourself based on the current max ID, then if insert fails due to another concurrent insert, you’ll re-acquire the new max ID and retry the insert until it gets done.

Last Post!

OvermindDL1

OvermindDL1

UPDATE ... RETURNING will, but rarely do I ever update a single table at a time, transactions are almost always necessary for data consistency unless it’s an exceptionally simple database that isn’t using the database features… In addition, the primary key is rarely updated, especially as manually as that would be done. Once transactions are involved this will cause major issues either in terms of handling no concurrency or values getting out of sync.

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

Other popular topics Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New

We're in Beta

About us Mission Statement