Mnesia tables size limitation is 2GB or more for `disc_copies`?

Reading the Erlang docs for Mnesia:

11.5 How much data can be stored in Mnesia?

It depends on the storage type of your tables.

  • For ram_copies and disc_copies, the entire table is kept in memory, so data size is limited by available RAM.
    Note that for disc_copies tables, the entire table needs to be read from disk to memory on node startup, which can take a long time for large tables.

  • disc_only_copies tables are limited to 2 GB each. If your table is fragmented, each fragment counts as a separate table, and the combined size can thus exceed 2 GB.
    The reason for this limit is that disc_only_copies tables are backed by Dets tables, and the Dets file format uses signed 32-bit integers for file offsets.

An earlier version of this FAQ entry claimed that all tables are limited by the Dets limit. This was the case until R7B-4 (released on 30th September 2001), when disc_copies tables were moved from Dets to disk_log.

So my doubt is that for disc_only_copies it clearly states a 2GB limit, unless we fragment the table, but for disc_copies doesn’t refer any limit, and I am wondering if it infers the same limit of disc_only_copies?

disc_copies is limited to the available RAM on the node as it holds all the data in RAM as well as on disk (not dets so no 2GN limit). disc_only_copies uses dets which has the 2GB limit and it doesn’t hold data in RAM.

Nowadays there are other mnesia backends as well (leveldb for example


So what is the backend storage engine being used?

disk_log for disk persistence (or for the WAL logging). Well disk_log is part of it. Disk log doesn’t hold the data in memory, that is done by ETS. but the disk_log format is used for disk persistence if I remember correctly.

(and sorry for the multiple edits, I can’t seem to express myself clearly today and it was a while since I looked at this)

1 Like

I was not even aware Mnesia’s backend could be swapped. Pretty interesting, thanks for sharing!

Do you know of other backends?

I only know about the two mentioned here:

1 Like