Mnesia tables size limitation is 2GB or more for `disc_copies`?

Reading the Erlang docs for Mnesia:

11.5 How much data can be stored in Mnesia?

It depends on the storage type of your tables.

  • For ram_copies and disc_copies, the entire table is kept in memory, so data size is limited by available RAM.
    Note that for disc_copies tables, the entire table needs to be read from disk to memory on node startup, which can take a long time for large tables.

  • disc_only_copies tables are limited to 2 GB each. If your table is fragmented, each fragment counts as a separate table, and the combined size can thus exceed 2 GB.
    The reason for this limit is that disc_only_copies tables are backed by Dets tables, and the Dets file format uses signed 32-bit integers for file offsets.

An earlier version of this FAQ entry claimed that all tables are limited by the Dets limit. This was the case until R7B-4 (released on 30th September 2001), when disc_copies tables were moved from Dets to disk_log.

So my doubt is that for disc_only_copies it clearly states a 2GB limit, unless we fragment the table, but for disc_copies doesn’t refer any limit, and I am wondering if it infers the same limit of disc_only_copies?

disc_copies is limited to the available RAM on the node as it holds all the data in RAM as well as on disk (not dets so no 2GN limit). disc_only_copies uses dets which has the 2GB limit and it doesn’t hold data in RAM.

Nowadays there are other mnesia backends as well (leveldb for example https://github.com/klarna/mnesia_eleveldb)

2 Likes

So what is the backend storage engine being used?

disk_log for disk persistence (or for the WAL logging). Well disk_log is part of it. Disk log doesn’t hold the data in memory, that is done by ETS. but the disk_log format is used for disk persistence if I remember correctly.

(and sorry for the multiple edits, I can’t seem to express myself clearly today and it was a while since I looked at this)

1 Like

I was not even aware Mnesia’s backend could be swapped. Pretty interesting, thanks for sharing!

Do you know of other backends?

I only know about the two mentioned here:

1 Like