Mnesia with Memento lost all records for all tables

Well, it wasn’t necessarily you who changed it :smiley: so don’t write this off until you can confirm it is not the problem.

As an example from the memento configuration:

# config/config.exs
config :mnesia,
  dir: '.mnesia/#{Mix.env}/#{node()}'        # Notice the single quotes

Which means if you change node name or your mix env you will end up with a different directory.

I don’t know if you followed this configuration but that could give an idea why something would change.

Regarding :mnesia.system_info I would probably just check that the directory is what you expect, that the node is started and accessible and that the tables are of the type you expect.

Interactive Elixir (1.10.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Memento.stop

14:34:19.460 [info]  Application mnesia exited: :stopped
:ok
iex(2)> Memento.Schema.create([node()])
:ok
iex(3)> Memento.start()
:ok
iex(5)> :mnesia.system_info()
===> System info in version "4.16.1", debug level = none <===
opt_disc. Directory "/home/martink/mnesia/what/Mnesia.nonode@nohost" is used.
use fallback at restart = false
running db nodes   = [nonode@nohost]
stopped db nodes   = []
master node tables = []
remote             = []
ram_copies         = []
disc_copies        = [schema]
disc_only_copies   = []
[{nonode@nohost,disc_copies}] = [schema]
2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []

Gives you some information. Especially the Directory "/home/martink/mnesia/what/Mnesia.nonode@nohost" is used line shows you where you store the mnesia datafiles.

You can also double-check that the node is started and that the tables are in fact in :disc_copies mode.

If you search for schema.Dat you might find others mnesia schemas being created.

I don’t think so but perhaps if there is some funny race condition? Normally mnesia will detect that there is a database there already and give you and already_exists error.

1 Like

Thanks for the clarification.

I have the same setup from above and my node and env haven’t change from .mnesia/dev/nonode@nohost.

I confirmed that the directory is the same.

Also, I can work with the application immediately after it boots, like for adding new records to it.

This is a binary file, thus I cannot read it in the editor, but the Erlang docs say it’s possible to read it when Mnesia is not running, and when I run the suggested command:

{ok, N} = dets:open_file(schema, [{file, "./.mnesia/dev/nonode@nohost/schema.DAT"},{repair,false}, 
{keypos, 2}]),
F = fun(X) -> io:format("~p~n", [X]), continue end,
dets:traverse(N, F),
dets:close(N).

I can see the Schemas for my tables in the output :slight_smile:

Maybe it was some race condition caused by trying to create a schema that already exists… who knows now???

So, if it returns already_exists error then it shouldn’t be a cause to wipe-out the database tables :thinking:


I also get an error report when starting the Erlang shell:

=ERROR REPORT==== 15-Dec-2020::10:13:31.160320 ===
file:path_eval(["/home/developer"],".erlang"): error on line 50: 50: evaluation failed with reason error:undef and stacktrace [{docsh,activated,[path],[]},{erl_eval,do_apply,6,[{file,[101,114,108,95,101,118,97,108,46,101,114,108]},{line,680}]},{erl_eval,exprs,5,[{file,[101,114,108,95,101,118,97,108,46,101,114,108]},{line,126}]},{erl_eval,expr,5,[{file,[101,114,108,95,101,118,97,108,46,101,114,108]},{line,462}]},{file,eval_stream2,6,[{file,[102,105,108,101,46,101,114,108]},{line,1422}]},{file,path_eval,3,[{file,[102,105,108,101,46,101,114,108]},{line,1088}]},{c,f_p_e,2,[{file,[99,46,101,114,108]},{line,811}]},{init,eval_script,2,[]}]

Maybe is because I have not compiled it with docs?

I am now to the point that I am thinking in ditching Mnesia, because I lost my trust on it for using it in production :frowning:

Anyone in the community with enough experience with Mnesia to tell me that this was a mistake made by me and not a glitch in the Mnesia runtime?

Maybe @rvirding(sorry to disturb you) can tell if there has been a known case similar to mine?

Maybe it was some race condition caused by trying to create a schema that already exists… who knows now???

I took a quick look on how Memento creates the schema. It checks that the path exists and then simply calls mnesia:create_schema which does never delete or overwrite an existing schema. Given the age and robustness of Mnesia, I would also exclude almost certainly a bug in Mnesia.

I can only guess the problem but I would say that either the environment variables changed (something you already checked though) or somewhere else in your pipeline the target directory gets deleted.

1 Like

Thanks for the update. I had started to inspect my self the Memento code some minutes ago, but had not went that far yet :wink:

Well, Mnesia is known for a lot of quirks, but to be honest all databases have them, thus a bug cannot be put aside, or even this being an unexpected behavior, like this one:

As far as I am aware nothing have changed. The app is running from a folder in my laptop that was created with:

git clone my_repo
cd my_repo
git checkout dev-branch
mix deps.get
mix phx.server

This server is only stopped when I restart my laptop(forced by the OS) or by me when doing ctrl c + c.

If, I do a git diff I can confirm that nothing have changed in the folder, and I have not done a git pull to update the branch in months.

Maybe my lost of data was due to the disaster recovery mechanism of Mnesia?

6.9.4 Disaster Recovery

The system may become inconsistent as a result of a power failure. The UNIX fsck feature can possibly repair the file system, but there is no guarantee that the file contents will be consistent.

If Mnesia detects that a file has not been properly closed, possibly as a result of a power failure, it will attempt to repair the bad file in a similar manner. Data may be lost, but Mnesia can be restarted even if the data is inconsistent. The configuration parameter -mnesia auto_repair can be used to control the behavior of Mnesia at start-up. If has the value true, Mnesia will attempt to repair the file; if has the value false, Mnesia will not restart if it detects a suspect file. This configuration parameter affects the repair behavior of log files, DAT files, and the default backup media.

The configuration parameter -mnesia dump_log_update_in_place controls the safety level of the mnesia:dump_log() function. By default, Mnesia will dump the transaction log directly into the DAT files. If a power failure happens during the dump, this may cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia will copy the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files will be renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files will be much smaller with this strategy. On the other hand, the actual dumping of the transaction log will be considerably slower. The system designer must decide whether speed or safety is the higher priority.

Replicas of type disc_only_copies will only be affected by this parameter during the initial dump of the log file at start-up. When designing applications which have very high requirements, it may be appropriate not to use disc_only_copies tables at all. The reason for this is the random access nature of normal operating system files. If a node goes down for reason for a reason such as a power failure, these files may be corrupted because they are not properly closed. The DAT files for disc_only_copies are updated on a per transaction basis.

If a disaster occurs and the Mnesia database has been corrupted, it can be reconstructed from a backup. This should be regarded as a last resort, since the backup contains old data. The data is hopefully consistent, but data will definitely be lost when an old backup is used to restore the database.

Sorry, can’t help you there. While I know a bit of Mnesia but not enough to be of assistance.

1 Like

Please, feel free to point anyone in the community that you may think that can help with this issue, because I really want to keep using Mnesia, but I need to first understand the cause of loosing all my records.

Would it make sense to open a Github issue?

If I read your description correctly it seems that you have experienced silent complete data lost without power lost. This is pretty bad. However, it can be easier to reproduce by running it in a loop with kill -9 and checking for data in the application level. If you can reproduce the failure I am sure other people on this forum can offer more help, such as to add instrumentation, code change, etc, to pin point the problem.

1 Like

If I understand you correctly I just need to write a script that keeps writing and reading from Mnesia in an infinite loop and then I kill it with kill -9?

If yes, then my application was not in active use when I lost the data, by other words it was running but I was not using it.

Anyway I can try it, maybe using a script like the one used to show the issue with the Mnesia durability guarantees?

Or are you thinking that instead I should just use my app:

  1. start my app as usual with mix phx.server.
  2. use it from the browser as usual
  3. leave it running for a while
  4. Kill it with kill -9

You can make some simple API so you can access the app from the outside using curl or something simple. You need 2 API access:

  1. check data integrity, return ok or not
  2. do some random read write

Then you can make a shell script with an infinite loop:

  1. mix phy.server
  2. check integrity. If failed, stop right here
  3. random access
  4. kill -9

Then you can run it over night to see if you can trigger something.

2 Likes

Thanks for the suggestion :slight_smile:

I will try to implement it and let you know the results.


Meanwhile I have tried this scripts from the issue I mention previous for Mnesia durability…
write.exs:

:mnesia.create_schema([node()])
|> IO.inspect(label: "<write.exs> Creating schema:")

:mnesia.start()
|> IO.inspect(label: "<write.exs> Starting mnesia:")

# {:atomic, :ok} = :mnesia.create_table(:example, [{:disc_copies, [node()]}, type: :set, attributes: [:name, :value]])
:mnesia.create_table(:example, [{:disc_copies, [node()]}, type: :set, attributes: [:name, :value]])
|> IO.inspect(label: "<write.exs> Creating table:")

:mnesia.transaction(fn -> :mnesia.write({:example, :datetime, NaiveDateTime.utc_now()}) end)
|> IO.inspect(label: "<write.exs> Writing :datetime to database:")
# Enum.each(1..10, fn n -> {:atomic, :ok} = :mnesia.sync_transaction(fn -> :mnesia.write({:example, :foo, "foo-#{n}"}) end) end)

:mnesia.sync_transaction(fn -> :mnesia.read({:example, :datetime}) end)
|> IO.inspect(label: "<write.exs> Reading :datetime from database:")

# :mnesia.stop()
:timer.sleep(2000)
IO.puts "<write.exs> Exiting script.\n"

and read.exs:

:mnesia.start()
|> IO.inspect(label: "<read.exs> Starting mnesia:")

:mnesia.wait_for_tables([:example], :infinity)
|> IO.inspect(label: "<read.exs> Waiting until table is loaded:")

:mnesia.transaction(fn -> :mnesia.read({:example, :datetime}) end)
|> IO.inspect(label: "<read.exs> Reading :datetime form database:")

IO.puts "<read.exs> Exiting script.\n"

So if I invoke the script without sleeping I loose all records in the database:

$ rm -rf Mnesia.nonode@nohost; mix run write.exs; mix run read.exs
<write.exs> Creating schema:: :ok
<write.exs>Starting mnesia:: :ok
<write.exs> Creating table:: {:atomic, :ok}
<write.exs> Writing :datetime to database:: {:atomic, :ok}
<write.exs> Reading :datetime from database:: {:atomic, [{:example, :datetime, ~N[2020-12-19 18:26:13.784003]}]}
<write.exs> Exiting script.

<read.exs> Starting mnesia:: :ok
<read.exs> Waiting until table is loaded:: :ok
<read.exs> Reading :datetime form database:: {:atomic, []}
<read.exs> Exiting script.

If I sleep for 1 second I also loose all the records, but if I sleep for 2 seconds I get the records back:

$ rm -rf Mnesia.nonode@nohost; mix run write.exs; mix run read.exs
<write.exs> Creating schema:: :ok
<write.exs>Starting mnesia:: :ok
<write.exs> Creating table:: {:atomic, :ok}
<write.exs> Writing :datetime to database:: {:atomic, :ok}
<write.exs> Reading :datetime from database:: {:atomic, [{:example, :datetime, ~N[2020-12-19 18:28:28.173542]}]}
<write.exs> Exiting script.

<read.exs> Starting mnesia:: :ok
<read.exs> Waiting until table is loaded:: :ok
<read.exs> Reading :datetime form database:: {:atomic, [{:example, :datetime, ~N[2020-12-19 18:28:28.173542]}]}
<read.exs> Exiting script.

This is just not the same as my scnerario, because my application was running, but not being used. anyway, this looks like exists a delay of more then 1 second for Mnesia to really persists data into the disk, and I am using an SSD.

I can’t help you more, but in I have never seen mnesia lose entire data sets like your first problem where all the data is completely gone and I haven’t heard of a similar problem before either. That said I cannot guarantee it is not the fault of mnesia.

As for your other experiment, mnesia does not fsync to disk every commit (regardless of transaction mode) meaning you can lose the last set of transactions in case of a crash. You can control this to certain degree with the two parameters dump_log_write_threshold and dump_log_time_threshold. With these you have some control how often the disc should be dumped. You can play around with them but if you set them to a too low value you will get performance problem.

1 Like

Will disc_only_copy be better? I mean to return success of an transaction after the fsync? IMHO durability is what set *real" database apart from toys.

1 Like

It was a while since I looked this up but from what I remember neither mnesia nor dets (the underlying file format) do fsync after writing to disk. You can force syncing by dets:sync and disk_log:sync but I don’t think they would be part of transaction.

mnesia is an in memory configuration database first and tries to maintain its durability by being distributed and holding multiple replicas of the data.

I doubt things have changed. Here are two related threads:

http://erlang.2086793.n4.nabble.com/mnesia-sync-transactions-not-fsynced-td4673313.html

1 Like

Understood. So Mnesia is not durable for single node operation.

1 Like

No, not in the sense of a traditional RDBMS. If you use sync_transaction you know that multiple nodes have received and handled the transaction but you still don’t know if they’ve flushed to disk. A server room outage could make you lose data here as well. But then again, if you lose enough disks in a raid you lose data too.

1 Like

Disabling the Linux write cache could help here?

Not all system’s belong to the same “turn-on write-back caching” recommendation group as write-back caching caries a risk of data loss in the event such as power failure etc. In the event of power failure, data residing in the hard drive’s cache do not get a chance to be stored and a lost. This fact is especially important for database system. In order to disable write-back caching set write-caching to 0:

# hdparm -W0 /dev/sda

/dev/sda:
setting drive write-caching to 0 (off)
write-caching =  0 (off)

For modern SSDs it will be either:

  • disastrous performance degradation, or
  • Ok performance because the SSD vendor cheats, and it is still not durable

It is better to have explicit cache flush passed down from the stack.

1 Like