Mnesia with Memento lost all records for all tables

Questions / Help
, , ,
1

I have been developing an app in localhost, while keeping a staging version of the app running in another folder, with mix phx.server, that I use daily.

So, each time I shutdown or restart the laptop I don’t bother to manually stop it, but today I have stopped it accidentally with ctrl + c + c, and for my surprise when I started it again all my tables were empty :dizzy_face: :exploding_head:

This was not the first time I have used ctrl + c + c to stop it, but never had this outcome.

I am using disc_copies tables, and I am not running in clustered mode.

application.ex:

  def start(_type, _args) do
    # Create the Schema
    Memento.stop
    Memento.Schema.create([node()])
    Memento.start

    # Create the DB with Disk Copies
    # @TODO:
    # Use Memento.Table.wait when it gets implemented
    # @db.create!(disk: nodes)
    # @db.wait(15000)
    Memento.Table.create!(Users, disc_copies: nodes)
   
    # omitted usual code

end

So what can I do to pinpoint the cause of this catastrophic lost?

2 Likes
2

To me it sounds like a new set of schema was created so make sure that:

  • node name has not changed
  • The path to the mnesia files have not changed (look for :mnesia configuration options)

In case the path and node name are correct you can check the status of mnesia itself:

You can check the status of mnesia with :mnesia.system_info(), :mnesia.info(). See man pages for mnesia for more information, or perhaps this page: https://erlang.org/doc/apps/mnesia/Mnesia_chap7.html

You can look for an mnesia core dump to figure out if something crashed. They are called MnesiaCore.<nodename>.<when>.

A quick comment on your startup code. You should generally only create the schema and table once, not every startup. This is also mentioned in the Memento documentation. I don’t know if that has anything to do with your problem (likely not) but something you should consider before moving into production.

1 Like
3

Thanks for your detailed answer :slight_smile:

I have not done any changes to the staging folder location or to the code on it in the last 4 months, thus I think I can exclude this probable causes.

I don’t have any of it in my staging folder. That means a crash is out of question to have caused the lost of the data?

Thanks for alert me for it. I read that on time on the their docs, but then I forgot to remove it from the code. It was meant to be used in active development only, while I was playing around with the schemas, thus destroying the tables all the time.

Thanks for the advice, I will remove it immediately.

Wondering now if this had been the cause of the wipe-out of my tables???

I will read and see if can make some sense of it, because I have no Erlang or Mnesia exposure, and I have only used Elixir in localhost.

4

Well, it wasn’t necessarily you who changed it :smiley: so don’t write this off until you can confirm it is not the problem.

As an example from the memento configuration:

# config/config.exs
config :mnesia,
  dir: '.mnesia/#{Mix.env}/#{node()}'        # Notice the single quotes

Which means if you change node name or your mix env you will end up with a different directory.

I don’t know if you followed this configuration but that could give an idea why something would change.

Regarding :mnesia.system_info I would probably just check that the directory is what you expect, that the node is started and accessible and that the tables are of the type you expect.

Interactive Elixir (1.10.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Memento.stop

14:34:19.460 [info]  Application mnesia exited: :stopped
:ok
iex(2)> Memento.Schema.create([node()])
:ok
iex(3)> Memento.start()
:ok
iex(5)> :mnesia.system_info()
===> System info in version "4.16.1", debug level = none <===
opt_disc. Directory "/home/martink/mnesia/what/Mnesia.nonode@nohost" is used.
use fallback at restart = false
running db nodes   = [nonode@nohost]
stopped db nodes   = []
master node tables = []
remote             = []
ram_copies         = []
disc_copies        = [schema]
disc_only_copies   = []
[{nonode@nohost,disc_copies}] = [schema]
2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []

Gives you some information. Especially the Directory "/home/martink/mnesia/what/Mnesia.nonode@nohost" is used line shows you where you store the mnesia datafiles.

You can also double-check that the node is started and that the tables are in fact in :disc_copies mode.

If you search for schema.Dat you might find others mnesia schemas being created.

I don’t think so but perhaps if there is some funny race condition? Normally mnesia will detect that there is a database there already and give you and already_exists error.

1 Like
5

Thanks for the clarification.

I have the same setup from above and my node and env haven’t change from .mnesia/dev/nonode@nohost.

I confirmed that the directory is the same.

Also, I can work with the application immediately after it boots, like for adding new records to it.

This is a binary file, thus I cannot read it in the editor, but the Erlang docs say it’s possible to read it when Mnesia is not running, and when I run the suggested command:

{ok, N} = dets:open_file(schema, [{file, "./.mnesia/dev/nonode@nohost/schema.DAT"},{repair,false}, 
{keypos, 2}]),
F = fun(X) -> io:format("~p~n", [X]), continue end,
dets:traverse(N, F),
dets:close(N).

I can see the Schemas for my tables in the output :slight_smile:

Maybe it was some race condition caused by trying to create a schema that already exists… who knows now???

So, if it returns already_exists error then it shouldn’t be a cause to wipe-out the database tables :thinking:

I also get an error report when starting the Erlang shell:

=ERROR REPORT==== 15-Dec-2020::10:13:31.160320 ===
file:path_eval(["/home/developer"],".erlang"): error on line 50: 50: evaluation failed with reason error:undef and stacktrace [{docsh,activated,[path],[]},{erl_eval,do_apply,6,[{file,[101,114,108,95,101,118,97,108,46,101,114,108]},{line,680}]},{erl_eval,exprs,5,[{file,[101,114,108,95,101,118,97,108,46,101,114,108]},{line,126}]},{erl_eval,expr,5,[{file,[101,114,108,95,101,118,97,108,46,101,114,108]},{line,462}]},{file,eval_stream2,6,[{file,[102,105,108,101,46,101,114,108]},{line,1422}]},{file,path_eval,3,[{file,[102,105,108,101,46,101,114,108]},{line,1088}]},{c,f_p_e,2,[{file,[99,46,101,114,108]},{line,811}]},{init,eval_script,2,[]}]

Maybe is because I have not compiled it with docs?

6

I am now to the point that I am thinking in ditching Mnesia, because I lost my trust on it for using it in production :frowning:

Anyone in the community with enough experience with Mnesia to tell me that this was a mistake made by me and not a glitch in the Mnesia runtime?

7

Maybe @rvirding(sorry to disturb you) can tell if there has been a known case similar to mine?

8

Maybe it was some race condition caused by trying to create a schema that already exists… who knows now???

I took a quick look on how Memento creates the schema. It checks that the path exists and then simply calls mnesia:create_schema which does never delete or overwrite an existing schema. Given the age and robustness of Mnesia, I would also exclude almost certainly a bug in Mnesia.

I can only guess the problem but I would say that either the environment variables changed (something you already checked though) or somewhere else in your pipeline the target directory gets deleted.

1 Like
9

Thanks for the update. I had started to inspect my self the Memento code some minutes ago, but had not went that far yet :wink:

Well, Mnesia is known for a lot of quirks, but to be honest all databases have them, thus a bug cannot be put aside, or even this being an unexpected behavior, like this one:

As far as I am aware nothing have changed. The app is running from a folder in my laptop that was created with:

git clone my_repo
cd my_repo
git checkout dev-branch
mix deps.get
mix phx.server

This server is only stopped when I restart my laptop(forced by the OS) or by me when doing ctrl c + c.

If, I do a git diff I can confirm that nothing have changed in the folder, and I have not done a git pull to update the branch in months.

10

Maybe my lost of data was due to the disaster recovery mechanism of Mnesia?

6.9.4 Disaster Recovery

The system may become inconsistent as a result of a power failure. The UNIX fsck feature can possibly repair the file system, but there is no guarantee that the file contents will be consistent.

If Mnesia detects that a file has not been properly closed, possibly as a result of a power failure, it will attempt to repair the bad file in a similar manner. Data may be lost, but Mnesia can be restarted even if the data is inconsistent. The configuration parameter -mnesia auto_repair can be used to control the behavior of Mnesia at start-up. If has the value true, Mnesia will attempt to repair the file; if has the value false, Mnesia will not restart if it detects a suspect file. This configuration parameter affects the repair behavior of log files, DAT files, and the default backup media.

The configuration parameter -mnesia dump_log_update_in_place controls the safety level of the mnesia:dump_log() function. By default, Mnesia will dump the transaction log directly into the DAT files. If a power failure happens during the dump, this may cause the randomly accessed DAT files to become corrupt. If the parameter is set to false, Mnesia will copy the DAT files and target the dump to the new temporary files. If the dump is successful, the temporary files will be renamed to their normal DAT suffixes. The possibility for unrecoverable inconsistencies in the data files will be much smaller with this strategy. On the other hand, the actual dumping of the transaction log will be considerably slower. The system designer must decide whether speed or safety is the higher priority.

Replicas of type disc_only_copies will only be affected by this parameter during the initial dump of the log file at start-up. When designing applications which have very high requirements, it may be appropriate not to use disc_only_copies tables at all. The reason for this is the random access nature of normal operating system files. If a node goes down for reason for a reason such as a power failure, these files may be corrupted because they are not properly closed. The DAT files for disc_only_copies are updated on a per transaction basis.

If a disaster occurs and the Mnesia database has been corrupted, it can be reconstructed from a backup. This should be regarded as a last resort, since the backup contains old data. The data is hopefully consistent, but data will definitely be lost when an old backup is used to restore the database.

11

Sorry, can’t help you there. While I know a bit of Mnesia but not enough to be of assistance.

1 Like
12

Please, feel free to point anyone in the community that you may think that can help with this issue, because I really want to keep using Mnesia, but I need to first understand the cause of loosing all my records.

Would it make sense to open a Github issue?

13

If I read your description correctly it seems that you have experienced silent complete data lost without power lost. This is pretty bad. However, it can be easier to reproduce by running it in a loop with kill -9 and checking for data in the application level. If you can reproduce the failure I am sure other people on this forum can offer more help, such as to add instrumentation, code change, etc, to pin point the problem.

1 Like
14

If I understand you correctly I just need to write a script that keeps writing and reading from Mnesia in an infinite loop and then I kill it with kill -9?

If yes, then my application was not in active use when I lost the data, by other words it was running but I was not using it.

Anyway I can try it, maybe using a script like the one used to show the issue with the Mnesia durability guarantees?

Or are you thinking that instead I should just use my app:

  1. start my app as usual with mix phx.server.
  2. use it from the browser as usual
  3. leave it running for a while
  4. Kill it with kill -9
15

You can make some simple API so you can access the app from the outside using curl or something simple. You need 2 API access:

  1. check data integrity, return ok or not
  2. do some random read write

Then you can make a shell script with an infinite loop:

  1. mix phy.server
  2. check integrity. If failed, stop right here
  3. random access
  4. kill -9

Then you can run it over night to see if you can trigger something.

2 Likes
17

Thanks for the suggestion :slight_smile:

I will try to implement it and let you know the results.

Meanwhile I have tried this scripts from the issue I mention previous for Mnesia durability…
write.exs:

:mnesia.create_schema([node()])
|> IO.inspect(label: "<write.exs> Creating schema:")

:mnesia.start()
|> IO.inspect(label: "<write.exs> Starting mnesia:")

# {:atomic, :ok} = :mnesia.create_table(:example, [{:disc_copies, [node()]}, type: :set, attributes: [:name, :value]])
:mnesia.create_table(:example, [{:disc_copies, [node()]}, type: :set, attributes: [:name, :value]])
|> IO.inspect(label: "<write.exs> Creating table:")

:mnesia.transaction(fn -> :mnesia.write({:example, :datetime, NaiveDateTime.utc_now()}) end)
|> IO.inspect(label: "<write.exs> Writing :datetime to database:")
# Enum.each(1..10, fn n -> {:atomic, :ok} = :mnesia.sync_transaction(fn -> :mnesia.write({:example, :foo, "foo-#{n}"}) end) end)

:mnesia.sync_transaction(fn -> :mnesia.read({:example, :datetime}) end)
|> IO.inspect(label: "<write.exs> Reading :datetime from database:")

# :mnesia.stop()
:timer.sleep(2000)
IO.puts "<write.exs> Exiting script.\n"

and read.exs:

:mnesia.start()
|> IO.inspect(label: "<read.exs> Starting mnesia:")

:mnesia.wait_for_tables([:example], :infinity)
|> IO.inspect(label: "<read.exs> Waiting until table is loaded:")

:mnesia.transaction(fn -> :mnesia.read({:example, :datetime}) end)
|> IO.inspect(label: "<read.exs> Reading :datetime form database:")

IO.puts "<read.exs> Exiting script.\n"

So if I invoke the script without sleeping I loose all records in the database:

$ rm -rf Mnesia.nonode@nohost; mix run write.exs; mix run read.exs
<write.exs> Creating schema:: :ok
<write.exs>Starting mnesia:: :ok
<write.exs> Creating table:: {:atomic, :ok}
<write.exs> Writing :datetime to database:: {:atomic, :ok}
<write.exs> Reading :datetime from database:: {:atomic, [{:example, :datetime, ~N[2020-12-19 18:26:13.784003]}]}
<write.exs> Exiting script.

<read.exs> Starting mnesia:: :ok
<read.exs> Waiting until table is loaded:: :ok
<read.exs> Reading :datetime form database:: {:atomic, []}
<read.exs> Exiting script.

If I sleep for 1 second I also loose all the records, but if I sleep for 2 seconds I get the records back:

$ rm -rf Mnesia.nonode@nohost; mix run write.exs; mix run read.exs
<write.exs> Creating schema:: :ok
<write.exs>Starting mnesia:: :ok
<write.exs> Creating table:: {:atomic, :ok}
<write.exs> Writing :datetime to database:: {:atomic, :ok}
<write.exs> Reading :datetime from database:: {:atomic, [{:example, :datetime, ~N[2020-12-19 18:28:28.173542]}]}
<write.exs> Exiting script.

<read.exs> Starting mnesia:: :ok
<read.exs> Waiting until table is loaded:: :ok
<read.exs> Reading :datetime form database:: {:atomic, [{:example, :datetime, ~N[2020-12-19 18:28:28.173542]}]}
<read.exs> Exiting script.

This is just not the same as my scnerario, because my application was running, but not being used. anyway, this looks like exists a delay of more then 1 second for Mnesia to really persists data into the disk, and I am using an SSD.

18

I can’t help you more, but in I have never seen mnesia lose entire data sets like your first problem where all the data is completely gone and I haven’t heard of a similar problem before either. That said I cannot guarantee it is not the fault of mnesia.

As for your other experiment, mnesia does not fsync to disk every commit (regardless of transaction mode) meaning you can lose the last set of transactions in case of a crash. You can control this to certain degree with the two parameters dump_log_write_threshold and dump_log_time_threshold. With these you have some control how often the disc should be dumped. You can play around with them but if you set them to a too low value you will get performance problem.

1 Like
19

Will disc_only_copy be better? I mean to return success of an transaction after the fsync? IMHO durability is what set *real" database apart from toys.

1 Like
20

It was a while since I looked this up but from what I remember neither mnesia nor dets (the underlying file format) do fsync after writing to disk. You can force syncing by dets:sync and disk_log:sync but I don’t think they would be part of transaction.

mnesia is an in memory configuration database first and tries to maintain its durability by being distributed and holding multiple replicas of the data.

I doubt things have changed. Here are two related threads:

http://erlang.2086793.n4.nabble.com/mnesia-sync-transactions-not-fsynced-td4673313.html

1 Like
21

Understood. So Mnesia is not durable for single node operation.

1 Like