Questions & Help>Questions

Mnesia crashing on nodeup in clustered configuration

#mnesia#distributed-systems#distribution#troubleshooting#distributed-database
beardedeagle

beardedeagle

I am running into a problem where mnesia is crashing on nodeup in clustered mode:

Mnesia('vanguard@test02'): mnesia_monitor starting: <0.259.0>
Mnesia('vanguard@test02'): Mnesia debug level set to verbose
Mnesia('vanguard@test02'): mnesia_subscr starting: <0.260.0>
Mnesia('vanguard@test02'): mnesia_locker starting: <0.261.0>
Mnesia('vanguard@test02'): mnesia_recover starting: <0.262.0>
Mnesia('vanguard@test02'): mnesia_tm starting: <0.263.0>
Mnesia('vanguard@test02'): Schema initiated from: default
Mnesia('vanguard@test02'): mnesia_controller starting: <0.265.0>
Mnesia('vanguard@test02'): mnesia_late_loader starting: <0.266.0>
Mnesia('vanguard@test02'): Intend to load tables: []
Mnesia('vanguard@test02'): mnesia_controller terminated: shutdown
Mnesia('vanguard@test02'): mnesia_tm terminated: shutdown
Mnesia('vanguard@test02'): mnesia_recover terminated: shutdown
Mnesia('vanguard@test02'): mnesia_locker terminated: shutdown
Mnesia('vanguard@test02'): mnesia_subscr terminated: shutdown
Mnesia('vanguard@test02'): mnesia_monitor terminated: shutdown
Mnesia('vanguard@test02'): mnesia_monitor starting: <0.280.0>
Mnesia('vanguard@test02'): Mnesia debug level set to verbose
Mnesia('vanguard@test02'): mnesia_subscr starting: <0.281.0>
Mnesia('vanguard@test02'): mnesia_locker starting: <0.282.0>
Mnesia('vanguard@test02'): mnesia_recover starting: <0.283.0>
Mnesia('vanguard@test02'): mnesia_tm starting: <0.284.0>
Mnesia('vanguard@test02'): Schema initiated from: default
Mnesia('vanguard@test02'): mnesia_controller starting: <0.286.0>
Mnesia('vanguard@test02'): mnesia_late_loader starting: <0.287.0>
Mnesia('vanguard@test02'): Intend to load tables: []
Mnesia('vanguard@test02'): Mnesia started, 0 seconds
[info] Application mnesia exited: :stopped
Mnesia('vanguard@test02'): Logging mnesia_up 'vanguard@test01'
Mnesia('vanguard@test02'): Logging mnesia_up 'vanguard@test02'
Mnesia('vanguard@test02'): Creating index for 'Elixir.ProcStore' 3 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.HandlerStore' 3 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.HandlerStore' 7 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.StatusStore' 3 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.StatusStore' 4 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.CheckStore' 3 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.CheckStore' 4 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.CheckStore' 10 ordered

If I kill node 1, wait, then bring it back up I get the following:

Mnesia('vanguard@test01'): mnesia_monitor starting: <0.263.0>
Mnesia('vanguard@test01'): Mnesia debug level set to verbose
Mnesia('vanguard@test01'): mnesia_subscr starting: <0.264.0>
Mnesia('vanguard@test01'): mnesia_locker starting: <0.265.0>
Mnesia('vanguard@test01'): mnesia_recover starting: <0.266.0>
Mnesia('vanguard@test01'): mnesia_tm starting: <0.267.0>
Mnesia('vanguard@test01'): Schema initiated from: disc
Mnesia('vanguard@test01'): Initial dump of log during startup: [dumped,
                                                                               dumped]
Mnesia('vanguard@test01'): mnesia_controller starting: <0.290.0>
Mnesia('vanguard@test01'): mnesia_late_loader starting: <0.291.0>
Mnesia('vanguard@test01'): Intend to load tables: [{'Elixir.ProcStore',
                                                                   ram_only}]
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test02'
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test03'
Mnesia('vanguard@test01'): mnesia_controller terminated: shutdown
Mnesia('vanguard@test01'): mnesia_tm terminated: shutdown
Mnesia('vanguard@test01'): mnesia_recover terminated: shutdown
Mnesia('vanguard@test01'): mnesia_locker terminated: shutdown
Mnesia('vanguard@test01'): mnesia_subscr terminated: shutdown
Mnesia('vanguard@test01'): mnesia_monitor terminated: shutdown
Mnesia('vanguard@test01'): mnesia_monitor starting: <0.319.0>
Mnesia('vanguard@test01'): Mnesia debug level set to verbose
Mnesia('vanguard@test01'): mnesia_subscr starting: <0.320.0>
Mnesia('vanguard@test01'): mnesia_locker starting: <0.321.0>
Mnesia('vanguard@test01'): mnesia_recover starting: <0.322.0>
Mnesia('vanguard@test01'): mnesia_tm starting: <0.323.0>
Mnesia('vanguard@test01'): Schema initiated from: default
Mnesia('vanguard@test01'): mnesia_controller starting: <0.325.0>
Mnesia('vanguard@test01'): mnesia_late_loader starting: <0.326.0>
Mnesia('vanguard@test01'): Intend to load tables: []
Mnesia('vanguard@test01'): Mnesia started, 0 seconds
[error] Error in process #PID<0.295.0> on node :"vanguard@test01" with exit value:
{:badarg, [{:ets, :insert, [:mnesia_gvar, {:last_error, {{:aborted, {:node_not_running, :"vanguard@test01"}}, [{:mnesia_loader, :wait_on_load_complete, 1, [file: 'mnesia_loader.erl', line: 358]}, {:mnesia_tm, :apply_fun, 3, [file: 'mnesia_tm.erl', line: 836]}, {:mnesia_tm, :execute_transaction, 5, [file: 'mnesia_tm.erl', line: 811]}, {:mnesia_loader, :init_receiver, 5, [file: 'mnesia_loader.erl', line: 284]}, {:mnesia_loader, :do_get_network_copy, 5, [file: 'mnesia_loader.erl', line: 220]}, {:mnesia_controller, :"-load_table_fun/1-fun-3-", 5, [file: 'mnesia_controller.erl', line: 2180]}, {:mnesia_controller, :"-load_and_reply/2-fun-0-", 2, [file: 'mnesia_controller.erl', line: 2138]}]}}], []}, {:mnesia_lib, :set, 2, [file: 'mnesia_lib.erl', line: 444]}, {:mnesia_lib, :fix_error, 1, [file: 'mnesia_lib.erl', line: 906]}, {:mnesia_tm, :return_abort, 3, [file: 'mnesia_tm.erl', line: 958]}, {:mnesia_loader, :init_receiver, 5, [file: 'mnesia_loader.erl', line: 284]}, {:mnesia_loader, :do_get_network_copy, 5, [file: 'mnesia_loader.erl', line: 220]}, {:mnesia_controller, :"-load_table_fun/1-fun-3-", 5, [file: 'mnesia_controller.erl', line: 2180]}, {:mnesia_controller, :"-load_and_reply/2-fun-0-", 2, [file: 'mnesia_controller.erl', line: 2138]}]}

[error] Error in process #PID<0.294.0> on node :"vanguard@test01" with exit value:
{:badarg, [{:ets, :insert, [:mnesia_gvar, {:last_error, {{:aborted, {:node_not_running, :"vanguard@test01"}}, [{:mnesia_loader, :wait_on_load_complete, 1, [file: 'mnesia_loader.erl', line: 358]}, {:mnesia_tm, :apply_fun, 3, [file: 'mnesia_tm.erl', line: 836]}, {:mnesia_tm, :execute_transaction, 5, [file: 'mnesia_tm.erl', line: 811]}, {:mnesia_loader, :init_receiver, 5, [file: 'mnesia_loader.erl', line: 284]}, {:mnesia_loader, :do_get_network_copy, 5, [file: 'mnesia_loader.erl', line: 220]}, {:mnesia_controller, :"-load_table_fun/1-fun-3-", 5, [file: 'mnesia_controller.erl', line: 2180]}, {:mnesia_controller, :"-load_and_reply/2-fun-0-", 2, [file: 'mnesia_controller.erl', line: 2138]}]}}], []}, {:mnesia_lib, :set, 2, [file: 'mnesia_lib.erl', line: 444]}, {:mnesia_lib, :fix_error, 1, [file: 'mnesia_lib.erl', line: 906]}, {:mnesia_tm, :return_abort, 3, [file: 'mnesia_tm.erl', line: 958]}, {:mnesia_loader, :init_receiver, 5, [file: 'mnesia_loader.erl', line: 284]}, {:mnesia_loader, :do_get_network_copy, 5, [file: 'mnesia_loader.erl', line: 220]}, {:mnesia_controller, :"-load_table_fun/1-fun-3-", 5, [file: 'mnesia_controller.erl', line: 2180]}, {:mnesia_controller, :"-load_and_reply/2-fun-0-", 2, [file: 'mnesia_controller.erl', line: 2138]}]}

[info] Application mnesia exited: :stopped
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test03'
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test02'
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test01'
Mnesia('vanguard@test01'): Creating index for 'Elixir.ProcStore' 3 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.CheckStore' 3 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.CheckStore' 4 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.CheckStore' 10 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.HandlerStore' 3 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.HandlerStore' 7 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.StatusStore' 3 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.StatusStore' 4 ordered

I’m not really able to make heads or tails of the situation unfortunately. I am using libcluster, Swarm and Mnesiam for distributed libs. Any and all help/tips/refrences are appreciated.

8 comments
#mnesia#distributed-systems#distribution#troubleshooting#distributed-database
2640 8

Showing Posts 1 to 8

beardedeagle

beardedeagle OP

The above was with debug logging enabled. Here is trace:

0
Post #1
tty

tty

Can you start fresh ?

From the first set of logs you may have started vanguard@test02 too soon and it was still waiting for vanguard@test1 to start. Or still waiting for the node with the schema defined to start.

The second set of logs were likely caused by killing node 1.

What is you cluster configuration ? Do you have access to the SASL logs ?

0
Post #2
beardedeagle

beardedeagle OP

@tty I posted a gist with the logs from the following scenario:

  1. Bring node 1 up
  2. wait N minutes
  3. Bring node 2 up

Let me look into how to gather the SASL logs

0
Post #3
tty

tty

Read the logs a bit closer and its not due to network connectivity. SASL logs would be helpful.

0
Post #5
beardedeagle

beardedeagle OP

They all have the same cookie. That’s expected behavior and comes from libcluster, since it’s the first node to come up. I did however notice this on node 2:

Mnesia('vanguard@test02'): write performed by {tid,3,<0.291.0>} on record:
        {schema,schema,
            [{name,schema},
             {type,set},
             {ram_copies,['vanguard@test02']},
             {disc_copies,['vanguard@test01']},
             {disc_only_copies,[]},
             {load_order,0},
             {access_mode,read_write},
             {majority,false},
             {index,[]},
             {snmp,[]},
             {local_content,false},
             {record_name,schema},
             {attributes,[table,cstruct]},
             {user_properties,[]},
             {frag_properties,[]},
             {storage_properties,[]},
             {cookie,
                 {{1532443549373280408,-576460752303423485,1},
                  'vanguard@test01'}},
             {version,
                 {{4,0},
                  {'vanguard@test02',{1532,443782,97859}}}}]}
[info] Application mnesia exited: :stopped

I wonder if this is the cause? All schema/tables should be :disk_copies.

iex(vanguard@test01)4> Node.ping(:"vanguard@test02")
:pong
0
Post #6
tty

tty

Likely it is. Tables default to ram_copies if the schema wasn’t distributed as disc_copies over all nodes.

0
Post #7
beardedeagle

beardedeagle OP

ok so likely it’s somewhere in this fork of mnesiam:
https://github.com/beardedeagle/mnesiam

0
Post #8
beardedeagle

beardedeagle OP

Ok, I’ve since forked mnesiam here (with the blessing of the original lib author):
https://github.com/beardedeagle/mnesiac

And i’ve documented the issue here to work on it as I have time:
https://github.com/beardedeagle/mnesiac/issues/2

2
Post #9
— All posts loaded —

Where Next? Top

View thread on forum (has 8 replies!)
mnesia
distributed-systems
distribution
troubleshooting
distributed-database
Home Questions & Help>Questions
3 2553 8
Last post
REPLY VIEW THREAD WITH 8 POSTS

Trending in Questions Top

RSP87
Questions & Help>Questions

Programming Phoenix chat example - iPhone lack of socket updates
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
/phoenix#troubleshooting
1 950 6
New
nseaSeb
Questions & Help>Questions

What's the approach for handling lists that allows for optimized traversal?
Hello, I know there is an approach for handling lists that allows for optimized traversal, but I can’t recall the specific method (somet...
#learning-elixir#list
13 419 15
New
RemyXRenard
Questions & Help>Questions

Livebook DataTable displaying charlist as list
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
/livebook#charlist
2 325 3
New
velrest
Questions & Help>Questions

`:erlang.float` vs `value * 1.0`: Which is the prefered way?
So my question is quite simple and i have found no conclusive answer on forum, google or AI. Should we use :erlang.float for Integer to ...
#learning-elixir
12 638 5
New
brecabral
Questions & Help>Questions

Scoped Routes section of the Phoenix guides refers to a problem without explaining what it is ("but there is a problem")
Documentation While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
/phoenix
2 232 2
New
samoloth
Questions & Help>Questions

Prevent_hijacking? on email change confirmation with magic link strategy
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
/ash#ash-authentication
0 196 1
New
FlyingNoodle
Questions & Help>Questions

Should change and preparation modules validate arguments?
If a change or preparation module uses Ash.Changeset.get_argument/2 or Ash.Query.get_argument/2 (or any of the other get_argument functio...
/ash
1 138 1
New

Other Trending Topics Top

ancatrusca
Blogs & Podcasts>Podcasts

: BEAM There, Done That with Louis Pilfold on Gleam, Static Types, and Building a Language for Six Years
New episode with Louis Pilfold - the most detailed conversation about Gleam’s design I’ve come across. Worth knowing for the Elixir comm...
#gleam#podcasts-by-ancatrusca#beam-there-done-that
0 1 0
New
mudasobwa
Chat & Discussions>Discussions

Cure - a new language compiled to BEAM
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
#static-types#type-checking#types#fsm
168 5507 69
New
marciok
News>Announcing

Gust - A task orchestration system built in Elixir
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
/phoenix/oban#library#workflow#background_job#reactor
111 4469 40
New
jimsynz
News>Announcing

Beam Bots - Resilient Robotics on the BEAM
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
/beam-bots#library#robotics#bb
119 2916 28
New
Dmk
News>Announcing

Xamal - Deploy Elixir releases to bare metal servers via SSH
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
#library#deployment#deploy#deployments#xamal
75 2512 33
New
netoum
News>Announcing

Corex - Accessible and unstyled UI Phoenix components
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
/phoenix#library#ui#liveview#hooks#accessibility#component#tableau
63 2662 20
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking, logs, uptime, and Just Enough APM. Built for developers.

Manning

Master modern tools, write cleaner code, and tackle real-world challenges.

WyeWorks

**Building the future of software with AI, trust, and human connection.**

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking, logs, uptime, and Just Enough APM. Built for developers.

Manning

Master modern tools, write cleaner code, and tackle real-world challenges.

WyeWorks

**Building the future of software with AI, trust, and human connection.**

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews