I followed this link to connect observer to the remote app. I followed the following steps:
epmd -names # on local
# returns: epmd: up and running on port 4369 with data:
ssh -L 4369:localhost:4369 -L 58769:localhost:58769 ubuntu@server-ip # on local for port forwarding
# epmd: up and running on port 4369 with data:
# name elixirapp at port 45547
erl -name debug@127.0.0.1 -setcookie sec_cookie -hidden -run observer # on local to open observer
Via the last command, the observer window opens. When I click on Node on the menu, I get a option to connect to elixirapp@127.0.0.1. When I click on that I get the following error in the console:
## Name of the node
-name elixirapp@127.0.0.1
## Cookie for distributed erlang
-setcookie sec_cookie
## Heartbeat management; auto-restarts VM if it dies or becomes unresponsive
## (Disabled by default..use with caution!)
##-heart
## Enable kernel poll and a few async threads
##+K true
##+A 5
## Increase number of concurrent ports/sockets
##-env ERL_MAX_PORTS 4096
## Tweak GC to run more often
##-env ERL_FULLSWEEP_AFTER 10
# Enable SMP automatically based on availability
-smp auto
And I do have :runtime_tools present in mix.exs
...
def application do
[
mod: {ElixirApp.Application, []},
extra_applications: [:logger, :runtime_tools]
]
end
...
Can anyone please let me know why I am getting nodedown error.?
Is there any step that I am missing to connect remote node via observer?
ssh -L 4369:localhost:4369 -L 58769:localhost:58769 ubuntu@server-ip # on local for port forwarding
# epmd: up and running on port 4369 with data:
# name elixirapp at port 45547
Perhaps a typo but you say: “elixirapp at port 45547 in the comment” but forwarding says port 57679
This should work, and just to be on the safe side, I’ve tried it out on one production, and it worked as expected.
So I expect you either have a typo (e.g. cookie, node name, or the wrong port forwarded), or there is some unusual condition none of us have seen before. To be on the safe side, try the following:
Close all ssh sessions
Kill the local epmd process
Verify that on dev box no process is listening on the required ports (4369 and 45547)
Establish ssh session and forward the required ports
Verify that you can access these ports on the dev machine
Verify the local node list with epmd -names
On the production node, start a remote console, and check the node name and the cookie.
Start a hidden Erlang node with the correct cookie and try to connect with Node.connect/1.
If the step 8 fails, then the observer won’t work too.
One other thing I remember is that I wasn’t able to make this work with older Erlang versions (IIRC it was 17 or maybe even 18), so you may want to double check which Erlang version do you have, and try with the newer one.
This link says that in order to have nodes connect remotely, one needs to make sure that it resolves correctly. In my setup, I haven’t done anything as such. My local system’s hostname is different from server’s hostname. Does that matter?
Do I need to configure /etc/hosts and /etc/resolv.conf too?
If the production node name is yourapp@foo.bar.baz, then you need to add /etc/hosts entry locally, to resolve the symbolic hostname to 127.0.0.1, but only after you’ve established the ssh connection (or alternatively use ssh with the IP address).
If the production node name is yourapp@127.0.0.1, then it should just work without any changes to /etc/hosts.
The weird thing is, that I am unable to connect locally on the server either.
On Server, I got the node address and cookie by running:
./app_release/cflogs/bin/cflogs remote_console
> Node.self # elixirapp@127.0.0.1
> Node.get_cookie # test
On Server, now, I exit the console and then run:
iex --name debug@127.0.0.1 --cookie test
> Node.connect :"elixirapp@127.0.0.1" # false
This is strange. Am I missing something? Why is it returning false?
Edit:
Turned out I had different versions of elixir installed on the server. I fixed and ensured that only one version is present. I am able to connect to node on the same machine on the server now.
However, I am still unable to connect from local to prod. And, I have the same erlang version installed on both the machines.
Ok. So I got it running finally. Apart from rectifying my initial mistake (had two versions of erlang running on the server), I did two things more this time, I configured the server to accept all incoming and outgoing connections from any port by doing:
sudo iptables -A OUTPUT -j ACCEPT
sudo iptables -A INPUT -j ACCEPT
I don’t know if this really helped or not.
Next, I changed the cookie value from vk1285h/I)a6Q>T=;yY!hLDNjkW5?xz:nunHon4Jz(};KK!([C~zOoqh]a^XH5}3 to test
And yes, if I change the cookie name to a simple and a short one, it works.
So when the cookie name was long (on the server) I ran this on local (after ssh tunneling): iex --name debug@127.0.0.1 --cookie 'vk1285h/I)a6Q>T=;yY!hLDNjkW5?xz:nunHon4Jz(};KK!(C~zOoqh]a^XH5}3' --hidden -e ":observer.start" # doesn't connect to elixirapp@127.0.0.1
I encapsulated the cookie name inside single quotes so that it is evaluated properly by zsh. I thought that this wrapping of the cookie inside the quotes could be the issue. But then on local I ran ps aux | grep iex and checked the cookie name. It returned: