EMPD ports prevent Obverver from running remotely

So the first part is according to my notes, but for the second part I have:
iex --name local@127.0.0.1 --cookie $COOKIE --hidden

Then in iex do: :observer.start()
In the observer do: Nodes > Connect Node (menu)
Insert: myapp@127.0.0.1

Now your observer should show the information for the remote node.

You probably could also connect the node in iex as code, but that’s how I’ve done it by now.

Yep, i tried that way as well. The only thing it tells me is: “connect failed”.

Which is more aggravating having in mind that proper English dictates it should be “connection failed” or something among those lines

Back at stage 0. I just don’t know what else I can do to troubleshoot this. Not even journalctl -f is showing any logs now…

Doing erl -version in the remote machine returns:

Erlang (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 9.3

Perhaps this is the issue? Perhaps this version is just too old?

That’s how I do it, if it helps.

# ssh port forwarding

$  ssh APP@HOST -L4369:localhost:4369 -L47636:localhost:47636

# remote shell

$  iex --name USR@127.0.0.1 --cookie COOKIE --remsh APP@127.0.0.1

# observer

$  iex --name USR@127.0.0.1 --cookie COOKIE
> Node.connect :"APP@127.0.0.1"
> :observer.start
# Change Node in GUI

Is epmd running locally? In my understanding (which could be wrong) you don’t want epmd running locally in this setup, only remotely.

Don’t you mean

$ ssh remote-user@remote-ip -L4369:localhost:4369 -L47636:localhost:47636

instead?

iex --name USR@127.0.0.1 --cookie COOKIE --remsh APP@127.0.0.1

I take it you execute this in the remote machine right?
What do you do if the app is already running and you can’t use iex? (which is the case)

  iex --name USR@127.0.0.1 --cookie COOKIE
> Node.connect :"APP@127.0.0.1"
> :observer.start
# Change Node in GUI

I assume this is done in localhost. What is the name of the app in the UI? Doesn’t show up for me

Ohh, this is quite likely true. How do I kill EMPD processes? Looks like I have to look it up.

Something like

sudo lsof -t -i:4369

Should show you the process that has bound port 4369 locally. So you can run sudo lsof -t -i:4369 | xargs kill

If lsof isn’t working for you you could also try something like netstat -anl |grep 4369

yes

This is what I run locally to connect to the remote node (the app in production, for example). USR is a local name, COOKIE is the cookie in the remote node, and APP the user in remote host.

Yes, you run it local just like the other iex command. The difference is that I use this one to run observer. The other to use the IEx.

You will see the applications in the observer menu, “Nodes”. There you should be able to switch to the remote application you wanna observe.

Does it work now?

What dark magic is this?

The past 2 days of my life have been consumed by this. And you, out of nowhere, somehow managed to bring me a solution. I never thought this would work. To be honest, I quite don’t understand why, but who cares, it works !

Summary

To summarize (since I am sure I will forget about this 4 steps of hell) I am putting it here everything I did.

All steps in local machine:

1. get the ports from the remote server:

   ssh remote-user@remote-ip "epmd -names"

   epmd: up and running on port 4369 with data:
   name super_duper_app at port 43175

2: create a ssh tunel with the ports:

ssh remote-user@remote-ip -L4369:localhost:4369 -L43175:localhost:43175

3. On another terminal in your local machine, run a iex terminal with the cookie the app in your remote server is using. Then connect to it and start observer:

   iex --name observer@127.0.0.1 --cookie super_duper_cookie

   Node.connect :"super_duper_app@127.0.0.1"
   true
   :observer.start

With observer started, select the machine from the Nodes menu.

Aren’t step 1 and 2 the same as in your original post? It sounds like the major issue you fixed is whatever was originally causing you this error after step 2:

bind [127.0.0.1]:4369: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 4369

I have this module that generate the instructions to connect that I’ve posted here, so you don’t need to get the epmd port every time. You need to replace some module variables, and beware @app is the same as remote user for us, not sure if it’s just gonna work for you.

Cheers