EMPD ports prevent Obverver from running remotely

Questions / Help
,
#1

Background

I have invested the past 2 days trying to connect my localhost machine to a production machine so I can use :observer.start() and see why it is consuming so much CPU.

At this point I am so desperate I decided to scratch everything and try another approach. This time I read the entirety of this discussion:

But to no avail.

My approach

This new approach is based on this tutorial (which originated the above mentioned post)

On 1st terminal in local machine:

ssh user@remote-host "epmd -names"
    epmd: up and running on port 4369 with data:
    name my_super_duper_app at port 42267
ssh -L 4369:localhost:4369 -L 42267:localhost:42267 remote-host

On 2nd terminal in local machine:

erl -name debug@127.0.0.1 -setcookie super_duper_cookie -remsh my_super_duper_app@remote-host

Whereremote-host is the public IP of the machine I want to connect to. The rest should be self explanatory.

Problem

After executing the command:

ssh -L 4369:localhost:4369 -L 42267:localhost:42267 remote-host

I am logged into the remote machine. This is good. What is not good is the message I see at the top:

bind [127.0.0.1]:4369: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 4369
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.15.0-1026-gcp x86_64)
…

Of course the port 4369 is being used. It is the port where empd is running. What does it mean it cannot bind and listen to it? which machine do I have to kill to fix this?

As a consequence (or so I believe), when I run the command of the 2nd terminal I see this:

*** ERROR: Shell process terminated! (^G to start new job) ***

I am so confused. What am I missing?

#2

So the first part is according to my notes, but for the second part I have:
iex --name local@127.0.0.1 --cookie $COOKIE --hidden

Then in iex do: :observer.start()
In the observer do: Nodes > Connect Node (menu)
Insert: myapp@127.0.0.1

Now your observer should show the information for the remote node.

You probably could also connect the node in iex as code, but that’s how I’ve done it by now.

#3

Yep, i tried that way as well. The only thing it tells me is: “connect failed”.

Which is more aggravating having in mind that proper English dictates it should be “connection failed” or something among those lines :stuck_out_tongue:

Back at stage 0. I just don’t know what else I can do to troubleshoot this. Not even journalctl -f is showing any logs now…

Doing erl -version in the remote machine returns:

Erlang (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 9.3

Perhaps this is the issue? Perhaps this version is just too old?

Running an IEx session within an app (running as a release)?
#4

That’s how I do it, if it helps.

# ssh port forwarding

$  ssh APP@HOST -L4369:localhost:4369 -L47636:localhost:47636

# remote shell

$  iex --name USR@127.0.0.1 --cookie COOKIE --remsh APP@127.0.0.1

# observer

$  iex --name USR@127.0.0.1 --cookie COOKIE
> Node.connect :"APP@127.0.0.1"
> :observer.start
# Change Node in GUI
#5

Is epmd running locally? In my understanding (which could be wrong) you don’t want epmd running locally in this setup, only remotely.

#6

Don’t you mean

$ ssh remote-user@remote-ip -L4369:localhost:4369 -L47636:localhost:47636

instead?

iex --name USR@127.0.0.1 --cookie COOKIE --remsh APP@127.0.0.1

I take it you execute this in the remote machine right?
What do you do if the app is already running and you can’t use iex? (which is the case)

  iex --name USR@127.0.0.1 --cookie COOKIE
> Node.connect :"APP@127.0.0.1"
> :observer.start
# Change Node in GUI

I assume this is done in localhost. What is the name of the app in the UI? Doesn’t show up for me :frowning:

Ohh, this is quite likely true. How do I kill EMPD processes? Looks like I have to look it up.

#7

Something like

sudo lsof -t -i:4369

Should show you the process that has bound port 4369 locally. So you can run sudo lsof -t -i:4369 | xargs kill

If lsof isn’t working for you you could also try something like netstat -anl |grep 4369

1 Like
#8

yes

This is what I run locally to connect to the remote node (the app in production, for example). USR is a local name, COOKIE is the cookie in the remote node, and APP the user in remote host.

Yes, you run it local just like the other iex command. The difference is that I use this one to run observer. The other to use the IEx.

You will see the applications in the observer menu, “Nodes”. There you should be able to switch to the remote application you wanna observe.

Does it work now? :smiley:

2 Likes
#9

What dark magic is this?

The past 2 days of my life have been consumed by this. And you, out of nowhere, somehow managed to bring me a solution. I never thought this would work. To be honest, I quite don’t understand why, but who cares, it works !

Summary

To summarize (since I am sure I will forget about this 4 steps of hell) I am putting it here everything I did.

All steps in local machine:

  1. get the ports from the remote server:

    ssh remote-user@remote-ip "epmd -names"

    epmd: up and running on port 4369 with data:
    name super_duper_app at port 43175

2: create a ssh tunel with the ports:

ssh remote-user@remote-ip -L4369:localhost:4369 -L43175:localhost:43175

  1. On another terminal in your local machine, run a iex terminal with the cookie the app in your remote server is using. Then connect to it and start observer:

    iex --name observer@127.0.0.1 --cookie super_duper_cookie

    Node.connect :"super_duper_app@127.0.0.1"
    true
    :observer.start

With observer started, select the machine from the Nodes menu.

2 Likes
#10

Aren’t step 1 and 2 the same as in your original post? It sounds like the major issue you fixed is whatever was originally causing you this error after step 2:

bind [127.0.0.1]:4369: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 4369
1 Like
#11

I have this module that generate the instructions to connect that I’ve posted here, so you don’t need to get the epmd port every time. You need to replace some module variables, and beware @app is the same as remote user for us, not sure if it’s just gonna work for you.

Cheers

2 Likes
#12

Following your suggestions to kill processes using the port also help immensely, that is true.

Thanks for the script and for the help. We will change some parts of the script but the overall idea is clear!

1 Like