Hello all, I have a machine in production running an elixir application (no access to
iex, only to
erl) and I am tasked with running an analysis on why we are consuming so much CPU. The idea here would be to launch observer, check the processes tab and see the processes with the most reductions.
How am I connecting?
To connect I am following a tutorial from a blog:
Their instructions are as follows:
- launch the app in the production machine with a cookie and a name
- from local run:
ssh user@public_ip "epmd -names" to get the name of the app and the port used
- create a ssh tunel to the remote machine:
ssh -L 4369:user@public_ip:4369 -L 42877:user@public_ip:42877 user@public_ip (4369 is the epmd port by default, 42877 is the port of the app)
- connect to the remote machine via public ip:
erl -name "user@public_ip" -setcookie "mah_cookie" -hidden -run observer
And now in theory I should be able to use observer on the machine. Instead however I am greeted with the following error:
Protocol ‘inet_tcp’: register/listen error: epmd_close
- What am I doing wrong?
I read that in older version of erlang I can’t monitor a machine with observer if they are in different networks (which is the case, because I want to monitor this machine from my localhost) but I didn’t find any information regarding modern days.
- If this is in fact impossible, what alternatives do I have?
At which step do you get this error?
erl -name "user@public_ip" -setcookie "mah_cookie" -hidden -run observer?
Right after typing it. I assume it would be step 0
In the remote machine, when I try to connect and fail, I get the following error message:
channel 5: open failed: administratively prohibited: open failed
What does it all mean?
After an hour of scouring the dark side of internet, I decided to use
sudo journalctl -f to check all the logs of the machine and I found this:
machine public ip:
99.999.99.999 (obviously not real)
channel 3: open failed: administratively prohibited: open failed
my_app_name sshd: error: connect_to firstname.lastname@example.org: unknown host (Name or service not known)
my_app_name CRON: pam_unix(cron:session): session closed for user flame
so it tells me, unknown host ?? I am confused since 99.999.99.999 is the public IP of the machine itself!
ifconfig show? Maybe the machine you are running on doesn’t know its public ip.
shows the following:
ens4 Link encap:Ethernet HWaddr 42:01:0a:84:00:10
inet addr:10.132.0.16 Bcast:10.132.0.16 Mask:255.255.255.255
inet6 addr: fe80::4001:aff:fe84:10/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1460 Metric:1
RX packets:11755 errors:0 dropped:0 overruns:0 frame:0
TX packets:11770 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:31795901 (31.7 MB) TX bytes:8658486 (8.6 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:57 errors:0 dropped:0 overruns:0 frame:0
TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:3801 (3.8 KB) TX bytes:3801 (3.8 KB)
Don’t ask me what any of it means
The public IP is indeed not shown anywhere in the list …
Maybe try using the loopback interface to connect to the erlang node after you’ve setup the ssh connection to the remote machine.
Can you give a detailed example on how to do that?