Remote Observer Connection Issues

Questions / Help
,
1

Hi,

I recently migrated from distillery to mix release. While on distillery, I had setup my release such that I was able to debug the remote production node with observer. After migrating to mix release, I find that whenever I try to connect to the remote node via observer (it shows up in the observer Nodes drop down menu), I get this error: Error occurred on remote node and the following error message in the console: [error] [node: :"production_node@127.0.0.1", call: {:observer_backend, :sys_info, []}, reason: {:badrpc, :nodedown}]

I have all the dependencies installed on the release (observer, wx and runtime_tools), so I am not sure why I am seeing this issue. Furthermore, it is weird that it worked on distillery and is giving me issues now. Anyone have any ideas?

Thanks!

2

Don’t know the difference.

Few things:

  1. It say production_node@127.0.0.1. Unless you connecting from the same node, or through ssh tunneling this will not work.
  2. Have you set the cookie to the same value on both nodes?

Additionally you don’t need wx and I don’t think you need observer on the production node to run observer remotely.

3

Thanks for replying. I am tunneling via ssh and setting the same cookie in both places. I had the same setup previously when releasing vis Distillery as well when it did work, so it not working when releasing via Mix makes me suspect that maybe I am not packaging some dependency correctly (esp since the remote production node is discoverable locally)

1 Like
4

Do you get anything in the logs of the production node?

5

I just checked, nothing related to this unfortunately :frowning:

6

I am having an exact same issue while migration from Distillery to Elixir 1.9 Mix Release, I would also appreciate if anyone can shed some light on this.

1 Like
7

I had the exact same issue. I fixed it but then another problem occurred.
To fix, check

  • You have the following in env.sh[.eex]. Note that using short names won’t work, or I couldn’t get it to work. Use long names.
export RELEASE_DISTRIBUTION=name
export RELEASE_NODE=<%= @release.name %>@127.0.0.1

  • Restrict the port used by node to a specified one, e.g. adding -kernel inet_dist_listen_min 60606 inet_dist_listen_max 60606 to ELIXIR_ERL_OPTIONS in env.sh[.eex].

  • Forward ports like ssh -L 4369:127.0.0.1:4369 -L 60606:127.0.0.1:60606 nodehost. Note you shouldn’t have epmd running in your system, and you have to use the remote one. To make sure you are not running epmd locally, run epmd -kill before opening ssh session.

  • Start IEx shell like iex --hidden --cookie 'yourcookie' --name local@127.0.0.1

Now when I run Node.connect :"distname@127.0.0.1" I get true and nodes connect. But when I do :observer.start locally and select the remote node in Nodes menu, I get the following error:

16:22:06.248 [error] [node: :"distname@127.0.0.1", call: {:observer_backend, :sys_info, []}, reason: {:badrpc, {:EXIT, {:undef, [{:observer_backend, :sys_info, [], []}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 197]}]}}}]

Then I found about :runtime_tools which contains :observer_backend. See Troubleshooting in http://blog.plataformatec.com.br/2016/05/tracing-and-observing-your-remote-node.

After this, it works like a charm!

7 Likes
8

Adding these two lines did the trick! Thanks a lot @vfsoraki :smiley: