Trouble trying to start etop in production

Questions / Help
, , ,
1

Hi!

I’m trying to run :etop.start after entering a remote iex session in production to troubleshoot high memory usage in my application, but it seems that I can’t include the tool with my release.

Trying to run etop gives me a module :etop is not available error message and adding :etop to :extra_applications gives me ** (Mix) Could not start application etop: could not find application file: etop.app when I start the app locally.

I have no trouble starting etop locally in an iex session though.

I have Erlang/OTP 23 and Elixir Elixir 1.10.4 in my machine, while in production I have a Docker image that runs with Erlang/OTP 22 and Elixir 1.10.3.

1 Like
2

Hi @thiagomajesk, it seems the module is Etop instead?

3

Hi @rodrigues… This is not the version that comes with Erlang though.
I’m talking about this built-in functionality: https://erlang.org/doc/man/etop.html

4

Ah ok, TIL :sweat_smile:

5

Didn’t try this myself (locally etop just works), just found in stackoverflow that maybe adding :runtime_tools and :observer (?) to :extra_applications can help.

And that there’s also the etop CLI tool that might be able circumvent this if the target node doesn’t have the extra applications: etop -node testnode@myhost -setcookie MyCookie

6

Hi, again @rodrigues…

Phoenix by default already includes :runtime_tools on :extra_applications, but this is not enough to enable observable or etop (don’t know why). I haven’t tried using the etop CLI though.

7

I’m also trying to to use etop, but hitting a wall with a cryptic message after connecting etop to a node.

  1. Install etop on Alpine linux:

    apk add erlang-observer

  2. Run etop:

    /usr/lib/erlang/lib/observer-2.14/priv/bin/etop -node file_service@file-service-78b6d8d748-jhpd7 -setcookie [LONG_COOKIE_VALUE]

  3. It seems to connect OK - otherwise there’s pretty straightforward connection error message, but then gives this output:

    Erlang/OTP 25 [erts-13.2.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Eshell V13.2.2.1  (abort with ^G)
(etop@file-service-78b6d8d748-jhpd7)1>
=ERROR REPORT==== 26-Sep-2023::21:38:55.218442 ===
Error in process <8934.15452.0> on node 'file_service@file-service-78b6d8d748-jhpd7' with exit value:
{{badmatch,{error,{{badfun,#Fun<dbg.14.63636429>},
                   [{dbg,remote_start,1,[{file,"dbg.erl"},{line,339}]},
                    {dbg,remote_start,1,[{file,"dbg.erl"},{line,339}]},
                    {dbg,do_relay,2,[{file,"dbg.erl"},{line,966}]}]}}},
 [{dbg,do_relay,2,[{file,"dbg.erl"},{line,966}]}]}

Any ideas where to look what this bad match might be about & how to fix this?

As pointed out by the stacktrace, tried checking multiple source lines dbg.erl in erlang/otp repository on GitHub, but found the code there to be too low-level for me to understand… Can’t wrap my head around what’s going on.

Maybe important: the Erlang VM node I am connecting to does have :runtime_tools app running, but does not have :etop app (it’s a separate OTP app, right?) included in the release intentionally.