OvermindDL1

OvermindDL1

Hmm, you should immediately see a lot of info, like here is what I get:

iex(devserver2@127.0.0.1)2> :etop.start()

========================================================================================
 'devserver2@127.0.0.1'                                                    21:11:05
 Load:  cpu         0               Memory:  total       62468    binary       1427
        procs     384                        processes   24416    code        22299
        runq        0                        atom         1017    ets          2741

Pid            Name or Initial Func    Time    Reds  Memory    MsgQ Current Function
----------------------------------------------------------------------------------------
<0.1027.0>     'Elixir.FileSystem.B     '-'43147764  689568       0 gen_server:loop/7
<0.286.0>      'Elixir.Phoenix.Code     '-'30204171 5692948       0 gen_server:loop/7
<0.9.0>        erl_prim_loader          '-'18242316  689524       0 erl_prim_loader:loop
<0.1040.0>     'Elixir.DBConnection     '-'17875451   25164       0 gen_server:loop/7
<0.1026.0>     phoenix_live_reload_     '-'11739249   42436       0 gen_server:loop/7
<0.277.0>      cowboy_clock             '-' 9445910    8932       0 gen_server:loop/7
<0.1.0>        erts_code_purger         '-' 7799639   44776       0 erts_code_purger:wai
<0.2.0>        erts_literal_area_co     '-' 6650619    2688       0 erts_literal_area_co
<0.1048.0>     'Elixir.DBConnection     '-' 4554875    4176       0 erlang:hibernate/3
<0.1044.0>     'Elixir.DBConnection     '-' 4494410    4176       0 erlang:hibernate/3
========================================================================================

Which keeps updating every few seconds (my dev server here is not under heavy use at all, or really any use). You have to kill your shell to stop it though so be careful not to kill your server instead, hence why remote shells are best. :slight_smile:

Just as an efficiency check, they are keeping an active TCP connection to the server to do that yes? Setting up and tearing down tons of HTTPS connections on Gateways does have a tendency to kill them pretty badly even though elixir handles it well.

0
Post #11
AyubOpen

AyubOpen OP

I type erl and I get the below shell

Eshell V9.0 (abort with ^G)
1>
here I type :etop.start()

not sure if I am in the correct shell

0
Post #12
OvermindDL1

OvermindDL1

Oh that’s erl! You’ll want to run etop:start() instead there, that’s the old Erlang shell. :slight_smile:

Use iex for the elixir shell in comparison.

And don’t forget to connect a remote shell to the other node unless you are spooling up the full server.

If you are using distillery it has a built-in easy command for it though, I highly recommend it, something like `my_server rpc ‘:etop.start()’. :slight_smile:

0
Post #13
AyubOpen

AyubOpen OP

I will try it out, I use the below command to build and execute

mix deps.get
MIX_ENV=final mix compile
MIX_ENV=final mix release

1
Post #14
AyubOpen

AyubOpen OP

I will have to investigate this further,I tried it on dev and if use iex and then try to kill the shell its shutting down the who beam process :slight_smile:

1
Post #15
OvermindDL1

OvermindDL1

Correct, it takes over the shell that it is run on so you want to use a remote shell or RPC or so. :slight_smile:

0
Post #16
axelson

axelson

Scenic Core Team

Have you check if the backend database is getting overwhelmed?

2
Post #17
OvermindDL1

OvermindDL1

Even if it were then it wouldn’t be giving back a 504, that’s a response when there isn’t an accessible server at all on the reverse proxy. Although if the reverse proxy had the timeout set too short and the database was overloaded then that could cause it, hmm, worth checking!

0
Post #18
axelson

axelson

Scenic Core Team

Yeah that was my thinking exactly.

1
Post #19
AyubOpen

AyubOpen OP

We dont have a reverse proxy, requests hit cloud flare which is just doing dns resolution, which resolves to aws load balancer where the vm is defined as the target. I only see a deregistration delay of 300 seconds defined for alb.

The database is an aws RDS postgresql and for it the cpu utilization for the instance hits 50% at peak and the write iops hits 6000. When I reindexed the tables the write iops came down. will try to monitor the server performance today to verify if I need to do autovaccum and reindex regularly

0
Post #20