Increasing resource usage with Phoenix

is this running as a “release” eg distillery etc ?

also I notice reductions are up.. so I assume “work in progress” memory usage is also up - eg. you are processing more data/it’s taking longer time - so memory is released at a slower pace..

Oh sorry, forgot to say that I am using mix phx.server for now.

My part of the system (the DynamicSupervisor and friends) aren’t doing anything most of the time since the system is used very rarely at the moment.

The graphs indicate the memory is primarily allocated to processes - have you explored which processes use that memory? You could obtain the top offenders either with observer or some other tool (with recon, e.g. :recon.proc_count(:memory, 10))

Since I am using mix phx.server inside a Docker container, I cannot execute code in the context of the running application right now. I do plan to switch to releases tho. But it will take some days to have it filled up again before I can inspect it.

Debugging a system without a way to inspect that system is kind of a lost cause

You could potentially try some of the web-based observer replacements, but YMMV.

Haha, I know. I was just too lazy to setup the proper release. The project started as a technical POC to show its benefits and now I am preparing it for the real world.

EDIT: I just started setting up Distillery. Then in a few days I can investigate what’s going on.

Building the release locally works fine, but building it inside a container using bitwalker/alpine-elixir:1.8.1 errors with the following:

==> Generating start_erl.data
==> Generating vm.args from rel/vm.args
==> Generating sys.config from config/config.exs
==> Including ERTS 10.3.5 from /usr/lib/erlang/erts-10.3.5
==> Generating boot scripts
==> Release failed, during .boot generation:
        Undefined applications: [observer,wx]

I am mainly following Deploying With Docker - Distillery Documentation with some adjustments.

As docker usually doesn’t have a graphical display server running, most erlang images (especially those that target small image sizes) strip out wx and dependants. So you either need to install a version of erlang that includes wx and other missing applications, choose another base image that has them already, or drop them from your :extra_applications.

Ah, I understand. I dropped them for now. Locally I am using iex anyway. Just thought it won’t hurt to have it in the release too.

So, having it running for quite some time now, I saw memory usage go up and then suddenly go down again. GC goes a little bit crazy but I don’t think that I should worry about it. IO goes up over time but I guess that’s just the /metrics endpoint serving more stuff.

291730×729 89.3 KB

What I do not understand is that the Reductions keep increasing.

For the memory part, this are the top 10 processes. Nothing special there:

1> :recon.proc_count(:memory, 10)
[
  {#PID<0.1521.0>, 4119660,
   [
     :code_server,
     {:current_function, {:code_server, :loop, 1}},
     {:initial_call, {:erlang, :apply, 2}}
   ]},
  {#PID<0.1520.0>, 1577780,
   [
     :kernel_sup,
     {:current_function, {:gen_server, :loop, 7}},
     {:initial_call, {:proc_lib, :init_p, 5}}
   ]},
  {#PID<0.1741.0>, 1159136,
   [
     current_function: {:gen_server, :loop, 7},
     initial_call: {:proc_lib, :init_p, 5}
   ]},
  {#PID<0.9.0>, 689524,
   [
     :erl_prim_loader,
     {:current_function, {:erl_prim_loader, :loop, 3}},
     {:initial_call, {:erlang, :apply, 2}}
   ]},
  {#PID<0.1683.0>, 426596,
   [
     :prometheus_sup,
     {:current_function, {:gen_server, :loop, 7}},
     {:initial_call, {:proc_lib, :init_p, 5}}
   ]},
  {#PID<0.1514.0>, 319328,
   [
     :application_controller,
     {:current_function, {:gen_server, :loop, 7}},
     {:initial_call, {:erlang, :apply, 2}}
   ]},
  {#PID<0.1692.0>, 88620,
   [
     :release_handler,
     {:current_function, {:gen_server, :loop, 7}},
     {:initial_call, {:proc_lib, :init_p, 5}}
   ]},
  {#PID<0.1677.0>, 29628,
   [
     :hackney_manager,
     {:current_function, {:gen_server, :loop, 7}},
     {:initial_call, {:proc_lib, :init_p, 5}}
   ]},
  {#PID<0.2790.0>, 26656,
   [
     :inet_gethost_native,
     {:current_function, {:inet_gethost_native, :main_loop, 1}},
     {:initial_call, {:inet_gethost_native, :server_init, 2}}
   ]},
  {#PID<0.1611.0>, 26580,
   [
     :tls_connection_sup,
     {:current_function, {:gen_server, :loop, 7}},
     {:initial_call, {:proc_lib, :init_p, 5}}
   ]}
]