Massive increase in binaries using Finch

ijunaidfarooq

I am using finch in my application, what my application do is:

It sends a request to the camera for getting a jpeg, (Basic/Digest auth) and get the binary image, and then send it to another cloud.

so if I have 500 cameras they are sending 500 * 2 requests per second.

I was using HTTPoison before and my binary state on phoenix dashboard was like this

but when I deployed with Finch.

I had these pool settings with hackney

  :hackney_pool.child_spec(:snapshot_pool, timeout: 50_000, max_connections: 10_000),
  :hackney_pool.child_spec(:seaweedfs_upload_pool, timeout: 50_000, max_connections: 10_000),
  :hackney_pool.child_spec(:seaweedfs_download_pool, timeout: 50_000, max_connections: 10_000)

and I have these settings with Finch now

  {Finch,
   name: EvercamFinch,
   pools: %{
     :default => [size: 500, max_idle_time: 25_000, count: 5]
   }},

I want to know the reason of such massive binary increase?

UPDATE:

this is what I am using to make requests

defmodule EvercamFinch do
  def request(method, url, headers \\ [], body \\ nil, opts \\ []) do
    transformed_headers = transform_headers(headers)

    Finch.build(method, url, transformed_headers, body)
    |> Finch.request(__MODULE__, opts)
    |> case do
      {:ok, %Finch.Response{} = response} ->
        {:ok, response}

      {:error, reason} ->
        {:error, reason}
    end
  end

  defp transform_headers([]), do: []

  defp transform_headers([{key, _value} | _rest] = headers) do
    case is_atom(key) do
      true -> transform_headers(headers, :atom)
      false -> headers
    end
  end

  defp transform_headers(headers, :atom) do
    headers
    |> Enum.map(fn {k, v} ->
      {Atom.to_string(k), v}
    end)
  end
end

This is how my process look for NimblePool

even with small size and count. why the NimblePool has increased to 2500+?

18 comments

#troubleshooting #finch

0 3400 18

2021-01-04 15:03:12 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

keathley

I’m not really surprised that NimblePool has a lot of processes based on your settings. Your configured for 5 pools with 500 connections per pool.

Post #1

ijunaidfarooq

I changed it to

  {Finch,
   name: EvercamFinch,
   pools: %{
     :default => [size: 50, max_idle_time: 25_000, count: 5]
   }},

and still

what you suggest?

500 * 2 request per second?

also why is there massive increase in binaries?

Post #2

keathley

I can’t really make a suggestion on your pools since it’s dependent on the servers that you’re interacting with and what they’re capable of handling. My suspicion is that the responses you’re dealing with are quite large (since you’re dealing with images) and the processes dealing with those responses aren’t being killed. Large binaries have historically caused issues for the garbage collector. I’m not sure if that’s changed in recent OTP releases or not. Killing the process can help reduce memory bloat. You can also use recon, observer, and the built in beam tools to try to track down which processes are causing you issues.

Post #3

sb8244

Author of Real-Time Phoenix

To expand on what Chris is suggesting. You can trigger GC manually across all processes by using Process.list() |> Enum.each(&:erlang.garbage_collect/1). If you do this and the memory drops, then you know you have a memory leak.

Here’s a post on an old memory leak I tracked down and some options for how to handler them Stephen Bussey - Elixir Memory - Not Quite Free

Post #4

ijunaidfarooq

I am not interacting with servers but I am interacting with cameras,

they have an IP , port , username and password..

Also the API which I am using to upload to cloud is capable of 1000+ requests per second.

I have decreased the pool settings but isn’t it strange that with HTTPoison and Hackney the binaries were in control? or the Hackney was hiding the memory leak somehow?

Post #5

ijunaidfarooq

Okay thank you.

but what are the out comes if I will let this continue to happen?

Would that crash the application at some point?

Post #6

keathley

It’s really hard to say without understanding what your code is doing. I have a general idea, but in this case the details matter. It would be useful to see which processes are using up the most memory. It’s definitely interesting and worth digging into, but I can only speculate without more information.

Post #7