Hello everybody,

I’m creating an HTTP wrapper to download files Stream. Currently, it works with :httpc, :hackney and :ibrowse and has the following functions:

• stream/2 which creates an Elixir Stream.
• read/2 which get the content and returns a string.
• download/2 which downloads to a file.

I realized that I can use just stream/2 to implement the other two functions. This would simplify a lot the library code, but before making the change I just wanted to benchmark. I’m surprised with the results for a simple test with a 1MB file. If I use the stream/2 function the memory consumption is much higher (with the same speed):

$  mix run benchs/download.exs
...

Memory usage statistics:

Name                     average  deviation         median         99th %
download                 3.18 KB     ±1.00%        3.20 KB        3.20 KB
stream                  32.58 KB     ±0.40%       32.62 KB       32.67 KB
stream_file_open        41.06 KB     ±0.44%       41.12 KB       41.17 KB

Comparison:
download                 3.20 KB
stream                  32.62 KB - 10.21x memory usage
stream_file_open        41.12 KB - 12.87x memory usage

$ mix run benchs/read.exs
...

Memory usage statistics:

Name           Memory usage
read_stream        26.39 KB
read                3.29 KB - 0.12x memory usage

The read benchmark is something like this:

read = fn -> {:ok, r} = Down.read(url, backend: :httpc); r end
read_stream = fn ->
  url
  |> Down.stream(backend: :httpc)
  |> Enum.into([])
  |> IO.iodata_to_binary()
end

The Down.read/2 function internally uses a very similar strategy to the read_stream function. Every time it receives a new chunk it appends to a list and when it finishes, it calls to the same IO.iodata_to_binary/1:

• down/lib/down.ex at master · alexcastano/down · GitHub
• down/lib/down.ex at master · alexcastano/down · GitHub

With the download function, the situation is very similar. Instead of inserting chunks in a list, I just write them to a file.

The Down.stream/2 function just sends received chunks to the PID using Stream.resource/3:

• down/lib/down.ex at master · alexcastano/down · GitHub

The benchmarks are here:

• https://github.com/alexcastano/down/blob/master/benchs/download.exs
• https://github.com/alexcastano/down/blob/master/benchs/read.exs

The full code can be found in GitHub - alexcastano/down: Elixir library for streaming, flexible and safe downloading of remote files · GitHub
I don’t use any buffer or any caching system.

If someone wants to make some test I use the following docker command to have an HTTP test server:

$ docker run --name httpbin --restart=unless-stopped -p 6080:80 -d kennethreitz/httpbin

I cannot find an explanation for this, at the moment. So my questions are:

• Do you know why is this happening?
• Should I be worried about this or 33K is acceptable? Of course, it would be nicer 3KB
• Are Benchee memory benchmarks accurate?

Thanks in advance.