Stream consumes much more memory

Hello everybody,

I’m creating an HTTP wrapper to download files Stream. Currently, it works with :httpc, :hackney and :ibrowse and has the following functions:

  • stream/2 which creates an Elixir Stream.
  • read/2 which get the content and returns a string.
  • download/2 which downloads to a file.

I realized that I can use just stream/2 to implement the other two functions. This would simplify a lot the library code, but before making the change I just wanted to benchmark. I’m surprised with the results for a simple test with a 1MB file. If I use the stream/2 function the memory consumption is much higher (with the same speed):

$  mix run benchs/download.exs

Memory usage statistics:

Name                     average  deviation         median         99th %
download                 3.18 KB     ±1.00%        3.20 KB        3.20 KB
stream                  32.58 KB     ±0.40%       32.62 KB       32.67 KB
stream_file_open        41.06 KB     ±0.44%       41.12 KB       41.17 KB

download                 3.20 KB
stream                  32.62 KB - 10.21x memory usage
stream_file_open        41.12 KB - 12.87x memory usage

$ mix run benchs/read.exs

Memory usage statistics:

Name           Memory usage
read_stream        26.39 KB
read                3.29 KB - 0.12x memory usage

The read benchmark is something like this:

read = fn -> {:ok, r} =, backend: :httpc); r end
read_stream = fn ->
  |> :httpc)
  |> Enum.into([])
  |> IO.iodata_to_binary()

The function internally uses a very similar strategy to the read_stream function. Every time it receives a new chunk it appends to a list and when it finishes, it calls to the same IO.iodata_to_binary/1:

With the download function, the situation is very similar. Instead of inserting chunks in a list, I just write them to a file.

The function just sends received chunks to the PID using Stream.resource/3:

The benchmarks are here:

The full code can be found in
I don’t use any buffer or any caching system.

If someone wants to make some test I use the following docker command to have an HTTP test server:

$ docker run --name httpbin --restart=unless-stopped -p 6080:80 -d kennethreitz/httpbin

I cannot find an explanation for this, at the moment. So my questions are:

  • Do you know why is this happening?
  • Should I be worried about this or 33K is acceptable? Of course, it would be nicer 3KB :slight_smile:
  • Are Benchee memory benchmarks accurate?

Thanks in advance.

1 Like

Stream uses function thunks, which will use more memory, plus there is more passing of data, so the GC will need to run more often, though it doesn’t run often as it so it can reclaim it all en masse later, so it is using available RAM to make the operation faster. Remember, only use Stream when you need true unbounded or unknown bound operations or when your overall structure exceeds available memory, else keep with immediate constructs. :slight_smile:

And yeah, 33k is basically nothing for that kind of stuff.


Thank you for your response.

Ok, this makes sense :slight_smile:

I think an HTTP download is a good case to use Stream. Don’t you?

In fact, one of the main features is to stop the download after a maximum size, because the remote server can:

  • send an invalid size header
  • don’t send the file size at all
  • send just junk data trying to fill our RAM or our disk

Ok, I agreed with this, until I found a big problem. I did the same test with a 300MB file and those are the results:

Benchmarking download...
Benchmarking stream...

Name               ips        average  deviation         median         99th %
stream            0.30         3.36 s     ±1.98%         3.36 s         3.40 s
download          0.29         3.46 s     ±5.99%         3.46 s         3.60 s

stream            0.30
download          0.29 - 1.03x slower

Memory usage statistics:

Name        Memory usage
stream          72.34 MB
download      0.00397 MB - 0.00x memory usage

So, when the library writes to the file the memory size is constant 4K. When I delegate to Stream it depends on the file size. It’s strange, isn’t it?

I’d like to investigate more about this, but I don’t know how to do it. Not enough knowledge about ErlangVM or debug tools.

Thank you!

Depends, can you use the data piecemeal or do you need all of the returned data en masse to work on it?

That all depends on the download API being used then, which one are you using?

I bet it’s storing to a binary and not being processed anywhere, binaries are extremely efficient especially cross-actors as they have their own global heap storage when above 64 bytes in size or so (as well as fast appending if only one owner, etc… etc… A stream will end up allocating a whole ton of binaries! But if your download API can write straight to a file then it can use the base BEAM calls, which use the kernel calls to pass the socket information straight over to the kernel so no real allocations need to be done. Need to see code to see what all is being done.

Well, I’m trying to do a generic library to stream HTTP request. So the main reason to use it is that you can use the data piece by piece. Another reason to use it could be to avoid an attack of huge files.

I’m not sure if I follow you here. The library can use the streaming options of :hackney, :ibrowse and :httpc. It checks the size of every chunk is received. When the sum of the chunk sizes is bigger than the given limit, it stops the download. It also checks the header size.

Exactly, you’re right. When the library receives a new chunk and it is written straight to the file, it only consumes 4KB. When I use a stream like this:

file = Temp.path!() |>!([:write, :delayed])

  |> :httpc)
  |> Stream.each(fn c -> IO.binwrite(file, c) end)


or like this:

  file_stream = Temp.path!() |>!([:write, :delayed])

  |> :httpc)
  |> Stream.into(file_stream)

is when it consumes so much memory.

It is open source:

But of course, I know it is too much to ask you to take a look :slight_smile:

If you can give me any advice to debug where the memory is consumed would be great.

Thank you for your time

1 Like