Stream consumes much more memory

Stream uses function thunks, which will use more memory, plus there is more passing of data, so the GC will need to run more often, though it doesn’t run often as it so it can reclaim it all en masse later, so it is using available RAM to make the operation faster. Remember, only use Stream when you need true unbounded or unknown bound operations or when your overall structure exceeds available memory, else keep with immediate constructs.

And yeah, 33k is basically nothing for that kind of stuff.

Thank you for your response.

Ok, this makes sense

I think an HTTP download is a good case to use Stream. Don’t you?

In fact, one of the main features is to stop the download after a maximum size, because the remote server can:

• send an invalid size header
• don’t send the file size at all
• send just junk data trying to fill our RAM or our disk

Ok, I agreed with this, until I found a big problem. I did the same test with a 300MB file and those are the results:

Benchmarking download...
Benchmarking stream...

Name               ips        average  deviation         median         99th %
stream            0.30         3.36 s     ±1.98%         3.36 s         3.40 s
download          0.29         3.46 s     ±5.99%         3.46 s         3.60 s

Comparison:
stream            0.30
download          0.29 - 1.03x slower

Memory usage statistics:

Name        Memory usage
stream          72.34 MB
download      0.00397 MB - 0.00x memory usage

So, when the library writes to the file the memory size is constant 4K. When I delegate to Stream it depends on the file size. It’s strange, isn’t it?

I’d like to investigate more about this, but I don’t know how to do it. Not enough knowledge about ErlangVM or debug tools.

Thank you!

Depends, can you use the data piecemeal or do you need all of the returned data en masse to work on it?

That all depends on the download API being used then, which one are you using?

I bet it’s storing to a binary and not being processed anywhere, binaries are extremely efficient especially cross-actors as they have their own global heap storage when above 64 bytes in size or so (as well as fast appending if only one owner, etc… etc… A stream will end up allocating a whole ton of binaries! But if your download API can write straight to a file then it can use the base BEAM calls, which use the kernel calls to pass the socket information straight over to the kernel so no real allocations need to be done. Need to see code to see what all is being done.

Well, I’m trying to do a generic library to stream HTTP request. So the main reason to use it is that you can use the data piece by piece. Another reason to use it could be to avoid an attack of huge files.

I’m not sure if I follow you here. The library can use the streaming options of :hackney, :ibrowse and :httpc. It checks the size of every chunk is received. When the sum of the chunk sizes is bigger than the given limit, it stops the download. It also checks the header size.

Exactly, you’re right. When the library receives a new chunk and it is written straight to the file, it only consumes 4KB. When I use a stream like this:

file = Temp.path!() |> File.open!([:write, :delayed])

  url
  |> Down.stream(backend: :httpc)
  |> Stream.each(fn c -> IO.binwrite(file, c) end)
  |> Stream.run()

File.close(file)

or like this:

  file_stream = Temp.path!() |> File.stream!([:write, :delayed])

  url
  |> Down.stream(backend: :httpc)
  |> Stream.into(file_stream)
  |> Stream.run()

is when it consumes so much memory.

It is open source: https://github.com/alexcastano/down/blob/master/benchs/download.exs

But of course, I know it is too much to ask you to take a look

If you can give me any advice to debug where the memory is consumed would be great.

Thank you for your time