How to download big files?

Questions / Help
4

If the file is big, it might not fit into memory, so it’s better to use stream_to option with httpoison, and append to the file using IO.binwrite.

So it’ll be something like

def download!(file_url, filename) do
  file = if File.exists?(filename) do
    File.open!(filename, [:append])
  else
    File.touch!(filename)
    File.open!(filename, [:append])
  end
  
  %HTTPoison.AsyncResponse{id: ref} = HTTPoison.get!(file_url, %{}, stream_to: self())
  
  append_loop(ref, file)
end

defp append_loop(ref, file) do
  receive do
    %HTTPoison.AsyncChunk{chunk: chunk, id: ^ref} ->
      IO.binwrite(file, chunk)
      append_loop(ref, file)
    %HTTPoison.AsyncEnd{id: ^ref} ->
      File.close(file)
    # need something to handle errors like request timeout and such
    # otherwise it will loop forever
    # don't know what httpoison returns in case of an error ...
    # you can inspect `_other` below to find out
    # and match on the error to exit the loop early
    _other ->
      append_loop(ref, file)
  end
end

Note that receive won’t work in a genserver callback.

7 Likes