File fails to download while working with streams

[fireproofsocks]

I’m working to download some large files… this lead me into working with streams. This post and the related article were helpful:

How to stream file from aws to client through elixir backend

I had the time to write down an idea of wrapper around HTTPoison, to make it an Elixir stream. defmodule HTTPDownload do def stream!(url) do Stream.resource( fn -> start_request(url) end, fn ref -> case receive_response(ref) do #returning the chunk to the stream {:ok, {:chunk, chunk}} -> HTTPoison.stream_next(ref) {[chunk], ref} {:ok, msg} -> IO.inspect(msg) HTTPoison.stream_next(ref) …

The basic pattern is:

url
|> HTTPDownload.stream!()
|> Stream.into(File.stream!("save.file"))
|> Stream.run()

Most files download fine, but the specific problematic file is a Walmart site map: https://www.walmart.com/sitemap_store_dept1.xml.gz – this isn’t a large file. I don’t know if the host is doing something to prevent programmatic download, but the download just hangs (whereas using a browser works no problem).

This all lead me to the http_stream package, but I’m experiencing errors using it (see bug: Example fails to download · Issue #7 · subvisual/http_stream · GitHub). I think everything works so long as I can provide some additional headers, including a timeout, but I can’t seem to find a client that offers streaming AND supports custom headers and options.

Can anyone explain how or why the request hangs? Or if there’s some other client that can stream files with options?

[al2o3cr]

Based on the stacktraces in that issue, there are two problems:

• the second error is because :hackney hasn’t been started

• the first one is because that version of Mint (1.1.0, released May 2020) uses an API that was removed in ssl-10.4, part of OTP 24.0 (released May 2021)

Mint.Core.Transport.SSL was also specifically updated in post-1.1.0 versions to support TLS 1.3’s cipher-selection preferences, which seems relevant since that URL is served with TLS 1.3.