Interesting question. I don’t know of any library, but I’ve recently built something similar - in my case the source of the data is a rest API providing data about invoices and invoice lines. My code abstracts away the API’s details and creates an Elixir stream that can be consumed. In my case the stream generates a series of invoice lines (each one is a maps), whereas in your case it’s generating bytes.

Also, if I understand your use case correctly, each stream generates a single HTTP request. In my case however, it takes a large number of calls to different endpoints to extract all the data that I need. Similar to your case, sometimes some calls may fail (for different reasons) and I want the stream to protect the consumer from this. Therefore, I built a retry mechanism in the stream that retries failed requests a number of times. As long as the maximum number of retries is not exceed, the consume won’t notice anything and will get its data in the end.

In my case the retry mechanism was simple - just retry the failed request. I guess your case is more complicated because each request fetches a large amount of data and you don’t want to throw away what you’ve already downloaded if the request fails in the middle.

I’m sure you already have good ideas on how to implement the error recovery in case you haven’t found a library.

From my part, a possible idea could be to split the large download into smaller requests that use Range requests from the start, so that none of them exceeds the connection timeout limit.

If retries on API failure works, then maybe the following library can help:

https://github.com/wojtekmach/req

I feel like it’s responsibility of providers to have resumability in mind when designing services. Like chunks or Torrents for download, & multipart or Tus protocol for upload.

If the source is well designed, then perhaps the following library can help:

https://github.com/dashbitco/broadway

Hi, you can take a look at this article from [Poeticoding] - Elixir stream and large http response processing (Elixir Stream and large HTTP responses: processing text). Hope it help.