Down - library for streaming and safer downloads

Your Libraries & OS Mentoring Libraries
1

Hello,

I’ve just published Down, an HTTP wrapper for streaming and safer downloads.

The main features are:

Down.download("http://example.com/image.jpg", max_size: 5 * 1024 * 1024) # 5 MB
{:error, :too_large}

iex> Down.stream("https://elixir-lang.org") |> Enum.find_value(&Regex.run(~r/\<title\>(.*)\<\/title\>/i, &1, capture: :all_but_first))
["Elixir"]

It is compatible with :hackney, :httpc, :ibrowser and Mint. All of them are optional dependencies.

You can find more information in Down — down v0.0.1

The code: GitHub - alexcastano/down: Elixir library for streaming, flexible and safe downloading of remote files

The package is in alpha state, but it works fine. I’d like to add a couple of features and to improve the tests and the documentation. Suggestions–I don’t have experience publishing libraries–and help are very welcomed.

I hope you enjoy it :slight_smile:

9 Likes
2

Am I correct in guessing that streaming with Mint is complicated by the socket behavior being active?

3

With :hackney, :httpc and :ibrowse I can pause the stream at any moment, so they are perfect if you want to analyze the remote file slowly. I couldn’t do the same with Mint for the moment.

4

Yeah, I’ve also been struggling with that recently.

Best solution I’ve come up with thus far is to queue the chunks up in a way that can be demanded by the stream.

I did see that there are a couple places in Mint where they set the socket to {:active, :once}, so there might be a way to do that and drive the process more like you can in the other libs.

5

I want to do the same! I want to implement a buffer, but with a max_buffer size tough. I’ll need to stop the download to have a right behaviour with Mint backend.

This is a question I wanted to ask to Mint developers. Maybe can @ericmj tell us if this will be a feature in a future release? :slight_smile:

1 Like
6

Or maybe @whatyouhide can speak to it?

7

With Mint we set active: :once so it is streaming already. If you don’t call Mint.stream/2 on a TCP/SSL message, we won’t set active: :once again so effectively you can block already. If you want to actually stop receiving stuff from the socket, then you should set active: false but we don’t support doing that through Mint yet. Is that what you mean?

2 Likes
8

This might just be a misunderstanding then.

Since the docs say:

The connection socket runs in active mode

My assumption was that it’s run in {:active, true}, not {:active, once}. I believe this used to be the case?

1 Like
9

I don’t believe we ever used active: true. I’ll fix the docs to clarify this later today.

3 Likes
10

Sounds good, thanks!

Looks like it moved from active: true to active: :once six months ago. Set sockets to active: :once · elixir-mint/mint@943dff4 · GitHub

Edit: Which apparently I should have remembered because there’s a record of me talking about it in my company Slack when it happened. :smiley:

2 Likes