Download - the easiest way to download files from the internet

Your Libraries & OS Mentoring Libraries
1

Hey there!

I wrote a download elixir package which does exactly what its name about - an easy way to download files.

I saw solutions about how to do it with HTTPoison but these solutions have disadvantages:

  • The whole response is loading to the RAM, and only after then is going to the File.write/1. You have to choose async HTTPoison request to escape high memory consumption.
  • There is no restriction of the file size to download. If you handle, saying, user input and then you’re trying to download any file user provided, your server can go down for downloading a file which has 1TB size.

download depends on HTTPoison and successfully sidestep these issues.

Enjoy :wink:

6 Likes
2

@asiniy: Nice, but I have a few ideas:

  1. Support for POST requests, because some pages requires it
  2. Support for parameters (GET) and data (POST), because it’s easier to read code (parameters) + some pages requires POST with data
  3. Support for finders, for example: Download.from(url, finder: Download.Finders.Video)
  4. Support for Traditional units
1 Like
3

I do not like that page you linked @Eiji, it seems to confuse metric and binary prefixes.

Even gives megabyte as an example for binary and mebibyte as example while on the linked pages about binary and metric prefixes they are the other way round.

So I am not sure what you exactly mean by “traditional units”?

1 Like
4

@Eiji

  1. Download via POST? Never heard about it
  2. Take a look at 1.
  3. finder? What should it find?
  4. I dislike units because of mebibyte vs megabyte conflict (like @NobbZ said)
5

@NobbZ: tip: look at table headers :slight_smile:

6

@asiniy:

ehh, you don’t heard about lots of sites then … you are so lucky! :smiley:
Imagine that you need something like:
a) go to main page (yes, it’s important)
b) click links + follow redirects + follow redirects to https version + follow redirects in JavaScript
c) submit some forms
d) look for download link (generated for current session, that is initialized at main page)
e) FINALLY: download this file - APS.NET old pages
*) optionally JavaScript code could be minimized, so it’s harder to search what it do
*) really often pages like that have invalid HTML, so your parser needs to be really well tested

Imagine that you want to download video from youtube page and you call this method with normal video page url and then finder is looking for video.

7

Table headers aren’t saying anything, since the table itself is based on false nomenclature in the abstract.

Also in the table they use (warning, I do use correct names now!) metric prefixes but binary exponenents.

That article you linked mixed everything up. Please explain what behaviour you really want.

8

Okay,

  1. I cloned the repo.
  2. get all dependencies
  3. mix
  4. iex -S mix
  5. download(“youtubeurl.com”)

returned

{:ok, "/home/ijunaidfarooq/download/watch?v=P7nQQ5dNCpM"}

very fast.

BUT the file is not playable. VLC gives such errors

ijunaidfarooq@ijunaidfarooq-HP-ProBook-450-G3:~/download$ vlc watch\?v\=P7nQQ5dNCpM 
VLC media player 2.2.1 Terry Pratchett (Weatherwax) (revision 2.2.1~trusty2)
[0000000002605018] core libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action 
QVariantMap DBusMenuExporterDBus::getProperties(int, const QStringList&) const: Condition failed: action
9

I would think you just downloaded the html file.

You probably need to use something like https://github.com/rg3/youtube-dl/ to get to the video file - you might want to use Porcelain to interface it https://github.com/alco/porcelain

1 Like
10

Thanks. I know that.

My question was: what this project do? Download anything or not from youtube, Which I think it’s not working with youtube.

11

This project is about downloading a single resource from the specified URL. It does not any magic to determine what the resource actually is. It gets saved as is and then you get path of the saved file.

This is in contrast to using HTTPoison without thinking about it and loading the full response into RAM.

One can get the same behaviour with HTTPosion directly though using async requests, but this package makes it a bit easier to process those.

3 Likes
12

Okay thanks.

just as a question is that possible in Elixir and with HTTPoison to download something from Youtube? as we have dlls build in Python, Can Elixir do that too? I want to build that, But I have no clue where to start it.

13

I’m sure it is able to do so.

At the end, it is just reading a socket and dumping received data to disk.

Important part is, that you probably need to implement the protocol used from scratch. I’m not what youtube actually uses to stream and therefore I can’t know if this protol has already been implemented in elixir or erlang.

Also to be honest, I’d just delegate to one of the many available ready-made and well supported command line downloaders and orchestrate them through a worker pool.

1 Like
14

Important to note is that with, for example, youtube-dl you also get support for downloading from Twitch and vimeo.

2 Likes