Web scraping tools

I think the choice depends on whether you need to evaluate js or not.

If you do, then in elixir there is hound [0] for this. I haven’t used it though since I usually use selenium with python. Actually, I usually try to avoid evaluating js completely by figuring out what apis it calls and calling them myself.

If you don’t, then just an html parser like floki [1] will do. You might be interested in reading the source code for magnetissimo [2] to see how a scraper works with it.

[0] GitHub - HashNuke/hound: Elixir library for writing integration tests and browser automation · GitHub
[1] GitHub - philss/floki: Floki is a simple HTML parser that enables search for nodes using CSS selectors. · GitHub
[2] GitHub - sergiotapia/magnetissimo: Web application that indexes all popular torrent sites, and saves it to the local database. · GitHub

You can try to combine httpoison, floki and poolboy.

There is a nice article here https://www.akitaonrails.com/2015/11/18/ex-manga-downloadr-um-exercicio-com-elixir/.

There’s also GitHub - mischov/meeseeks: An Elixir library for parsing and extracting data from HTML and XML with CSS or XPath selectors. · GitHub if you don’t want to use Floki for whatever reason

@_russellb: My list:

1. Floki - fetching data from HTML pages
2. HTTPoison - fetching files (also HTML pages)
3. Poison - decoding JSON for example when you have json in element attribute value
4. NimbleCSV - some data you can fetch from CSV files on some pages for example export search results to CSV
5. ExVCR - recording library for HTTPoison - useful in TDD (Test Driver Development)
6. Retry - just to retry
7. Hound - fallback for HTTPoison and Floki for pages using JavaScript frameworks/libraries

There is also Meeseeks (really good replacement for Floki), but for me it requires Rust that in my case I need to compile (Funtoo Linux) that takes lots of time on my old laptop.

I’ve done a lot of scraping work on the Node side. PhantomJS and Node’s NightmareJS are potent tools.

This thread introduced me to Hound. It can pair with PhantomJS and looks well done. I’d start there. You’re going to run into JS rendered HTML and it’s rarely beneficial to deconstruct their API.

This is a good list.

It is worth noting that while Floki defaults to using :mochiweb_html (an Erlang library) to parse HTML, it gives you the option to use the Rust library html5ever instead and I strongly suggest using that unless you have a compelling reason not to. :mochiweb_html is not HTML5 spec compliant and malformed HTML that renders correctly in your browser might not be parsed correctly by :mochiweb_html.

The Rust compilation time is annoying, but I think compilation should only occur when you update your Meeseeks or (for Floki) html5ever_elixir dependency or need to fetch dependencies for the first time, so it shouldn’t be a problem most of the time.

I have a small benchmark comparing Meeseeks and Floki performance if such things might matter.

@mischov: 100% agree, but I’m using this not for HTML5 pages, but for old and really bad written ASP.NET pages …

@Eiji If it works for your case, good.

I just want people to be aware that, while :mochiweb_html isn’t bad and will work well in many cases including some malformed ones, it won’t always behave like a browser when parsing HTML.

For instance, note the differences when parsing this very malformed HTML.

iex> html = "<p =a>One<a <p>Something</p>Else"
iex> :mochiweb_html.parse(html)
{"p", [{"=", "="}, {"a", "a"}], ["One", {"a", [{"<p", "<p"}], ["Something"]}]}

iex> Html5ever.parse(html)
{:ok,
 [{"html", [],
   [{"head", [], []},
    {"body", [],
     [{"p", [{"=a", ""}], ["One", {"a", [{"<p", ""}], ["Something"]}]},
      {"a", [{"<p", ""}], ["Else"]}]}]}]}

That’s a pretty extreme example, but it shows the kind of differences that are possible.

I just saw GitHub - nietaki/crawlie: A simple Elixir library for writing decently-performing crawlers with minimum effort. · GitHub on github. It seems quite neat, maybe check it out? =)

You might want to check excrawl (GitHub - mlankenau/excrawl: Elixir web crawler · GitHub). It is a slim DSL above floki. It produces a hash/array struct out of html defined by a dsl.