Ok. Looks like the Meeseeks also have Xpath support. I will check that.
oltarasenko
mischov
I want to be clear that just because it’s Rust doesn’t mean it’ll be faster than the mochiweb_html parser all the time. My observation is that it can be faster, particularly for large input, but the mochiweb parser does a less thorough job and can be faster on small input, particularly when you take into account NIF overhead. That said, using Rust does seem to have a very positive impact memory-wise in most cases.
The really big reason I advocate Rust is because it lets us use html5ever which mean we’re getting an HTML5 compliant parser that will give us very similar results to a browser when parsing HTML, which means that it generally will handle malformed content in a more desirable fashion than mochiweb_html.
I will try to get a benchmark together this evening based on the code in your post.
3
mischov
@oltarasenko Took me a little longer than planned, but put together the promised benchmark, which compares the version of the HTML parsing and extraction code from your post against a version using the optimizations I suggested and a Meeseeks translation of the optimized version.
Name ips average deviation median 99th %
Meeseeks 225.19 4.44 ms ±3.55% 4.39 ms 5.02 ms
Floki optimized 166.03 6.02 ms ±7.36% 5.96 ms 7.82 ms
Floki unoptimized 55.46 18.03 ms ±6.26% 17.77 ms 22.34 ms
Comparison:
Meeseeks 225.19
Floki optimized 166.03 - 1.36x slower +1.58 ms
Floki unoptimized 55.46 - 4.06x slower +13.59 ms
Memory usage statistics:
Name Memory usage
Meeseeks 0.42 MB
Floki optimized 6.25 MB - 14.98x memory usage +5.84 MB
Floki unoptimized 20.90 MB - 50.04x memory usage +20.48 MB
You can see the source along with some notes in the benchmark repo.
5
dimitarvp
jacksmith
For web scraping, you can try this tool https://app.scrape.works/create-project to explore more scraping information.
1
oltarasenko
I wonder if it’s built with elixir? E.g. I did not find any information regarding how the framework works.
0
oltarasenko
Btw here I have made another article about using Crawly with Tensorflow, which allows to organize machine learning project: Build Machine Learning Elixir Project
0
dogweather
I’m really excited about giving Crawly a try. I’m using Scrapy currently, but I’d like to move away from Python.
0