mischov
Meeseeks - A library for extracting data from HTML and XML with CSS or XPath selectors
import Meeseeks.CSS
html = HTTPoison.get!("https://news.ycombinator.com/").body
for story <- Meeseeks.all(html, css("tr.athing")) do
title = Meeseeks.one(story, css(".title a"))
%{title: Meeseeks.text(title),
url: Meeseeks.attr(title, "href")}
end
#=> [%{title: "...", url: "..."}, %{title: "...", url: "..."}, ...]
Meeseeks is a library for parsing and extracting data from HTML and XML with CSS or XPath selectors.
GitHub: GitHub - mischov/meeseeks: An Elixir library for parsing and extracting data from HTML and XML with CSS or XPath selectors. · GitHub
HexDocs: Meeseeks — Meeseeks v0.18.0
Features
- Friendly API
- Browser-grade HTML5 parser
- Permissive XML parser
- CSS and XPath selectors
- Rich, extensible selector architecture
- Helpers to extract data from selections
Why?
Meeseeks exists in the same space as an earlier library called Floki, so why was Meeseeks created and why would you use it instead of Floki?
Floki is a couple years older than Meeseeks, so why does Meeseeks even exist?
Meeseeks exists because Floki used to be unable to do what I needed.
When I started learning Elixir I reimplemented a small project I had written in another language. Part of that project involved extracting data from HTML, and unbeknownst to me some of the HTML I needed to extract data from was malformed.
This had never been a problem before because the HTML parser I was using in the other language was HTML5 spec compliant and handled the malformed HTML just as well as a browser. Unfortunately for me, Floki used (and still uses by default) the :mochiweb_html parser which is nowhere near HTML5 spec compliant, and just silently dropped the data I needed when parsing.
Meeseeks started out as an attempt to write an HTML5 spec compliant parser in Elixir (spoiler: it’s really hard), then switched to using Mozilla’s html5ever via Rustler after Hans wrote html5ever_elixir.
Floki gained optional support for using html5ever_elixir as its parser around the same time, but it still used :mochiweb_html (which doesn’t require Rust to be part of the build process) by default and I released Meeseeks as a safer alternative.
Why should I use Meeseeks instead of Floki?
When Meeseeks was released it came with a safer default HTML parser, a more complete collection of CSS selectors, and a more extensible selector architecture than Floki.
Since then Meeseeks has been further expanded with functionality Floki just doesn’t have, such as an XML parser and XPath selectors.
It won’t matter to most users, but the selection architecture is much richer than Floki’s, and permits the creation all kinds of interesting custom, stateful selectors (in fact, both the CSS and XPath selector strings compile down to the same selector structs that anybody can define).
What probably will matter more to users is the friendly API, extensive documentation, and the attention to the details of usability seen in such places as the custom formatting for result structs (#Meeseeks.Result<{ <p>1</p> }>) and the descriptive errors.
Is Floki ever a better choice than Meeseeks?
Yes, there are two main cases when Floki is clearly a better choice than Meeseeks.
Firstly, if you absolutely can’t include Rust in your build process AND you know that the HTML you’ll be working with is well-formed and won’t require an HTML5 spec compliant parser then using Floki with the :mochiweb_html parser is a reasonable choice.
However, if you have any doubts about the HTML you’ll be parsing you should probably figure out a way to use a better parser because using :mochiweb_html in that situation may be a timebomb.
Secondly, if you want to make updates to an HTML document Floki provides facilities to do so while Meeseeks, which is entirely focused on selecting and extracting data, does not.
How does performance compare between Floki and Meeseeks?
Performance is similar enough between the two that it’s probably not worth choosing one over the other for that reason.
For details and benchmarks, see Meeseeks vs. Floki Performance.
Trending in Announcing
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #elixirconf-us
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex










First 10 of 86 Posts!
Eiji
I don’t have Rust compiled, so I can’t test it yet, but I have some questions:
Meeseeks.Resultto (for example)Meeseeks.Document.Element?CSSselectors? For example parent!selector fromCSS 4 Selectors? Example:!div.parent > p.child. Or user selectors like:p:custom-selector, so it’s possible to add dynamically custom selectors for example from parameters (CSS selector) and plug-ins (custom handler dynamically loaded) combination.mischov
Thank you for your questions.
data-attributes.Edit: Brainfart,
!doesn’t mean “not,” it means “select me.” Also, forum has no strike-through?brightball
Just wanted to give you credit for a great name choice.
Eiji
I have two ideas. Simpler:
and version with casting:
So what we need is to create simple callbacks like:
mischov
I opened an issue on dataset, but I’m going to think about node a bit.
At the very least, your
Meeseeks.one_nodesuggestion would need to return{document, node}because otherwise there could be no guarantee that the caller would have theMeeseeks.Documentcapable of resolving the node ids contained in node. This is whyMeeseeks.Resultis how it is.mischov
Release v0.3.1
I added the discussed
datasetextractor and now raise a more helpful error when you try to select using a string instead of selectors.I’ve been dipping my toes into Rust and I should have an interesting (performance related) release soon.
OvermindDL1
Ooo, any details? Are you using Rustler to integrate to the VM or using a port?
mischov
Rustler. I’ve already been using html5ever_elixir which is hansihe’s Rustler NIF for html5ever but I’ve specialized things a bit for Meeseeks.
More details: Use meeseeks_html5ever instead of html5ever_elixir · Issue #2 · mischov/meeseeks · GitHub
mischov
Release v0.4.0
The largest change was the switch from
html5ever_elixirtomeeseeks_html5ever, which was a performance driven change (see the issue for details).Additionally, the
:not()CSS selector now supports lists of selectors.Meeseeks vs. Floki Performance
Since a lot of this release was focused around performance, I put together a benchmark comparing performance between Meeseeks and Floki for a couple real-world-ish scenarios. Benchmarking is tricky, but I’ve done my best to create something useful.
I go into a lot more detail on the benchmark, but in short the results are:
It’s performance benchmarking, though, so take that with a grain of skepticism.
mischov
Release v0.4.1
I recently ran across a bug in the CSS selector tokenizer that was breaking descendant combinators that were followed by a wildcard or pseudo-class, so I wanted to get a patch out for that before it confused somebody.
I also added CI.
Last Post!
mischov
Release v0.18.0
Updates
meeseeks_html5evertov0.15.0so that more modern versions of Elixir/Erlang are supported and fixes some compilation warnings.Thanks to @hkrutzer for the updates to
meeseeks_html5ever.