mischov

mischov

Meeseeks - A library for extracting data from HTML and XML with CSS or XPath selectors

import Meeseeks.CSS

html = HTTPoison.get!("https://news.ycombinator.com/").body

for story <- Meeseeks.all(html, css("tr.athing")) do
  title = Meeseeks.one(story, css(".title a"))
  %{title: Meeseeks.text(title),
    url: Meeseeks.attr(title, "href")}
end
#=> [%{title: "...", url: "..."}, %{title: "...", url: "..."}, ...]

Meeseeks is a library for parsing and extracting data from HTML and XML with CSS or XPath selectors.

GitHub: GitHub - mischov/meeseeks: An Elixir library for parsing and extracting data from HTML and XML with CSS or XPath selectors. · GitHub
HexDocs: Meeseeks — Meeseeks v0.18.0

Features

  • Friendly API
  • Browser-grade HTML5 parser
  • Permissive XML parser
  • CSS and XPath selectors
  • Rich, extensible selector architecture
  • Helpers to extract data from selections

Why?

Meeseeks exists in the same space as an earlier library called Floki, so why was Meeseeks created and why would you use it instead of Floki?

Floki is a couple years older than Meeseeks, so why does Meeseeks even exist?

Meeseeks exists because Floki used to be unable to do what I needed.

When I started learning Elixir I reimplemented a small project I had written in another language. Part of that project involved extracting data from HTML, and unbeknownst to me some of the HTML I needed to extract data from was malformed.

This had never been a problem before because the HTML parser I was using in the other language was HTML5 spec compliant and handled the malformed HTML just as well as a browser. Unfortunately for me, Floki used (and still uses by default) the :mochiweb_html parser which is nowhere near HTML5 spec compliant, and just silently dropped the data I needed when parsing.

Meeseeks started out as an attempt to write an HTML5 spec compliant parser in Elixir (spoiler: it’s really hard), then switched to using Mozilla’s html5ever via Rustler after Hans wrote html5ever_elixir.

Floki gained optional support for using html5ever_elixir as its parser around the same time, but it still used :mochiweb_html (which doesn’t require Rust to be part of the build process) by default and I released Meeseeks as a safer alternative.

Why should I use Meeseeks instead of Floki?

When Meeseeks was released it came with a safer default HTML parser, a more complete collection of CSS selectors, and a more extensible selector architecture than Floki.

Since then Meeseeks has been further expanded with functionality Floki just doesn’t have, such as an XML parser and XPath selectors.

It won’t matter to most users, but the selection architecture is much richer than Floki’s, and permits the creation all kinds of interesting custom, stateful selectors (in fact, both the CSS and XPath selector strings compile down to the same selector structs that anybody can define).

What probably will matter more to users is the friendly API, extensive documentation, and the attention to the details of usability seen in such places as the custom formatting for result structs (#Meeseeks.Result<{ <p>1</p> }>) and the descriptive errors.

Is Floki ever a better choice than Meeseeks?

Yes, there are two main cases when Floki is clearly a better choice than Meeseeks.

Firstly, if you absolutely can’t include Rust in your build process AND you know that the HTML you’ll be working with is well-formed and won’t require an HTML5 spec compliant parser then using Floki with the :mochiweb_html parser is a reasonable choice.

However, if you have any doubts about the HTML you’ll be parsing you should probably figure out a way to use a better parser because using :mochiweb_html in that situation may be a timebomb.

Secondly, if you want to make updates to an HTML document Floki provides facilities to do so while Meeseeks, which is entirely focused on selecting and extracting data, does not.

How does performance compare between Floki and Meeseeks?

Performance is similar enough between the two that it’s probably not worth choosing one over the other for that reason.

For details and benchmarks, see Meeseeks vs. Floki Performance.

First 10 of 86 Posts! Switch mode

Eiji

Eiji

I don’t have Rust compiled, so I can’t test it yet, but I have some questions:

  1. Do you want to support dataset JavaScript API? It could be useful to fetch some data.
  2. You are using some structs - they are good for pattern matching, so how I could change Meeseeks.Result to (for example) Meeseeks.Document.Element?
  3. Do you want to support custom CSS selectors? For example parent ! selector from CSS 4 Selectors? Example: !div.parent > p.child. Or user selectors like: p:custom-selector, so it’s possible to add dynamically custom selectors for example from parameters (CSS selector) and plug-ins (custom handler dynamically loaded) combination.
mischov

mischov

Thank you for your questions.

  1. I haven’t looked into supporting the dataset API, but I can imagine making a helper function to convert a node or result into a map whose keys and values would come from data- attributes.
Do you think that would be enough?
  1. I haven’t provided a helper function to go directly from a result to a node, and maybe I should. Currently you would need to do:
`Meeseeks.Document.get_node(result.document, result.id)` .
  1. I am currently only targeting CSS3 selectors, but that might change in the future.
I am undecided on whether I want to support custom selectors in the CSS selector macro I provide, but I can definitely see the utility of allowing users to hook their own custom selectors into the CSS selector syntax.

I have, however, done my best to provide support for custom Meeseeks selectors, and one could without much difficulty adapt the code I use in my `css` macro to make a custom `css` macro, which would be as easy to use as:

```elixir
iex> import Your.CSS # instead of Meeseeks.CSS
...
iex> Meeseeks.all(source, css("!div.parent > p.child"))
...
```

Edit: Brainfart, ! doesn’t mean “not,” it means “select me.” Also, forum has no strike-through?

brightball

brightball

Just wanted to give you credit for a great name choice.

Eiji

Eiji

I have two ideas. Simpler:

iex> import Meeseeks.CSS
Meeseeks.CSS
iex> html = Tesla.get("https://news.ycombinator.com/").body
"..."
iex> for story <- Meeseeks.all(html, css("tr.athing")) do
       story
       |> Meeseeks.one(css(".title a"))
       |> Meeseeks.dataset
       |> Map.fetch!("id") # data-id attribute
       |> String.to_integer
     end
[1, 2, 4, 9, 13]

and version with casting:

iex> import Meeseeks.CSS
Meeseeks.CSS
iex> html = Tesla.get("https://news.ycombinator.com/").body
"..."
iex> for story <- Meeseeks.all(html, css("tr.athing")) do
       story
       |> Meeseeks.one(css(".title a"))
       |> Meeseeks.dataset(cast: :auto)
       |> Map.fetch!("id") # data-id attribute
     end
# or:
iex> for story <- Meeseeks.all(html, css("tr.athing")) do
       story
       |> Meeseeks.one(css(".title a"))
       |> Meeseeks.dataset(cast: %{"id" => :integer}) # cast only data-id attribute
       |> Map.fetch!("id") # data-id attribute
     end
[1, 2, 4, 9, 13]

So what we need is to create simple callbacks like:

result = Meeseeks.one(story, css(".title a"))
first_node_result = Meeseeks.Document.get_node(result.document, result.id) .
second_node_result = Meeseeks.one_node(story, css(".title a"))
assert first_node_result == second_node_result
mischov

mischov

I opened an issue on dataset, but I’m going to think about node a bit.

At the very least, your Meeseeks.one_node suggestion would need to return {document, node} because otherwise there could be no guarantee that the caller would have the Meeseeks.Document capable of resolving the node ids contained in node. This is why Meeseeks.Result is how it is.

mischov

mischov

Release v0.3.1

I added the discussed dataset extractor and now raise a more helpful error when you try to select using a string instead of selectors.

I’ve been dipping my toes into Rust and I should have an interesting (performance related) release soon.

OvermindDL1

OvermindDL1

Ooo, any details? Are you using Rustler to integrate to the VM or using a port? :slight_smile:

mischov

mischov

Rustler. I’ve already been using html5ever_elixir which is hansihe’s Rustler NIF for html5ever but I’ve specialized things a bit for Meeseeks.

More details: Use meeseeks_html5ever instead of html5ever_elixir · Issue #2 · mischov/meeseeks · GitHub

mischov

mischov

Release v0.4.0

The largest change was the switch from html5ever_elixir to meeseeks_html5ever, which was a performance driven change (see the issue for details).

Additionally, the :not() CSS selector now supports lists of selectors.

Meeseeks vs. Floki Performance

Since a lot of this release was focused around performance, I put together a benchmark comparing performance between Meeseeks and Floki for a couple real-world-ish scenarios. Benchmarking is tricky, but I’ve done my best to create something useful.

I go into a lot more detail on the benchmark, but in short the results are:

  • In the “Wiki Links” benchmark, Meeseeks and Floki perform similarly
  • In the “Trending JS” benchmark, Floki is about 1.4x slower than Meeseeks

It’s performance benchmarking, though, so take that with a grain of skepticism.

mischov

mischov

Release v0.4.1

I recently ran across a bug in the CSS selector tokenizer that was breaking descendant combinators that were followed by a wildcard or pseudo-class, so I wanted to get a patch out for that before it confused somebody.

I also added CI.

Last Post!

mischov

mischov

Release v0.18.0

Updates meeseeks_html5ever to v0.15.0 so that more modern versions of Elixir/Erlang are supported and fixes some compilation warnings.

Thanks to @hkrutzer for the updates to meeseeks_html5ever.

Where Next?

Trending in Announcing Top

bluzky
You may know https://ui.shadcn.com/, a UI component library for React. I really love it’s design style and components. I’ve built some co...
385 14863 120
New
JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
shahryarjb
The Chelekom project is a library of Phoenix and LiveView components generated via Mix tasks to fit developer needs seamlessly. One of i...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge &amp; Solve. They are GUI (Emerge) and State management (S...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
zachdaniel
Introducing AshStorage! Attachment and file management that slots directly into your resources :smiling_face_with_sunglasses: I had hope...
New

Other Trending Topics Top

type1fool
I just stumbled on a newly redesigned elixir-lang.org. :tada: It looks like @Software_Mansion did the work, and I think it is generally a...
New
akoutmos
@hugobarauna and I (Alex Koutmos) have been hard at work on writing a book on Nerves that takes you from simply blinking LEDs to building...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
bjorng
We want to introduce a new native datatype to Erlang: native records. Although replacing all tuple records with native records is not our...
New
spammy
I’m looking to build a personal workflow to quickly deploy web applications written in elixir/phoenix, for local consumption (ie not on t...
New
yureehuh
Introduction Founded in 2017 by landscape ecologist and fire mitigation expert Harry Statter, Frontline developed the first fully integra...
New

We're in Beta

About us Mission Statement