Meeseeks - A library for extracting data from HTML and XML with CSS or XPath selectors

I don’t have Rust compiled, so I can’t test it yet, but I have some questions:

1. Do you want to support dataset JavaScript API? It could be useful to fetch some data.
2. You are using some structs - they are good for pattern matching, so how I could change Meeseeks.Result to (for example) Meeseeks.Document.Element?
3. Do you want to support custom CSS selectors? For example parent ! selector from CSS 4 Selectors? Example: !div.parent > p.child. Or user selectors like: p:custom-selector, so it’s possible to add dynamically custom selectors for example from parameters (CSS selector) and plug-ins (custom handler dynamically loaded) combination.

Thank you for your questions.

1. I haven’t looked into supporting the dataset API, but I can imagine making a helper function to convert a node or result into a map whose keys and values would come from data- attributes.

Do you think that would be enough?

2. I haven’t provided a helper function to go directly from a result to a node, and maybe I should. Currently you would need to do:

`Meeseeks.Document.get_node(result.document, result.id)` .

3. I am currently only targeting CSS3 selectors, but that might change in the future.

I am undecided on whether I want to support custom selectors in the CSS selector macro I provide, but I can definitely see the utility of allowing users to hook their own custom selectors into the CSS selector syntax.

I have, however, done my best to provide support for custom Meeseeks selectors, and one could without much difficulty adapt the code I use in my `css` macro to make a custom `css` macro, which would be as easy to use as:

```elixir
iex> import Your.CSS # instead of Meeseeks.CSS
...
iex> Meeseeks.all(source, css("!div.parent > p.child"))
...
```

Edit: Brainfart, ! doesn’t mean “not,” it means “select me.” Also, forum has no strike-through?

Just wanted to give you credit for a great name choice.

I have two ideas. Simpler:

iex> import Meeseeks.CSS
Meeseeks.CSS
iex> html = Tesla.get("https://news.ycombinator.com/").body
"..."
iex> for story <- Meeseeks.all(html, css("tr.athing")) do
       story
       |> Meeseeks.one(css(".title a"))
       |> Meeseeks.dataset
       |> Map.fetch!("id") # data-id attribute
       |> String.to_integer
     end
[1, 2, 4, 9, 13]

and version with casting:

iex> import Meeseeks.CSS
Meeseeks.CSS
iex> html = Tesla.get("https://news.ycombinator.com/").body
"..."
iex> for story <- Meeseeks.all(html, css("tr.athing")) do
       story
       |> Meeseeks.one(css(".title a"))
       |> Meeseeks.dataset(cast: :auto)
       |> Map.fetch!("id") # data-id attribute
     end
# or:
iex> for story <- Meeseeks.all(html, css("tr.athing")) do
       story
       |> Meeseeks.one(css(".title a"))
       |> Meeseeks.dataset(cast: %{"id" => :integer}) # cast only data-id attribute
       |> Map.fetch!("id") # data-id attribute
     end
[1, 2, 4, 9, 13]

So what we need is to create simple callbacks like:

result = Meeseeks.one(story, css(".title a"))
first_node_result = Meeseeks.Document.get_node(result.document, result.id) .
second_node_result = Meeseeks.one_node(story, css(".title a"))
assert first_node_result == second_node_result

I opened an issue on dataset, but I’m going to think about node a bit.

At the very least, your Meeseeks.one_node suggestion would need to return {document, node} because otherwise there could be no guarantee that the caller would have the Meeseeks.Document capable of resolving the node ids contained in node. This is why Meeseeks.Result is how it is.

Release v0.3.1

I added the discussed dataset extractor and now raise a more helpful error when you try to select using a string instead of selectors.

I’ve been dipping my toes into Rust and I should have an interesting (performance related) release soon.

Ooo, any details? Are you using Rustler to integrate to the VM or using a port?

Rustler. I’ve already been using html5ever_elixir which is hansihe’s Rustler NIF for html5ever but I’ve specialized things a bit for Meeseeks.

More details: Use meeseeks_html5ever instead of html5ever_elixir · Issue #2 · mischov/meeseeks · GitHub

Release v0.4.0

The largest change was the switch from html5ever_elixir to meeseeks_html5ever, which was a performance driven change (see the issue for details).

Additionally, the :not() CSS selector now supports lists of selectors.

Meeseeks vs. Floki Performance

Since a lot of this release was focused around performance, I put together a benchmark comparing performance between Meeseeks and Floki for a couple real-world-ish scenarios. Benchmarking is tricky, but I’ve done my best to create something useful.

I go into a lot more detail on the benchmark, but in short the results are:

• In the “Wiki Links” benchmark, Meeseeks and Floki perform similarly
• In the “Trending JS” benchmark, Floki is about 1.4x slower than Meeseeks

It’s performance benchmarking, though, so take that with a grain of skepticism.

Release v0.4.1

I recently ran across a bug in the CSS selector tokenizer that was breaking descendant combinators that were followed by a wildcard or pseudo-class, so I wanted to get a patch out for that before it confused somebody.

I also added CI.