neuone

neuone

Floki parse and group by heading

The HTML has no attributes it’s very vanilla and plain

<!doctype html>
<html>
<body>
  <section class="body-copy">
  <h2>Topic 1</h2>
  <p>data-a</p>
  <p>data-b</p>
  <p>data-c</p>
  <h2>Topic 2</h2>
  <p>data-d</p>
  <p>data-e</p>
  <p>data-f</p>
  <h2>Topic 3</h2>
  <p>data-g</p>
  <p>data-h</p>
  <p>data-i</p>
  </section>
</body>
</html>

I’m using Floki and I’m trying to parse it so I can create a List of maps like so.

%{ topic: "Topic 1", data: "data-a" }
%{ topic: "Topic 1", data: "data-b" }
%{ topic: "Topic 1", data: "data-c" }
%{ topic: "Topic 2", data: "data-d" }
%{ topic: "Topic 2", data: "data-e" }
%{ topic: "Topic 2", data: "data-f" }

I’m struggling to get all the P tags under each H2 with Floki.

# Try loading this html
path = "/Users/Foo/Desktop/test.html"
{_, local_file } = File.read(path)

# This will return me all the h2
Floki.find(local_file, "h2")               
[{"h2", [], ["Topic 1"]}, {"h2", [], ["Topic 2"]}, {"h2", [], ["Topic 3"]}]

# This will return me the first p from a specific h2. But not all of them
Floki.find(local_file, "h2:nth-of-type(1) + p")
[{"p", [], ["data-a"]}]

# This return this first p for Topic 2 but I need 2 more p tags (data-e, data-f)
Floki.find(local_file, "h2:nth-of-type(2) + p")
[{"p", [], ["data-d"]}]

# This return this first p for Topic 3 but I need 2 more p tags (data-h, data-i)
Floki.find(local_file, "h2:nth-of-type(3) + p")
[{"p", [], ["data-g"]}]

Question
I cannot figure out how to get ONLY the P tags for each H2.

Marked As Solved

mischov

mischov

This is brittle as all get-out because it makes some serious assumptions about the shape of the data, but:

def extract_topic_maps(html) do
  nodes = Floki.find(html, "section.body-copy > *")

  {_, topic_maps} =
    Enum.reduce(nodes, {nil, []}, fn
      {"p", _, _}, {nil, _} -> raise "Invalid state: no topic"
      # New topic, set in accumulator
      {"h2", _, [topic]}, {_, topic_maps} -> {topic, topic_maps}
      # New value in topic, add appropriate topic_map to topic_maps in accumulator
      {"p", _, [data]}, {topic, topic_maps} -> {topic, [%{topic: topic, data: data} | topic_maps]}
      node, _ -> raise "Invalid state: unexpected node #{inspect(node)}"
    end)

  Enum.reverse(topic_maps)
end

With your input that returns something like

[
  %{data: "data-a", topic: "Topic 1"},
  %{data: "data-b", topic: "Topic 1"}, 
  %{data: "data-c", topic: "Topic 1"},
  %{data: "data-d", topic: "Topic 2"},
  ...
]

Also Liked

mischov

mischov

The problem is that the data is not actually grouped by anything other intent and order, which CSS selectors don’t do well with.

If each item looked like <div><h2>Header</h2><p>Item 1</p><p>Item 2</p></div> then trying to select with CSS selectors would be a lot easier because that’s the type of data those selectors work best with.

Here are a couple ways you could solve the problem (based on the example data):

  1. If there are an identical number of p tags after each h2 tag, you could get a list of all the children of the <section> and do a Enum.chunk_by to group the nodes that belong together and go from there.

  2. If there aren’t an identical number of p tags after each h2 tag or there are other unwanted children, you could get a list of all the children of the <section> and write a reducer that builds up a map of h2 element to a list of the sections that occur before the next h2, and go from there.

There might be better solutions I’m not thinking of, but those are places where you could start.

idi527

idi527

Maybe

def grouped_by_topics(body_nodes) do
  # nil here is a hack, you can run this function in two phases instead
  # 1. find the first `h1` with a topic
  # 2. then start this function with the rest of `body_nodes`
  grouped_by_topics(body_nodes, nil, [], [])
end

# when we meet a `h1` tag, start a new `inner_acc` for collecting the data for the topic in `h1`
defp grouped_by_topics([{"h1", [], ["Topic" <> _ = next_topic]} | rest], prev_topic, prev_inner_acc, outer_acc) do
  grouped_by_topics(rest, next_topic, [], [%{prev_topic => prev_inner_acc} | outer_acc])
end 

# when we meet a new `p` tag, add it to the `inner_acc` for the current topic
defp grouped_by_topics([{"p", [], ["data" <> _ = new_data]} | rest], current_topic, inner_acc, outer_acc) do
  grouped_by_topics(rest, current_topic, [new_data | inner_acc], outer_acc)
end

# neither a `p` nor an `h1` tag -- skip
defp grouped_by_topics([_other | rest], current_topic, inner_acc, outer_acc) do
  grouped_by_topics(rest, current_topic, inner_acc, outer_acc)
end

# no more html nodes -- finish
defp grouped_by_topics([], last_topic, inner_acc, outer_acc) do
  [%{last_topic => inner_acc} | outer_acc]
end

inner_acc is for collecting data-* in p tags
outer_acc is for collecting %{topic => data (aka final_inner_acc)} maps
current_topic is for keeping the topic from the last h2 tag

mischov

mischov

Simply, reducers are functions that take 1) the current item from the collection being reduced over, and 2) the current accumulator, and that return a value that will be used as the accumulator in the next call of the reducer (for the next item in the collection being reduced over).

For example, fn n, sum -> sum + n end is a reducer that take a number, and adds it to the running sum. It could be used like

iex> Enum.reduce([1, 2, 3], 0, fn n, sum -> sum + n end)
6

In the solution, my accumulator takes the shape of a tuple that hold the current topic (or nil) as the first element, and a list of topic maps as the second element.

The reducer has four clauses:

  1. If it encounters a <p> element before it has a topic, it raises.
  2. If it encounters a <h2> element, it assumes that that element has one child, which will be the topic, and sets the topic in the accumulator to that topic.
  3. If it encounters a <p> element and has a topic, it creates a topic map with the data from the <p> element (again assuming the element only has one child) and the topic, and adds that topic map to the topic maps in the accumulator.
  4. If it encounters any other shape of node (whether it’s a <span>, or a <p> element with no or multiple children, or anything else), it raises.

If you have any other questions about how some part of that works feel free to ask.

Where Next?

Popular in Questions Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement