I’m not sure if I understand correctly. In your real world example the grouping is already done, because <b> are children of the <p>.

But your number example seems to be another problem (chunk by 1s and all 2s that follow a 1)

Slow load (possibly timeout). I know what you feel. Hope those are not ASP.net pages with invalid HTML code.

Anyway, here is my solution:

Mix.install([:floki])

defmodule Example do
  def sample(list, acc \\ [])

  # for empty input after parsing
  def sample([], []), do: []

  # when all p elements are passed
  # reverse last section texts and wrap them into list
  # as otherwise a resulting list would be added
  # to a main list where each element contains a list of sections texts
  def sample([], acc), do: [:lists.reverse(acc)]

  # in case of first bold text simply add text to
  # as the only element in new acc
  # and call function recursively
  def sample([{"p", _, [{"b", [], [text]}]} | tail], []) when is_binary(text) do
    sample(tail, [text])
  end

  # however if there is some data in acc
  # reverse its contents and return it as a list of
  # last section texts and recursive call
  def sample([{"p", _, [{"b", [], [text]}]} | tail], acc) when is_binary(text) do
    [:lists.reverse(acc) | sample(tail, [text])]
  end

  # when we got a normal text simply add it to acc
  # and call function recursively
  def sample([{"p", _, [text]} | tail], acc) when is_binary(text) do
    sample(tail, [text | acc])
  end
end

"""
<p><b>Section 1, name, and text</b></p>
<p>Section 1 more text</p>
<p>Section 1 more text</p>
<p><b>Section 2, name, and text</b></p>
<p>Section 2 more text</p>
<p><b>Section 3,  name, and text</b></p>
"""
|> Floki.parse_fragment!()
|> Example.sample()
|> dbg()

Pattern matching is a fastest solution. You can take a look at this post to see possible alternative solutions.

Sorry, yeah - I left it abstract. I used the 1’s to represent paragraphs with a <b>, and 2’s for <p>'s without.

Here’s a screenshot of the page.

Screen Shot 2022-09-25 at 5.24.05 AM1112×540 68.9 KB

And I’m trying to produce a list of %Section{}.

The actual HTML looks like this. All of the <p>'s just run on; there’s no hierarchy. The only clue is that the first line of a Section has a <b>.

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><b><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      838.025
Election laws apply.</span></b><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>
(1) ORS chapter 255 governs the following:</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (a)
The nomination and election of district board members.</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (b)
The conduct of district elections.</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (2)
The electors of a district may exercise the powers of the initiative and
referendum regarding a district measure, in accordance with ORS 255.135 to
255.205. [Formerly 494.043]</span></p>

Thanks! That’s pretty interesting. This meta tag is pretty chilling.

<meta name=Generator content="Microsoft Word 15 (filtered)">

It seems they’re somehow scripting MS Word to “Save as HTML”.

sounds like fun … not

Since you did not give a struct definition I wrote it myself:

Mix.install([:floki])

defmodule Section do
  defstruct ~w[contents id title]a

  def add_text(%__MODULE__{contents: contents} = section, {"span", _, [text]})
      when is_binary(text) do
    %{section | contents: [String.trim(text) | contents]}
  end

  def first_paragraph({"b", _, [{"span", _, [id_title]}]}, {"span", _, [text]})
      when is_binary(id_title) and is_binary(text) do
    [id, title] = id_title |> String.trim() |> String.split("\n", parts: 2)
    %Section{contents: [String.trim(text)], id: id, title: title}
  end

  def reverse_contents(%__MODULE__{contents: contents} = section) do
    %{section | contents: :lists.reverse(contents)}
  end
end

defmodule Example do
  def sample(list, section \\ nil)

  def sample([], []), do: []

  def sample([], section), do: [Section.reverse_contents(section)]

  def sample([{"p", _, [title, text]} | tail], nil) do
    sample(tail, Section.first_paragraph(title, text))
  end

  def sample([{"p", _, [title, text]} | tail], section) do
    new_section = Section.first_paragraph(title, text)
    [Section.reverse_contents(section) | sample(tail, new_section)]
  end

  def sample([{"p", _, [text]} | tail], section) do
    sample(tail, Section.add_text(section, text))
  end
end

"""
<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><b><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      838.025
Election laws apply.</span></b><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>
(1) ORS chapter 255 governs the following:</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (a)
The nomination and election of district board members.</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (b)
The conduct of district elections.</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (2)
The electors of a district may exercise the powers of the initiative and
referendum regarding a district measure, in accordance with ORS 255.135 to
255.205. [Formerly 494.043]</span></p>
"""
|> Floki.parse_fragment!()
|> Example.sample()
|> dbg()

What do you think about it?

This is not too bad; at least all the <p> have closing </p>. You know they don’t have to; and I;ve seen html that freely mix the 2 styles, with or without closing </p>

That’s very cool. Thanks for taking a whack at it. The <span>'s and other attributes aren’t important, though, because they’re the same on every element. (!) The big picture is, we want mostly plain text — simplified HTML. This code’s purpose is to produce well formed JSON with all the important info from the original texts. I publish the JSON to a datasets public repo.

You can see how I solved it: The actual Section:

defmodule Crawlers.ORS.Models.Section do
  @moduledoc """
  An ORS Section.
  """
  use TypedStruct

  typedstruct enforce: true do
    @typedoc "An ORS Section"

    field :kind, String.t(), default: "section"
    field :name, String.t()
    field :number, String.t()
    field :text, String.t()
    field :chapter_number, String.t()
  end
end

The group_with/2 function:

  @doc """
  Group a list of elements into sub-lists, where each sub-list is
  led by an element that satisfies the predicate. It skips initial
  elements that do not satisfy the predicate.

  iex> group_with([1, 2, 3, 1, 4], &(&1 == 1))
  [[1, 2, 3], [1, 4]]

  iex> group_with(["a", "b", "x", "c", "d"], &(&1 == "x"))
  [["x", "c", "d"]]
  """
  def group_with(list, predicate) do
    result_reversed =
      reduce(list, [], fn e, acc ->
        case predicate.(e) do
          true ->
            # Start a new group.
            [[e]] ++ acc

          false ->
            # Try to extract the current group.
            {curr, rest} = List.pop_at(acc, 0)

            case curr do
              # Skip until predicate is true.
              nil -> acc
              # Append to current group.
              group -> [group ++ [e] | rest]
            end
        end
      end)

    reverse(result_reversed)
  end

• The sections feature branch
• Pull Request

I’ll just pray that Floki can handle it. I’m sure I’ll be dealing with that at some point.