kerryb

kerryb

Is there a known input length limitation with the Regex module (or I assume probably :re under the covers)?

I just spent a while debugging some code that generally worked, but was failing to match for certain inputs which seemed valid. Eventually I narrowed it down to the sheer size of the strings where it failed.

Here’s a simplified example. This works:

text = "START\n" <> Enum.map_join(1..100_000, "\n", &"Line #{&1}") <> "\nEND\n"
Regex.run(~r/START(.*?)END/ms, text)
# => ["START\nLine 1\nLine 2\nLine 3\nLine 4\nLine 5\n\n" <> ...]

But if I increase the number of lines by another order of magnitude, the same pattern stops matching:

text = "START\n" <> Enum.map_join(1..1_000_000, "\n", &"Line #{&1}") <> "\nEND\n"
Regex.run(~r/START(.*?)END/ms, text)
# => nil

No doubt this is a terrible regex, and I can imagine it having performance implications, but I was a bit surprised that it silently failed, indicating no matches, rather than raising an exception.

Showing Posts 1 to 2

hauleth

hauleth

Assuming OTP 28 there are flags for re:run/3. Flags that are interesting for you are:

  • :report_errors
  • :match_limit, quoting docs:

    The default value 10,000,000 is compiled into the Erlang VM.

  • :match_limit_recursion

Also, in this particular case using regex is IMHO pointless as using binary pattern matching would be faster and cleaner IMHO.

def extract("START\n" <> input), do: do_extract(input, <<>>)
def extract(other), do: {:error, :no_start}

defp do_extract("\nEND\n", data), do: {:ok, data}
defp do_extract("", _), do: {:error, :no_end}
defp do_extract(<<c>> <> rest, acc), do: do_extract(rest, acc <> <<c>>)
kerryb

kerryb OP

Assuming OTP 28 there are flags for re:run/3. Flags that are interesting for you are:

  • :report_errors
  • :match_limit, quoting docs:

The default value 10,000,000 is compiled into the Erlang VM.

Thanks!

Also, in this particular case using regex is IMHO pointless as using binary pattern matching would be faster and cleaner IMHO.

Yeah, it was a simplified example just to demonstrate the issue I was having. But thanks for the pattern matching example anyway.

— All posts loaded —

Where Next? Top

Trending in Questions Top

stjefim
Hello! Suppose you are building workflow (order / task / payment) processing system with the following requirements: Each workflow con...
New
Blokh
Hey guys, I’ve got a huge CSV ( around 10 GB ) that needs to be processed hourly Do you guys have any suggestions what is the best prac...
New
kszambelanczyk
Hello! Could someone please give me a help/sample code, how to delete a file from s3 using waffle/waffle_ecto from Phoenix app. I creat...
New
Onor.io
I have what I’ve heard referred to as a “lookup table” in my database. This is a way of assigning codes to common values. One common lo...
New
jaybe78
Hello, I’m developing a online persistent chat system (what’s app) like using elixir/dynamodb/aws for a mobile app(flutter). The diffic...
New
Trolleger
What approach to take when sending live updates to “random” users Hi! I have a question, I have a little chat app, and when I create a DM...
New
matt-savvy
Anyone here using Honeybadger? My Honeybadger account is being overwhelmed with noise from some bots. Seeing a lot of Bandit.HTTPError...
New

Other Trending Topics Top

garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge &amp; Solve. They are GUI (Emerge) and State management (S...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
wintermeyer
There are three potential reasons for members of this forum to have a look at https://vutuv.de You are tired or annoyed of LinkedIn. Yo...
New
webofbits
Aludel - LLM Evaluation Workbench Aludel is an embeddable Phoenix LiveView dashboard for evaluating and comparing LLM prompts across mult...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews