kerryb

kerryb

String length limitation for regex matching?

Is there a known input length limitation with the Regex module (or I assume probably :re under the covers)?

I just spent a while debugging some code that generally worked, but was failing to match for certain inputs which seemed valid. Eventually I narrowed it down to the sheer size of the strings where it failed.

Here’s a simplified example. This works:

text = "START\n" <> Enum.map_join(1..100_000, "\n", &"Line #{&1}") <> "\nEND\n"
Regex.run(~r/START(.*?)END/ms, text)
# => ["START\nLine 1\nLine 2\nLine 3\nLine 4\nLine 5\n\n" <> ...]

But if I increase the number of lines by another order of magnitude, the same pattern stops matching:

text = "START\n" <> Enum.map_join(1..1_000_000, "\n", &"Line #{&1}") <> "\nEND\n"
Regex.run(~r/START(.*?)END/ms, text)
# => nil

No doubt this is a terrible regex, and I can imagine it having performance implications, but I was a bit surprised that it silently failed, indicating no matches, rather than raising an exception.

Most Liked

hauleth

hauleth

Assuming OTP 28 there are flags for re:run/3. Flags that are interesting for you are:

  • :report_errors
  • :match_limit, quoting docs:

    The default value 10,000,000 is compiled into the Erlang VM.

  • :match_limit_recursion

Also, in this particular case using regex is IMHO pointless as using binary pattern matching would be faster and cleaner IMHO.

def extract("START\n" <> input), do: do_extract(input, <<>>)
def extract(other), do: {:error, :no_start}

defp do_extract("\nEND\n", data), do: {:ok, data}
defp do_extract("", _), do: {:error, :no_end}
defp do_extract(<<c>> <> rest, acc), do: do_extract(rest, acc <> <<c>>)

Last Post!

kerryb

kerryb

Assuming OTP 28 there are flags for re:run/3. Flags that are interesting for you are:

  • :report_errors
  • :match_limit, quoting docs:

The default value 10,000,000 is compiled into the Erlang VM.

Thanks!

Also, in this particular case using regex is IMHO pointless as using binary pattern matching would be faster and cleaner IMHO.

Yeah, it was a simplified example just to demonstrate the issue I was having. But thanks for the pattern matching example anyway.

Where Next?

Popular in Questions Top

RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" =&gt; #BSON.ObjectId&lt;58eb1a7a9ad169198c3dXXXX&gt;, "email" =&gt; ...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

Other popular topics Top

minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 55125 245
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New

We're in Beta

About us Mission Statement