Improve regex performance or temporarely disable during bulk insertion?

Interesting. You could likely switch to use a change to validate the values, and then do it conditionally in bulk action calls instead of on each attribute?

Regexes could be translated to binary pattern matches at compile time, that would be significantly faster.

That’s true I’ve wanted a generic regex transpiration library for a while, thought of writing it myself. Something to take an elixir regex and transpire it to match spec, js, and PostgreSQL especially. (naturally allowing that not all regexes can be translated that way).

In unicode_string i have a function Unicode.Set.to_pattern/1 to transpile Unicode sets into binary patterns (or to :re-compatible regexs). Adapting that to transpile a subset of regex into pattern match wouldn’t be too much work so I’ll look at that.

Another thought that strikes me is that maybe, for Ash, something like the old COBOL PIC clause might be a good abstraction - it is declarative and it would map directly to a binary pattern match. The examples in this thread would be supported by such a construct. match could be reserved for regex, and pattern could be added for binary pattern matching.

Here I’m using the following symbols but they can anything:

• X - ASCII alphanumeric
• A - ASCII alphbetic
• D - ASCII digits
• Anything other character - a literal

I can envisage additional symbols for

• Unicode alphanumeric
• Unicode digit
• ASCII alphabetic
• Unicode alphabetic
• Sign matching (+ / -)
• Upper/lower case matching
• Quote mark matching (open then close)
• Unicode general category matching (generalised form of the above)
• Unicode script matching

A pattern could look like the following (using the examples in the thread) :`

# ~r/^[A-Za-z0-9]{8}$/
X(8)

# ~r/^\d{6}$/
D(6)

# ~r/^\d{2}\.\d{3}\.\d{3}\/\d{4}-\d{2}$/
DD.DDD.DDD/DDDD-DD

I think this is quite an elegant and declarative way to express many (but clearly not all!) data format expectations. And since they can be transpiled to binary pattern matches they would be a good fit for the BEAM. They can also abstract away some of the complexities of “what is a letter”, “what is a digit” and “what is whitespace” given Unicode has quite extensive character repertories for these categories and others.

Hmm…I much prefer something that doesn’t require the user learning a new pattern language. Its a neat idea though, and pulling from an existing standard is the only thing that really makes it tenable.

sick

So @zachdaniel I just tried your suggestion, basically I changed my attributes to this:

     attribute :cnpj_basico, :string do
       public? true
       allow_nil? false

       constraints min_length: 8, max_length: 8
     end

     attribute :cnpj_sufixo, :string do
       public? true
       allow_nil? false

       constraints min_length: 6, max_length: 6
     end

     attribute :cnpj_formatado, :string do
       public? true
       allow_nil? false

       constraints min_length: 18, max_length: 18
     end

And added these changes to my create action:

       change fn changeset, _ ->
        regex = ~r/^[A-Za-z0-9]{8}$/

        cnpj_basico = Ash.Changeset.get_attribute(changeset, :cnpj_basico)

        if not is_nil(cnpj_basico) and Regex.match?(regex, cnpj_basico) do
          changeset
        else
          exception = [value: cnpj_basico, field: :cnpj_basico, message: "must match the pattern %{regex}", vars: [regex: inspect(regex)]] |> Ash.Error.Changes.InvalidAttribute.exception()

          Ash.Changeset.add_error(changeset, exception)
        end
      end

      change fn changeset, _ ->
        regex = ~r/^\d{6}$/

        cnpj_sufixo = Ash.Changeset.get_attribute(changeset, :cnpj_sufixo)

        if not is_nil(cnpj_sufixo) and Regex.match?(regex, cnpj_sufixo) do
          changeset
        else
          exception = [value: cnpj_sufixo, field: :cnpj_sufixo, message: "must match the pattern %{regex}", vars: [regex: inspect(regex)]] |> Ash.Error.Changes.InvalidAttribute.exception()

          Ash.Changeset.add_error(changeset, exception)
        end
      end

      change fn changeset, _ ->
        regex = ~r/^\d{2}\.\d{3}\.\d{3}\/\d{4}-\d{2}$/

        cnpj_formatado = Ash.Changeset.get_attribute(changeset, :cnpj_formatado)

        if not is_nil(cnpj_formatado) and Regex.match?(regex, cnpj_formatado) do
          changeset
        else
          exception = [value: cnpj_formatado, field: :cnpj_formatado, message: "must match the pattern %{regex}", vars: [regex: inspect(regex)]] |> Ash.Error.Changes.InvalidAttribute.exception()

          Ash.Changeset.add_error(changeset, exception)
        end
      end

So, theorically, AFAIK, these two should be equal and should take the same time.

But, when I compared them both in my bulk loading, the times where very different, for example, here is the time with my changes:

process establishments: 0.235459
process establishments: 0.235015
process establishments: 0.256663
process establishments: 0.229017
process establishments: 0.241808
process establishments: 0.264744
process establishments: 0.264392
process establishments: 0.241751
process establishments: 0.236187
process establishments: 0.24704

Around 0.24s per batch

Now with the match constraint:

process establishments: 1.889976
process establishments: 1.869798
process establishments: 1.88271
process establishments: 1.924556
process establishments: 1.94613
process establishments: 2.154315
process establishments: 2.37499
process establishments: 2.378589
process establishments: 2.349797
process establishments: 2.36511

Around 2.1s per batch

Is this expected? I would expect that they would take the same amount of time. This makes me wonder if the match constraint is actually doing more work than necessary?

that is very surprising. I’m wondering if we may be applying the regex more than once for case where its an attribute constraint

Yep, that seems to be the case, I run both codes now with cprof and I can see some interesting changes:

With the changes, I get this number of calls for these functions:

  :re.run/3                                                            3
  :re.import/1                                                         3
  Regex.safe_run/3                                                     3
  Regex.match?/2                                                       3
  anonymous fn/4 in Regex.safe_run/3                                   3

Which makes sense, I have 3 changes and each are doing one Regex.match? call.

Now, removing the changes and having the match constraint, I get these numbers:

  :re.import/1                                                       495
  :re.run/3                                                            3
  Regex.safe_run/3                                                     3
  Regex.match?/2                                                       3
  anonymous fn/4 in Regex.safe_run/3                                   3

As you can see, all the calls are the same except for :re.import/1 which is called 495.

Going back to eprof this is the heaviest call:

:re.import/1                                                        495 33.19  525    1.06 

In other words, it seems that, for some reason, having the match constraint makes Ash call :re.import/1 hundreds of times per time during an Ash.Changeset.for_create call.

Can you do constraints: [match: {Spark.Regex, :cache, ["^\\d{8}$", ""]}] instead of constraints: [match: ~r/^\d{8}$/]?

You will need to change match: [ type: :regex ] to match: [ type: :regex_as_mfa ] in ‎lib/ash/type/string.ex.

It was reverted in this commit.

https://github.com/ash-project/ash/commit/d77537dd03f7e31dc7a127a415a0a935be15043c#diff-1d18069e5b6963e5934ed6ccc230df9a8095f4a3785d93cc0dc328e1a419b83f

https://github.com/ash-project/spark/blob/b985057765789cbae5c45321fbae8f10c263d6dd/lib/spark/options/options.ex#L194

Ah, that explains it, I did saw some code in the regex match part that allowed calling a function with the mfa, but when I tried myself it would break somewhere before because the support was removed.