sezaru

Improve regex performance or temporarely disable during bulk insertion?

I have a resource that has the following fields with regexes:

    attribute :cnpj_basico, :string do
      public? true
      allow_nil? false

      constraints min_length: 8, max_length: 8, match: ~r/^[A-Za-z0-9]{8}$/
    end

    attribute :cnpj_sufixo, :string do
      public? true
      allow_nil? false

      constraints min_length: 6, max_length: 6, match: ~r/^\d{6}$/
    end

    attribute :cnpj_formatado, :string do
      public? true
      allow_nil? false

      constraints min_length: 18, max_length: 18, match: ~r/^\d{2}\.\d{3}\.\d{3}\/\d{4}-\d{2}$/
    end

I need to bulk insert millions of rows into the database using this resource.

I noticed that casting the resource would have a major slowdown if the match constraints are in place.

Here are the most expensive calls with match enabled using eprof:

#                                                                                  CALLS     % TIME µS/CALL
Total                                                                               3676 100.0 2155    0.59
...
:crypto.strong_rand_bytes_nif/1                                                        1  0.79   17   17.00
Ecto.Type.cast_fun/1                                                                  13  0.84   18    1.38
Ash.Changeset.force_change_attribute/3                                                10  1.07   23    2.30
Enum."-map/2-lists^map/1-1-"/2                                                        80  1.16   25    0.31
:lists.member/2                                                                       47  1.16   25    0.53
:lists.keyfind/3                                                                      40  1.16   25    0.63
:ets.match_object/2                                                                    4  1.30   28    7.00
:re.import/1                                                                         495 24.41  526    1.06
Core.Cnpj.Estabelecimento.persisted/0                                                 41 38.61  832   20.29

And here are the most expensive ones if I remove the match constraint

#                                                                                  CALLS    % TIME µS/CALL
Total                                                                               3160 100.  592    0.19
Spark.Dsl.Extension.persisted!/3                                                      43 1.69   10    0.23
Ash.Changeset.do_change_attribute/4                                                    7 1.86   11    1.57
:erlang.module_loaded/1                                                               30 1.86   11    0.37
anonymous fn/1 in Ash.Changeset.expand_upsert_fields/2                                37 2.20   13    0.35
Enum."-map/2-lists^map/1-1-"/2                                                        80 2.36   14    0.18
:crypto.strong_rand_bytes_nif/1                                                        1 2.53   15   15.00
Ash.Changeset.force_change_attribute/3                                                10 2.70   16    1.60
:erlang.binary_to_atom/2                                                               8 2.70   16    2.00
:ets.match_object/2                                                                    4 4.22   25    6.25

As you can see, the call went from 2155 µS to 592 µS.

This is even more expressive when processing the data in bulk, my times when from (10_000 chunks) ~6 seconds to 0.3~0.5 seconds.

So, is there some way to optimize this checks in Ash.Changeset calls during casting?

If not, is there some way for me to disable the regex check during bulk insertion so I can make it faster?

20 comments

/ash #performance #regex

5 473 20

2026-01-14 17:45:16 UTC

Marked As Solved

sezaru

The PR was merged, so I’m closing this

Post #21

Also Liked

kip

ex_cldr Core Team

In unicode_string i have a function Unicode.Set.to_pattern/1 to transpile Unicode sets into binary patterns (or to :re-compatible regexs). Adapting that to transpile a subset of regex into pattern match wouldn’t be too much work so I’ll look at that.

Another thought that strikes me is that maybe, for Ash, something like the old COBOL PIC clause might be a good abstraction - it is declarative and it would map directly to a binary pattern match. The examples in this thread would be supported by such a construct. match could be reserved for regex, and pattern could be added for binary pattern matching.

Here I’m using the following symbols but they can anything:

X - ASCII alphanumeric
A - ASCII alphbetic
D - ASCII digits
Anything other character - a literal

I can envisage additional symbols for

Unicode alphanumeric
Unicode digit
ASCII alphabetic
Unicode alphabetic
Sign matching (+ / -)
Upper/lower case matching
Quote mark matching (open then close)
Unicode general category matching (generalised form of the above)
Unicode script matching

A pattern could look like the following (using the examples in the thread) :`

# ~r/^[A-Za-z0-9]{8}$/
X(8)

# ~r/^\d{6}$/
D(6)

# ~r/^\d{2}\.\d{3}\.\d{3}\/\d{4}-\d{2}$/
DD.DDD.DDD/DDDD-DD

I think this is quite an elegant and declarative way to express many (but clearly not all!) data format expectations. And since they can be transpiled to binary pattern matches they would be a good fit for the BEAM. They can also abstract away some of the complexities of “what is a letter”, “what is a digit” and “what is whitespace” given Unicode has quite extensive character repertories for these categories and others.

Post #5

sezaru

Ok, I tested your suggestion and it indeed fixed the issue, I didn’t even needed to change my constraint, I just leave it as a regex since spark will convert it into a cache regardless:

  defp validate_type(:regex_as_mfa, _key, %Regex{} = regex) do
    source = Regex.source(regex)
    opts = Regex.opts(regex)
    {:ok, {Spark.Regex, :cache, [source, opts]}}
  end

So, the only change I did was change the match type in lib/ash/type/string.ex from :regex to :regex_as_mfa.

With this, now the code is at least twice as fast. Here is a comparison with my code above but looped 10k times with eprof:

with :regex:

#                                                                 CALLS     %   TIME µS/CALL
Total                                                            966500 100.0 146590    0.15

with :regex_as_mfa

#                                                                 CALLS    %  TIME µS/CALL
Total                                                            912500 100. 59629    0.07

It is almost 3x faster.

@zachdaniel would you consider reverting the types for string.ex, ci_string.ex, type.ex and match.ex from that commit from :regex to :regex_as_mfa?

Post #18