Compiling files with many function heads is very slow(OTP 26 issue?)

I’ve only seen such huge slowdown when I was trying to compile for ARM on my x64 Mac. Are you sure you are using fully ARM-native terminal, homebrew and everything?

It looks like everything is running on arm.

$ uname -m
arm64

And ActivityMonitor shows beam.smp running as Apple, which I believe is arm64

It would be nice if you showed at least some example on what your functions look like, from what I understand you are generating code based on this file?

Here’s a stripped down version of what i’m doing. This example doesn’t use the Unihan file because that part has a bit of preprocessing, but it’s ultimately similar. Here hepburn.tsv and ja_punct.tsv, are files of two columns, that look like this:

あ	a
リュ	ryu
ロー	rō
ヲー	wō
ヤ	ya

Hepburn is 604 lines, ja_punct is 91 lines.

defmodule JA do
  @priv :code.priv_dir(:my_app)
  @ja_path Path.join(@priv, "langs/ja")
  @punct_path Path.join(@priv, "ja_punct.tsv")
  @hepburn_path Path.join(@ja_path, "hepburn.tsv")
  @hepburn File.read!(@hepburn_path)
  |> String.trim()
  |> String.split("\n")
  |> Enum.map(&String.split(&1, "\t"))
  |> Enum.map(&List.to_tuple/1)

  @punct File.read!(@punct_path)
  |> String.trim()
  |> String.split("\n")
  |> Enum.map(&String.split(&1, "\t"))
  |> Enum.map(&List.to_tuple/1)

  @all Enum.sort_by(@hepburn ++ @punct, fn {a, _} -> -byte_size(a) end)

  def replace("" <> string), do: replace(string, [])

  def replace("", acc), do: acc

  Enum.each(@all, fn {k, v} ->
    def replace(<<unquote(k), rest::binary>>, acc),
      do: replace(rest, [unquote(Macro.escape(v)) | acc])
  end)

  def replace(<<char::binary-size(1), rest::binary>>, acc), do: replace(rest, [char | acc])
end

This might be a good place to start. From what I remember Enum.sort_by/3 is not one of the most performant of the functions. I guess a place to start debugging might be by using timer, and I would start from this sort function as is one of the possible bottleneck places.

Consider something like nimble_csv to parse the tsv files. You’re iterating a bit more often than needed through the data.

If I run those module attributes with an |> IO.inspect() at the end, they all finish within a second, which leads me to believe the issue is not so much with the parsing of the TSV or the sorting. I just logged a count, and it’s a total of 16,089 items that would be used to generate the functions dynamically. From what I can tell, the actual compile time code and the elixir side run in a matter of seconds, I think it’s the erlang compilation that’s taking 30minutes.

It’s certainly likely that this can be optimized, but from what I can tell, it’s not the culprit. I’m not sure if this is an accurate way to test how long those module attributes take to run, but if I do something like this:

defmodule JA do
  @time System.monotonic_time(:millisecond)

  @data ....
    |> ....
    |> IO.inspect(label: "Done #{System.monotonic_time(:millisecond) - @time}")
end

It shows that all my module attributes run in <200 milliseconds.

That’s useful information. In this case it’s really likely that this is due to the compilation of functions, not the compile time logic. Iirc from older discussions anything beyond like the 1000 of functions/heads would be considered problematic, so 16k is indeed large. Iirc cldr libraries did also run into this every now an then. That might be something to search for.

Yeah, it’s probably certainly beyond what they’re developing it for, but it is a fairly clean way to code it, so that’s what drew me to it. Since there’s so much variation in the byte size of what I’m matching, and ultimately I can’t ask all the worlds languages to change, so there’s going to be the number of items in the search space that the language dictates, I could recode it to do something like progressively pulling a byte off the front of the string, and accumulate that until it matches a key in a Map, which would hold the 16k items, but that seems a bit convoluted in comparison to the current solution.

But with all that said, my current solution works just fine, and in OTP 25 it compiled in a reasonable 20-30 seconds rather than 20-30 minutes, so maybe I’ll just go back down to OTP 25 for now.