I’m trying to optimize the codepagex
library so that it doesn’t use massive amounts of memory in compilation, because loading a lot of encodings requires an ungodly amount of GBs of RAM. The core issue is that the library loads all encodings’ transformation functions into a single module, so if there are many heavy encodings there will be many functions in that module in memory while it compiles.
My first thought was to split each encoding into it’s own module, then let each module compile in parallel and deload from memory once all its function clauses have been loaded. However, I’m suspecting this somehow disables the private_append optimization because I’m calling another module and passing the binary reference there, so the benchmarks show a massive increase in memory usage.
Here’s how the function clauses are built (my only change here is making the function names to_string
and making them public. I’ve also tested making them private but that doesn’t change the benchmark.
defmacro def_to_string(name, encoding) do
quote(bind_quoted: [n: name, e: encoding], generated: true, unquote: false) do
alias Codepagex.Mappings.Helpers
fn_name = Helpers.function_name_for_mapping_name("to_string", n)
for encoding_point <- e do
case encoding_point do
{from, to} ->
defp unquote(fn_name)(
unquote(from) <> rest,
acc,
missing_fun,
outer_acc
) do
unquote(fn_name)(
rest,
acc <> <<unquote(to)::utf8>>,
missing_fun,
outer_acc
)
end
end
end
defp unquote(fn_name)("", result, _, outer_acc) do
{:ok, result, outer_acc}
end
defp unquote(fn_name)(rest, acc, missing_fun, outer_acc) do
case missing_fun.(rest, outer_acc) do
res = {:error, _, _} ->
res
{:ok, codepoints, new_rest, new_outer_acc} ->
unquote(fn_name)(
new_rest,
acc <> codepoints,
missing_fun,
new_outer_acc
)
end
end
end
end
Here’s how the original code builds the clauses:
for {n, m} <- @encodings, do: Helpers.def_to_string(n, m)
# define methods to forward to_string(...) to a specific implementation
for {name, _} <- @encodings do
fun_name = Helpers.function_name_for_mapping_name("to_string", name)
def to_string(binary, unquote(name |> String.to_atom()), missing_fun, acc) do
unquote(fun_name)(binary, <<>>, missing_fun, acc)
end
end
def to_string(_, encoding, _, acc) do
{:error, "Unknown encoding #{inspect(encoding)}", acc}
end
# define the from_string_xxx for each encoding
for {n, m} <- @encodings, do: Helpers.def_from_string(n, m)
# define methods to forward from_string(...) to a specific implementation
for {name, _} <- @encodings do
fun_name = Helpers.function_name_for_mapping_name("from_string", name)
def from_string(
string,
unquote(name |> String.to_atom()),
missing_fun,
acc
) do
unquote(fun_name)(string, <<>>, missing_fun, acc)
end
end
def from_string(_, encoding, _, acc) do
{:error, "Unknown encoding #{inspect(encoding)}", acc}
end
And it’s benchmark:
Name ips average deviation median 99th %
ascii_to_string 2170.95 0.00046 s ±10.50% 0.00045 s 0.00060 s
iso_to_string 2027.25 0.00049 s ±9.27% 0.00049 s 0.00064 s
iso_from_string 1588.24 0.00063 s ±8.83% 0.00061 s 0.00080 s
ascii_from_string 345.06 0.00290 s ±9.99% 0.00276 s 0.00342 s
ascii_from_gigantic_string 0.136 7.38 s ±0.00% 7.38 s 7.38 s
iso_from_gigantic_string 0.135 7.40 s ±0.00% 7.40 s 7.40 s
erlang_unicode_from_gigantic_string 0.0785 12.73 s ±0.00% 12.73 s 12.73 s
Comparison:
ascii_to_string 2170.95
iso_to_string 2027.25 - 1.07x slower +0.00003 s
iso_from_string 1588.24 - 1.37x slower +0.00017 s
ascii_from_string 345.06 - 6.29x slower +0.00244 s
ascii_from_gigantic_string 0.136 - 16016.26x slower +7.38 s
iso_from_gigantic_string 0.135 - 16056.97x slower +7.40 s
erlang_unicode_from_gigantic_string 0.0785 - 27643.14x slower +12.73 s
Memory usage statistics:
Name Memory usage
ascii_to_string 195.31 KB
iso_to_string 195.31 KB - 1.00x memory usage +0 KB
iso_from_string 195.31 KB - 1.00x memory usage +0 KB
ascii_from_string 6757.81 KB - 34.60x memory usage +6562.50 KB
ascii_from_gigantic_string 414.06 KB - 2.12x memory usage +218.75 KB
iso_from_gigantic_string 195.31 KB - 1.00x memory usage +0 KB
erlang_unicode_from_gigantic_string 15661817.52 KB - 80188.51x memory usage +15661622.20 KB
**All measurements for memory usage were the same**
And here’s my refactor:
for {name, encodings} <- @encodings do
parsed_name = String.replace(name, ["/", " "], "_")
module_name = Module.concat(Codepagex.Functions.Generated, parsed_name)
module_content =
quote bind_quoted: [name: name, module_name: module_name, encodings: encodings] do
defmodule module_name do
require Codepagex.Mappings.Helpers
alias Codepagex.Mappings.Helpers
Helpers.def_to_string(name, encodings)
Helpers.def_from_string(name, encodings)
end
end
{{:module, module_name, module_binary, _}, _} = Code.eval_quoted(module_content)
:code.load_binary(module_name, ~c"#{module_name}.beam", module_binary)
end
for {name, _} <- @encodings do
module_name = Helpers.module_name_for_mapping_name(name)
def to_string(binary, unquote(name |> String.to_atom()), missing_fun, acc) do
unquote(module_name).to_string(binary, <<>>, missing_fun, acc)
end
def from_string(binary, unquote(name |> String.to_atom()), missing_fun, acc) do
unquote(module_name).from_string(binary, <<>>, missing_fun, acc)
end
end
And its benchmark:
Name ips average deviation median 99th %
ascii_to_string 1165.07 0.00086 s ±12.50% 0.00084 s 0.00117 s
iso_to_string 1094.68 0.00091 s ±14.99% 0.00088 s 0.00125 s
iso_from_string 900.31 0.00111 s ±12.60% 0.00108 s 0.00160 s
ascii_from_string 295.03 0.00339 s ±14.80% 0.00315 s 0.00429 s
erlang_unicode_from_gigantic_string 0.0615 16.25 s ±0.00% 16.25 s 16.25 s
ascii_from_gigantic_string 0.0469 21.33 s ±0.00% 21.33 s 21.33 s
iso_from_gigantic_string 0.0395 25.35 s ±0.00% 25.35 s 25.35 s
Comparison:
ascii_to_string 1165.07
iso_to_string 1094.68 - 1.06x slower +0.00006 s
iso_from_string 900.31 - 1.29x slower +0.00025 s
ascii_from_string 295.03 - 3.95x slower +0.00253 s
erlang_unicode_from_gigantic_string 0.0615 - 18936.89x slower +16.25 s
ascii_from_gigantic_string 0.0469 - 24845.47x slower +21.32 s
iso_from_gigantic_string 0.0395 - 29528.73x slower +25.34 s
Memory usage statistics:
Name Memory usage
ascii_to_string 1.56 MB
iso_to_string 1.75 MB - 1.12x memory usage +0.191 MB
iso_from_string 2.14 MB - 1.37x memory usage +0.57 MB
ascii_from_string 8.54 MB - 5.46x memory usage +6.98 MB
erlang_unicode_from_gigantic_string 15294.74 MB - 9779.09x memory usage +15293.18 MB
ascii_from_gigantic_string 38147.34 MB - 24390.48x memory usage +38145.77 MB
iso_from_gigantic_string 38147.13 MB - 24390.34x memory usage +38145.56 MB
**All measurements for memory usage were the same**
Any help or pointer is appreciated!