Recent Spark performance improvements should have significant positive effects on Ash application performance

Hey folks, made some recent performance improvements to spark, the tool underlying all of our DSLs. GitHub - ash-project/spark: Tooling for building DSLs in Elixir · GitHub

It should make code that accesses DSL options significantly faster. If people can try it out with mix deps.update spark that would be great Bonus points to anyone who has some level of performance measuring, if you could show how its impacted things for you, that would be great

Some technical context around what was actually done:

Modules that use the DSL built by spark create a data structure under the hood, that looks like this:

%{
  persisted: %{...explicitly cached things that are used often},
  [:attributes] => %{entities: [...], options: [...]},
  [:json_api, :routes] => %{entities: [...], options: [...]},
  ... and so on
}

This data structure groups everything by section. However, when accessing DSL information, you always go through a set of functions provided by spark, for example Spark.Dsl.Extension.get_opt and Spark.Dsl.entities. We did this on purpose to allow us to optimize the underlying storage mechanisms and access patterns. While doing some recent benchmarking, we saw that the current naive implementations were taking quite a bit of time during the execution of Ash actions. The implementation would be something like dsl |> Map.get(path) |> Map.get(:options) |> Keyword.get(option_name). We would do similar things with entities.

What we’ve instead done is defined functions under the hood for each section path and option, that pattern matches and returns the value. It looks like this:

        for {path, %{opts: opts}} <- @spark_dsl_config, is_list(path) do
          for {key, value} <- opts do
            def fetch_opt(unquote(path), unquote(key)) do
              {:ok, unquote(Macro.escape(value))}
            end
          end
        end

        def fetch_opt(_, _), do: :error

Then we updated Extension.get_opt to call this generated function instead of getting the big DSL map and pulling out values. Measurements so far show significant performance improvements (especially in bulk creates, which s what sparked this recent investigation into unnecessary slowness).

Aside from this, we also leveraged the persist key to optimize a bunch of commonly accessed Ash DSL items.

Please give it a try! Bonus points to anyone who has some level of performance measuring, if you could show how its impacted things for you, that would be great