Autogenerating Elixir module files from DSL and compiling them in the same step

Questions / Help
1

Hello! I’ve been using Elixir for a while and loving it, with one successfully deployed project and another under development. For part of this project I’m autogenerating Elixir modules from a domain-specific language. The DSL and code generation was pretty straightforward (thank you :leex and :yecc) but now I’m wondering how to include the generated files into the same Mix compile step that the rest of the modules will be compiled in. Currently I have two thoughts so far:

  1. Use __mix_recompile__?() with wildcards to search for the generated files and pass them to Code.require_file().
  2. Add a new custom compiler to Mix.project(), :compilers which builds the modules before the main elixir compiler kicks in.

Both of these seem like viable options but both seem like overkill to me. I find myself asking the question, “shouldn’t there be a better way to do this?”. Any thoughts? Thanks in advance!

1 Like
2

Without knowing much of your project I’d naively go for generating the AST and then convert it to source and save it into files that can be included into something like lib/<your_app>/codegen or some such. That would skip all potential other complexities like the ones you enumerated; you just generate code and then treat it like normal project source, only isolated in a directory of files that you know not to edit by hand.

3

Thank you @dimitarvp! Indeed that is my plan, but I’m hoping for the whole process to be accomplished every time I run mix compile: it will look for directory(s) in the source containing a DSL file, generate as many Elixir modules as needed per the DSL code, and then immediately compile those modules as part of the current application in the same source tree. Then if there are changes made to any DSL file the next mix compile will notice those and regenerate the Elixir modules and recompile them.

So far I’m leaning towards option #2 (see my original post) but it’s going a bit slow since this seems to be sparsely documented Elixir territory :sweat_smile:

4

I’d say just aiming for mix compile to do the whole thing might introduce too many complexities.

A compromise I’d reach for is to have a separate (and custom, written by you) mix task that you invoke before compile and you can just alias those two to e.g. mix build.

Another thing that could help start you off is @external_resource.

5

Oh right! I hadn’t thought of breaking it out to a separate mix step, though I still prefer to do it inside the regular mix compile if possible. I’ll continue my current attempt and try your suggestion if it isn’t working out. Thanks!

6

Thanks again @dimitarvp. I thought I’d report on what I did since eventually it worked as I originally hoped, to build a custom compiler and include it in mix compile.

  • Move the transpiler code to a new app in my umbrella project and make my current app dependent on that so the compile-related code always gets compiled by elixir before my regular app begins compilation
  • Add a custom compile mix task (Mix.Task.Compiler — Mix v1.13.3) and reference it in my main app’s .exs file
    compilers: [:custom_compiler_name | Mix.compilers()]
  • Ensure the @recursive attribute is set to true in that custom compiler task
    defmodule Mix.Tasks.Compile.CustomCompilerName do
  use Mix.Task.Compiler

  # The recursive module attribute tells Mix that we want to get the
  # project config for the specific project even if it is called from
  # the umbrella root.  Otherwise we get the project config of the
  # umbrella project itself. Not intuitive at all!
  @recursive true
...
  • Follow the general template of these source files from the Elixir compiler to do proper change detection of each of our DSL files and store their hash values in a manifest file.
  • Once I saw how Elixir’s compiler generates its own manifest file, I realized I could simply include the compile.elixir manifest in my change detection algorithm to ensure that any changes to the compiler app (or any of its dependencies) will cause all my DSL files to be considered stale and force a recompile of all of them. This helper returns the full path of that manifest file for the specified app.
      defp get_dep_manifest(dep_app) do
    dep_compile_lib_path =
      Mix.Project.in_project(dep_app, Mix.Project.deps_paths()[dep_app], fn _module ->
        Mix.Project.app_path()
      end)
    Path.join(dep_compile_lib_path, ".mix/compile.elixir")
  end
    We can compare that file’s last modified time with our own manifest’s modified time to see if any dependencies have changed.
    modified = Mix.Utils.last_modified(my_manifest)

dep_manifest = get_dep_manifest(dep_app)
dep_manifest_modified = Mix.Utils.last_modified(dep_manifest)
dep_changed? = dep_manifest_modified > modified

Of course the full details of our implementation are specific to our needs but hopefully this much can be helpful for anyone else interested in doing something similar.

3 Likes