Hello! I’ve been using Elixir for a while and loving it, with one successfully deployed project and another under development. For part of this project I’m autogenerating Elixir modules from a domain-specific language. The DSL and code generation was pretty straightforward (thank you
:yecc) but now I’m wondering how to include the generated files into the same Mix compile step that the rest of the modules will be compiled in. Currently I have two thoughts so far:
__mix_recompile__?() with wildcards to search for the generated files and pass them to
- Add a new custom compiler to
:compilers which builds the modules before the main elixir compiler kicks in.
Both of these seem like viable options but both seem like overkill to me. I find myself asking the question, “shouldn’t there be a better way to do this?”. Any thoughts? Thanks in advance!
Without knowing much of your project I’d naively go for generating the AST and then convert it to source and save it into files that can be included into something like
lib/<your_app>/codegen or some such. That would skip all potential other complexities like the ones you enumerated; you just generate code and then treat it like normal project source, only isolated in a directory of files that you know not to edit by hand.
Thank you @dimitarvp! Indeed that is my plan, but I’m hoping for the whole process to be accomplished every time I run
mix compile: it will look for directory(s) in the source containing a DSL file, generate as many Elixir modules as needed per the DSL code, and then immediately compile those modules as part of the current application in the same source tree. Then if there are changes made to any DSL file the next
mix compile will notice those and regenerate the Elixir modules and recompile them.
So far I’m leaning towards option #2 (see my original post) but it’s going a bit slow since this seems to be sparsely documented Elixir territory
I’d say just aiming for
mix compile to do the whole thing might introduce too many complexities.
A compromise I’d reach for is to have a separate (and custom, written by you)
mix task that you invoke before
compile and you can just alias those two to e.g.
Another thing that could help start you off is
Oh right! I hadn’t thought of breaking it out to a separate mix step, though I still prefer to do it inside the regular
mix compile if possible. I’ll continue my current attempt and try your suggestion if it isn’t working out. Thanks!
Thanks again @dimitarvp. I thought I’d report on what I did since eventually it worked as I originally hoped, to build a custom compiler and include it in
- Move the transpiler code to a new app in my umbrella project and make my current app dependent on that so the compile-related code always gets compiled by elixir before my regular app begins compilation
- Add a custom compile mix task (Mix.Task.Compiler — Mix v1.13.3) and reference it in my main app’s .exs file
compilers: [:custom_compiler_name | Mix.compilers()]
- Ensure the
@recursive attribute is set to
true in that custom compiler task
defmodule Mix.Tasks.Compile.CustomCompilerName do
# The recursive module attribute tells Mix that we want to get the
# project config for the specific project even if it is called from
# the umbrella root. Otherwise we get the project config of the
# umbrella project itself. Not intuitive at all!
- Follow the general template of these source files from the Elixir compiler to do proper change detection of each of our DSL files and store their hash values in a manifest file.
- Once I saw how Elixir’s compiler generates its own manifest file, I realized I could simply include the
compile.elixir manifest in my change detection algorithm to ensure that any changes to the compiler app (or any of its dependencies) will cause all my DSL files to be considered stale and force a recompile of all of them. This helper returns the full path of that manifest file for the specified app.
We can compare that file’s last modified time with our own manifest’s modified time to see if any dependencies have changed.
defp get_dep_manifest(dep_app) do
Mix.Project.in_project(dep_app, Mix.Project.deps_paths()[dep_app], fn _module ->
modified = Mix.Utils.last_modified(my_manifest)
dep_manifest = get_dep_manifest(dep_app)
dep_manifest_modified = Mix.Utils.last_modified(dep_manifest)
dep_changed? = dep_manifest_modified > modified
Of course the full details of our implementation are specific to our needs but hopefully this much can be helpful for anyone else interested in doing something similar.