CodeGen - simple, succint and customizable code generation for your libraries

CodeGen

(experimental!) code here: GitHub - tmbb/ex_code_gen: Flexible code generation for Elixir · GitHub

Suppose you want to have your library generate some code in the user’s module. That is pretty common in Elixir, and it’s usually done through the use XXX macro.That is flexible and succint, in that it doesn’t add any source code to the user’s file. On the other extreme, you can have code generators (normally called using mix package.gen.something ...). These generators dump literal source code in the middle of your project, often adding hundreds of lines but allow you to customize those lines as much as you want.

However, up until now, there isn’t a simple way of combining both approaches above. Ideally, one would use a module but retain the ability to add the literal source code of several functions so that they can be customized. This (experimental!) package was prompted by the discussion here: https://forum.elixirforum.com/t/phoenix-1-7-feels-a-little-bit-locked-in-with-tailwindcss/54468/43

I’m tagging @josevalim and @thiagomajesk because of that discussion.

Let’s look at an example. First, let’s define a code template, which is simply a module that defines a __code_gen__(options) function (not a macro, there’s no need for it to be a macro!) which returns the AST of the code we want to inject into the module (the code is large, but it’s mainly because it’s highly commented, the actual injected code is very simple):

defmodule CodeGenTemplate do
  def __code_gen__(opts) do
    c1 = Keyword.get(opts, :c1, 1)
    c2 = Keyword.get(opts, :c3, 2)
    c3 = Keyword.get(opts, :c3, 3)

    quote do
      # Define some code inside a block, which we'll be able
      # to dump into our own module if we want
      CodeGen.block "f1/1" do
        @constant1 unquote(c1)

        def f1(x) do
          x + @constant1
        end
      end

      # Anothe code block
      CodeGen.block "f2/1" do
        def f2(x), do: x + unquote(c2)
      end

      # This code block is more complex and has comments
      CodeGen.block "f3/1" do
        # Comments are completely removed by Elixir's parser.
        # Ways of preserving comments while easily allowing AST
        # manipulation in an idiomatic way.
        #
        # The workaround is to use these custom attributes
        # which are completely removed from the AST and replaced
        # by comments when you dump the source code in the module
        @comment__ "This is a comment outside the function"
        @comment__ "This is a another comment outside the function"
        @comment__ "Yet another comment"
        @newline_after_comment__
        def f3(x) do
          @comment__ "this is a comment inside the function"
          x + unquote(c3)
        end
      end

      # You can define functions outside a code block
      # if you don't want the user to be able to redefine the function.
      def another_function() do
        # ...
      end

      # We need to mark some functions as overridable so that we can actually
      # dump their source code into the module and things will work.
      # It's too hard for CodeGen to understand which functions are being generated
      # and generate this list on its own.
      defoverridable f1: 1, f2: 1, f3: 1
    end
  end
end

Now, we can use that code in a module:

defmodule CodeGenExample do
  use CodeGen,
    # The `:module` is our template, which must define a `__code_gen__(options)` function
    module: CodeGenTemplate,
    # Use the options to customize the generated code
    options: [
      c1: 7
    ]
end

Looking at the example module, we see that the code above is quite similar to a normal use CodeGenTemplate invocation, except for the fact that it is more explicit regrading the fact that we’re using the CodeGen module, which will support additional features.

Now, suppose we want to customize the f1/1 function. The way to do it is to edit the source code directly, but the problem with macros that generate code is that there is no source code for us to edit! However, this is where the special features in the CodeGen module become useful. Remember we have defined a number of named blocks. First, we can query the module to see which block names are available (of course, the author of the template module should make that clear in the documentation, but querying the block names is always halpful):

iex> CodeGen.block_names(CodeGenExample)
["f1/1", "f2/1", "f3/1"]

Nice! But we’d like to be able to actually see the blocks’ contents, so that we know what we’ll be including in advance. That is also easy:

iex(6)> CodeGen.show_blocks(CodeGenExample)
┌────────────────────────────────────────────────────────
│ Block: f1/1
├───────────────────────────────────────────────────────
│ @constant1 7
│ def f1(x) do
│   x + @constant1
│ end
└───────────────────────────────────────────────────────
┌────────────────────────────────────────────────────────
│ Block: f2/1
├───────────────────────────────────────────────────────
│ def f2(x) do
│   x + 2
│ end
└───────────────────────────────────────────────────────
┌────────────────────────────────────────────────────────
│ Block: f3/1
├───────────────────────────────────────────────────────
│ # This is a comment outside the function
│ # This is a another comment outside the function
│ # Yet another comment
│
│ def f3(x) do
│   # this is a comment inside the function
│   x + 3
│ end
└───────────────────────────────────────────────────────

:ok

Now we know which code there is in each block. Suppose we want to dump the contents of block f1/1 into our own file. We just need to do:

iex> CodeGen.dump_source(CodeGenExample, "f1/1")
* injecting test/fixtures/immutable/code_gen_example.ex
:ok

The file contents have been replaced by:

defmodule CodeGenExample do
  use CodeGen,
    module: CodeGenTemplate,
    options: [
      c1: 7
    ]

  @constant1 7
  def f1(x) do
    x + @constant1
  end
end

You can now customize the f1/1 function at will, while not having your code polluted by the code of the other functions you don’t need.

Applications

This CodeGen module is useful in any situation where you want to put some code in a module without adding literal code to the file, but in which you think you might have to customize some of the functions by editing the source code.

I can think of some uses for this:

Library behaviours, such as GenServer

Let’s say that instead of writing use GenServer you could write use CodeGen, module: GenServer. That way the GenServer module could define all callbacks inside the module, but you could do things such as CodeGen.dumo_source(MyGenserver, "handle_info/2") to have skeleton implementtion which you can edit´

Phoenix CoreComponents

Phoenix CoreComponents are meant to be customized by the user. However, the truth is that the default Phoenix generators dump A LOT of source code into the default project, some of which the user doesn’t care about, at least at the beginning. Some people (myself included) have complained about it here: https://forum.elixirforum.com/t/phoenix-1-7-feels-a-little-bit-locked-in-with-tailwindcss/54468/43. One of the suggestions made by @josevalim is that people should publish CoreComponents files customized to certain CSS frameworks (such as Bulma, Bootstrap, etc.). With CodeGen, one can publish a package that provides a custom CoreComponents file, which can be used like this:

defmodule MyAppWeb.CoreComponents do
  use CodeGen,
    # The (foreign) module providing the core components
    module: Bootstrap5CoreComponents,
    # Module-level customizations
    options: [
      horizontal_forms_by_default?: true,
      label_width: 3,
      input_width: 9,
      # ...
    ]
end

If the user wants to customize something like an input/1 component, then it’s simple to just start iex and do:

iex> CodeGen.dump_source(MyAppWeb.CoreComponents, "input/1")
* injecting test/fixtures/immutable/code_gen_example.ex
:ok

The code above could insert the code for the input/1 component, whitout polluting the source with other components which the user doesn’t need to modify.

Inclusion in Elixir’s Standard Library

I’ve looked into many languages which provide intersting facilities for code generation. The main ones are variants of Lisp in one way or another, but there are non-SExpr-based languages with such capabilities, such as OCaml (through a more or less complex build step), Haskell (which seems to provide very useful metaprogramming facilities, but which I’ve never tried due to the general dificulty of doing most things in Haskell as well as fear of lazy evaluation), Rust (which is kinda too low level for me). Other languages, like C have preprocessor-based metaprogramming.

Elixir has very impressive metaprogramming capabilities, which sets it apart from most other languages I’ve tried, and at the level of Lisp (and certainly more ergonomic than all the non-Lisp languages I’ve tried). However, the most unique feature of Elixir’s development I’ve found is the emphasis on generators to generate actual code which can be customized by the user. Phoenix’s generators, in particular, are a marvel in terms of generalizability and implementation (I know it because I’m maintaining parallel generators for my Mandarin admin package, and it’s actually quite hard to implement, maintain and test such generators).

I think that the functionality I’ve built with CodeGen bridges the functionality of metaprogrammning facilities (succint, safe, non-customizable) together ith the functionality of code generators (verbose, less safe - we never know what the user will do to “our” code, very customizable). This is an idea I’ve had for a long time but which up until now I’ve never got around to implement it. The implemenation is in a single (very short) file.

Given the emphasis given in Elixir to both metaprogramming and code generation, I’d like to encourage experimentation with these kinds of ideas (not necessarily with CodeGen, but with other similar tools one can develop) so that this could be stabilized and maybe included in the Elixir standard library.