A script for re-writing AST: where to start?

Questions / Help
,
1

I’m trying to write a tool, a script in Elixir, that will read a file containing code written in Elixir, rewrite a part of it, then save the contents as a new file.

I understand that reading Elixir code and writing it back could be done like this:

{:ok, ast} = Code.string_to_quoted(File.read!("my_module.ex"))

# magic that generates new_ast

File.write!("my_updated_module.ex", Macro.to_string(new_ast))

I’m looking for how to re-write the following code:

defmodule MyModule do
  use DSL

  declare :operation do
    description("My description")
  end
end

…into the following code:

defmodule MyModule do
  alias MyApp.Operation

  def operation do
    %Operation{
      description: "My description"
    }
  end
end

I am most curious about the “magic” part, e.g. re-writing the AST.

Elixir documentation gives example on Macro.prewalk/2, which looks like exactly something I’d need to use in my case. But since my case is more complicated than example in the docs, I am a bit lost & need to figure out a principle on which I’ll build.

For context, I’m trying to get rid of a custom DSL in favour of (arguably simpler) plain functions & looking for examples or ideas how to achieve this & do so in the entire app, automatically.

A project I am working on has a ton of invocations of custom DSLs like the above, and I’d like to write a tool that will go over all files that contain a DLS & convert all of them to function. My previous attempt (although only partially successful) was based on a elaborate bash script that utilised a combination of ripgrep & sed invocations.

2

Overview — Sourceror v0.12.1 might be helpful both as a library and for some examples

3 Likes
3

Highly recommend Sourceror. I’ve been using it recently for a library of mine and it’s been a pleasure. There’s a #sourceror channel in the Elixir Slack as well that’s not extremely chatty but gets quick responses if you have a question or issue!

2 Likes
4

I’m just digging into all of this myself and third Sourceror.

The examples directory in Sourceror’s repo gives a good example of matching on a node then manipulating a that node’s children. This is half of what you want, you just need to also replace the parent in the same go.

@zachallaun That’s good to know the channel is open for questions—I joined the other day and wasn’t sure if it was just for development talk or not. I certainly won’t be flooding it with questions (I’m pretty new to working with trees so I have a lot, but everything has been pretty searchable) but good to know it’s open for them!

2 Likes
5

Thank you, everyone!

At the time of posting this, I was still learning and couldn’t wrap my head around even the simplest replacement like in my example.

Here’s the code I eventually ended up with, to make the conversion from one module to another, like in my original post:

{:ok, ast} = Code.string_to_quoted(File.read!("/tmp/my_module.ex"))

new_ast =
  Macro.prewalk(ast, fn
    {:use, _meta, [{:__aliases__, _another_meta, [:DSL]}]} ->
      quote do
        alias MyApp.Operation
      end

    {:declare, _meta, children} ->
      [
        my_operation,
        [
          do: {my_key, _meta, [my_value]}
        ]
      ] = children

      quote do
        def unquote(my_operation)() do
          %Operation{
            unquote(my_key) => unquote(my_value)
          }
        end
      end

    other ->
      other
  end)

File.write!("/tmp/my_updated_module.ex", Macro.to_string(new_ast))
File.read!("/tmp/my_updated_module.ex") |> IO.puts()

Right now it appears that Macro.to_string(new_ast) formats things nicely, which is enough for my simple use case. For more complicated cases I might use Sourceror, which I looked at and got and impression it pays a lot of attention to how code is formatted, as well as provides support for comments.

6

Sourceror author here, just wanted to say that I regularly check the #sourceror channel in the Elixir slack, but I’m most active in the elixir Discord server so feel free to reach out if you have questions about it.

You can use Sourceror for simple cases too, the focus of Sourceror is to make it easy to change AST and preserve comments, which is hard to do yourself until someone writes an alternative parser and formatter for Elixir. It’s not about “complex manipulations” but rather the primitive for these manipulations.

In your code example you just need to replace Code.string_to_quoted with Sourceror.parse_string, and Macro.to_string with Sourceror.to_string and you’ll get the comments preserved as long as you don’t discard the nodes metadata. It’s a drop-in replacement for standard lib functions.

For more complex use cases you can look at Sourceror.Zipper for traversal/modification, and the Recode project. Folks in the community are already building higher level tools on top of Sourceror so with time more tools wil be available for more complex tasks, but for now you can think of Sourceror as a drop-in replacement to parse and format elixir code with a bunch of small utilities.

To give a bit more context, I added Code.string_to_quoted_with_comments and Code.quoted_to_algebra to Elixir to enable source code manipulation, the latter is what today powers Macro.to_string, but actually handling comments was better handled by a community library(it’s deceivingly complex and Elixir itself doesn’t use that internally) so that portion of the work is what became Sourceror.

6 Likes