gmile
A script for re-writing AST: where to start?
I’m trying to write a tool, a script in Elixir, that will read a file containing code written in Elixir, rewrite a part of it, then save the contents as a new file.
I understand that reading Elixir code and writing it back could be done like this:
{:ok, ast} = Code.string_to_quoted(File.read!("my_module.ex"))
# magic that generates new_ast
File.write!("my_updated_module.ex", Macro.to_string(new_ast))
I’m looking for how to re-write the following code:
defmodule MyModule do
use DSL
declare :operation do
description("My description")
end
end
…into the following code:
defmodule MyModule do
alias MyApp.Operation
def operation do
%Operation{
description: "My description"
}
end
end
I am most curious about the “magic” part, e.g. re-writing the AST.
Elixir documentation gives example on Macro.prewalk/2, which looks like exactly something I’d need to use in my case. But since my case is more complicated than example in the docs, I am a bit lost & need to figure out a principle on which I’ll build.
For context, I’m trying to get rid of a custom DSL in favour of (arguably simpler) plain functions & looking for examples or ideas how to achieve this & do so in the entire app, automatically.
A project I am working on has a ton of invocations of custom DSLs like the above, and I’d like to write a tool that will go over all files that contain a DLS & convert all of them to function. My previous attempt (although only partially successful) was based on a elaborate bash script that utilised a combination of ripgrep & sed invocations.
Marked As Solved
gmile
Thank you, everyone!
At the time of posting this, I was still learning and couldn’t wrap my head around even the simplest replacement like in my example.
Here’s the code I eventually ended up with, to make the conversion from one module to another, like in my original post:
{:ok, ast} = Code.string_to_quoted(File.read!("/tmp/my_module.ex"))
new_ast =
Macro.prewalk(ast, fn
{:use, _meta, [{:__aliases__, _another_meta, [:DSL]}]} ->
quote do
alias MyApp.Operation
end
{:declare, _meta, children} ->
[
my_operation,
[
do: {my_key, _meta, [my_value]}
]
] = children
quote do
def unquote(my_operation)() do
%Operation{
unquote(my_key) => unquote(my_value)
}
end
end
other ->
other
end)
File.write!("/tmp/my_updated_module.ex", Macro.to_string(new_ast))
File.read!("/tmp/my_updated_module.ex") |> IO.puts()
Right now it appears that Macro.to_string(new_ast) formats things nicely, which is enough for my simple use case. For more complicated cases I might use Sourceror, which I looked at and got and impression it pays a lot of attention to how code is formatted, as well as provides support for comments.
Also Liked
dorgan
Sourceror author here, just wanted to say that I regularly check the sourceror channel in the Elixir slack, but I’m most active in the elixir Discord server so feel free to reach out if you have questions about it.
You can use Sourceror for simple cases too, the focus of Sourceror is to make it easy to change AST and preserve comments, which is hard to do yourself until someone writes an alternative parser and formatter for Elixir. It’s not about “complex manipulations” but rather the primitive for these manipulations.
In your code example you just need to replace Code.string_to_quoted with Sourceror.parse_string, and Macro.to_string with Sourceror.to_string and you’ll get the comments preserved as long as you don’t discard the nodes metadata. It’s a drop-in replacement for standard lib functions.
For more complex use cases you can look at Sourceror.Zipper for traversal/modification, and the Recode project. Folks in the community are already building higher level tools on top of Sourceror so with time more tools wil be available for more complex tasks, but for now you can think of Sourceror as a drop-in replacement to parse and format elixir code with a bunch of small utilities.
To give a bit more context, I added Code.string_to_quoted_with_comments and Code.quoted_to_algebra to Elixir to enable source code manipulation, the latter is what today powers Macro.to_string, but actually handling comments was better handled by a community library(it’s deceivingly complex and Elixir itself doesn’t use that internally) so that portion of the work is what became Sourceror.
jerdew
Overview — Sourceror v1.12.2 might be helpful both as a library and for some examples
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"