Sourceror - Utilities to work with Elixir source code

Random, pretty particular question. I was curious about writing something that could alphabetize my Mix dependencies. Would this be the right tool to build that with?

Yes! What you have to be mindful of with this kind of manipulations in contrast with macros is that you need to consider how line numbers move around. You need to both reorder the dependencies, and correct the line numbers, otherwise comments may be misplaced. The reason is that Code.quoted_to_algebra requires the ast and comments as separate arguments and mixes them by their line numbers.

One way to achieve what you want is by doing this:

"""
defp deps do
  [
    {:a, "~> 1.0"},
    {:z, "~> 1.0"},
    {:g, "~> 1.0"},
    # Comment for r
    {:r, "~> 1.0"},
    {:y, "~> 1.0"},
    # Comment for :u
    {:u, "~> 1.0"},
    {:e, "~> 1.0"},
    {:s, "~> 1.0"},
    {:v, "~> 1.0"},
    {:c, "~> 1.0"},
    {:b, "~> 1.0"},
  ]
end
"""
|> Sourceror.parse_string()
|> Sourceror.postwalk(fn
  {:defp, meta, [{:deps, _, _} = fun, body]}, state ->
    [{{_, _, [:do]}, block_ast}] = body
    {:__block__, block_meta, [deps]} = block_ast

    lines = Enum.map(deps, fn {:__block__, meta, _} -> meta[:line] end)

    deps =
      Enum.sort_by(deps, fn {:__block__, _, [{{_, _, [name]}, _}]} ->
        Atom.to_string(name)
      end)

    deps =
      Enum.zip([lines, deps])
      |> Enum.map(fn {old_line, dep} ->
        {_, tuple_meta, [{left, right}]} = dep
        line_correction = old_line - tuple_meta[:line]

        tuple_meta = Sourceror.correct_lines(tuple_meta, line_correction)
        left = Macro.update_meta(left, &Sourceror.correct_lines(&1, line_correction))
        right = Macro.update_meta(right, &Sourceror.correct_lines(&1, line_correction))

        {:__block__, tuple_meta, [{left, right}]}
      end)

    quoted = {:defp, meta, [fun, [do: {:__block__, block_meta, [deps]}]]}
    state = Map.update!(state, :line_correction, & &1)
    {quoted, state}

  quoted, state ->
    {quoted, state}
end)
|> Sourceror.to_string()
|> IO.puts()

# =>
defp deps do
  [
    {:a, "~> 1.0"},
    {:b, "~> 1.0"},
    {:c, "~> 1.0"},
    {:e, "~> 1.0"},
    {:g, "~> 1.0"},
    # Comment for r
    {:r, "~> 1.0"},
    {:s, "~> 1.0"},
    # Comment for :u
    {:u, "~> 1.0"},
    {:v, "~> 1.0"},
    {:y, "~> 1.0"},
    {:z, "~> 1.0"}
  ]
end

The other thing to note is that this is not your regular AST, Sourceror uses the literal_encoder: &{:ok, {:__block__, &2, [&1]}} option for Code.string_to_quoted_with_comments/2 under the hood, so you need to expect literals to be wrapped in blocks, so for example {:a, "~> 1.0"} will become:

{:__block__, [line: 1], [
  {{:__block__, [line: 1], [:a]},
   {:__block__, [line: 1, delimiter: "\""], ["~> 1.0"]}}
]}

This is explained a bit in the Formatting considerations section of the new functions

Of course I will try to expand Sourceror as we find more complex use cases that could be simplified

@dorgan this is exciting work - thank you!

Can you talk a little bit about who is the target audience, and your vision for possible use-cases?

Could this be used by something like elixir-ls to implement refactoring operations? (eg rename variable, rename function, extract function, inline function, rename module)

What types of contributions and testing would be most useful to you?

The target audience is primarily tool authors, like elixir-ls or credo.

Yes, those are the kind of use cases I had in mind The Sourceror.to_string/2 function has an option to set the indentation level of the resulting code for that particular use case. I will probably add functions to know how many lines an ast node uses, so one could replace a line range instead of the whole file.

This started while exploring ways to allow credo to autofix some of the issues it finds, the multi alias expansion example derived from that.

Mostly finding what people find most cumbersome or confusing to do, I think the most important thing right now is to start experimenting. There may be some bugs in Code.quoted_to_algebra/2 too, some experiments in that front would be nice as well so we can add more regression tests to core Elixir

Sourceror is now available on hex.pm and supports Elixir versions down to 1.10

@dorgan - thanks for the support down to 1.10!

Here’s a question about Sourceror and Elixir types and doctests…

I expect that a Sourceror transformation could modify a function signature or a return type…

Would Sourceror also have the ability to transform typespecs? Or inline documentation? (eg for doctests)

It should be possible, it’s information you have in the AST
For example:

iex(4)> Sourceror.parse_string(~S"""
...(4)> @spec foo(String.t()) :: :ok
...(4)> def foo(a \\ 5), do: a + 10
...(4)> """)
{:__block__, [trailing_comments: [], leading_comments: []],
 [
   {:@,
    [
      trailing_comments: [],
      leading_comments: [],
      end_of_expression: [newlines: 1, line: 1],
      line: 1
    ],
    [
      {:spec, [trailing_comments: [], leading_comments: [], line: 1],
       [
         {:"::", [trailing_comments: [], leading_comments: [], line: 1],
          [
            {:foo,
             [
               trailing_comments: [],
               leading_comments: [],
               closing: [line: 1],
               line: 1
             ],
             [
               {{:., [trailing_comments: [], leading_comments: [], line: 1],
                 [
                   {:__aliases__,
                    [trailing_comments: [], leading_comments: [], line: 1],
                    [:String]},
                   :t
                 ]},
                [
                  trailing_comments: [],
                  leading_comments: [],
                  closing: [line: 1],
                  line: 1
                ], []}
             ]},
            {:__block__, [trailing_comments: [], leading_comments: [], line: 1],
             [:ok]}
          ]}
       ]}
    ]},
   {:def, [trailing_comments: [], leading_comments: [], line: 2],
    [
      {:foo,
       [
         trailing_comments: [],
         leading_comments: [],
         closing: [line: 2],
         line: 2
       ],
       [
         {:\\, [trailing_comments: [], leading_comments: [], line: 2],
          [
            {:a, [trailing_comments: [], leading_comments: [], line: 2], nil},
            {:__block__,
             [trailing_comments: [], leading_comments: [], token: "5", line: 2],
             [5]}
          ]}
       ]},
      [
        {{:__block__,
          [
            trailing_comments: [],
            leading_comments: [],
            format: :keyword,
            line: 2
          ], [:do]},
         {:+, [trailing_comments: [], leading_comments: [], line: 2],
          [
            {:a, [trailing_comments: [], leading_comments: [], line: 2], nil},
            {:__block__,
             [trailing_comments: [], leading_comments: [], token: "10", line: 2],
             '\n'}
          ]}}
      ]
    ]}
 ]}

Typespecs are just module attributes, same for doc annotations, so you should be able to get the module attributes associated to a function by walking the tree with Macro.postwalk or Sourceror.postwalk if you’re also doing some transformation. You need to make some assumptions, though, for instance, considering a module attribute that comes before a function definition as being an annotation for such function. For doctests, since you already have access to the docstrings(they’re module attributes), it would be a matter of parsing the docstring looking for any doctest and then run Sourceror functions on it.

I think the real difficulty comes from the fact that you need to change a node that is not a children of the function, and that comes before the function. My first thought is that by postwalking, if you go up and find a function, and want to update it’s typespec, there could be a way to tell the postwalker that an update should be performed in a sibling node, maybe passing a callback and making it reduce over the parent’s children(I do something similar in the multi-alias expansion example). All of this while also applying line corrections.

I believe it is possible because we have all the data we need, but it’s something I need to put some thought on to be able to do it in a reliable and relatively straightforward way, and it would definitely be something worth adding to the library

Sourceror v0.4.0 was released.

It fixes a couple bugs, makes several improvements to comments line corrections, and adds more functions to work with ranges and positions, like Sourceror.get_range/1 and Sourceror.compare_positions/2.

I also converted the multi alias expansion example into a proper document you can import to Livebook, with step-by-step explanations of the process: sourceror/notebooks/expand_multi_alias.livemd at main · doorgan/sourceror · GitHub

You can see the full changelog here.

Was thinking in the same direction, but writing a Credo check. A script to auto-sort the deps would be nice (although my editor currently does it also without issues)

Sourceror 0.6.0 is out

This release introduces some breaking changes, as the way comments are handled by the library has been fundamentally changed. In essence, instead of requiring the user to calculate how line numbers should be shifted, Sourceror tries to “fix” the line numbers in a way that makes sense for the Elixir formatter when you call Sourceror.to_string/2 or Sourceror.extract_comments/2.

To illustrate this, the dependency sorting example can now be reduced to this traversal:

Macro.postwalk(fn
  {:defp, meta, [{:deps, _, _} = fun, body]} ->
    [{{_, _, [:do]}, block_ast}] = body
    {:__block__, block_meta, [deps]} = block_ast

    deps =
      Enum.sort_by(deps, fn {:__block__, _, [{{_, _, [name]}, _}]} ->
        Atom.to_string(name)
      end)

    {:defp, meta, [fun, [do: {:__block__, block_meta, [deps]}]]}

  quoted ->
    quoted
end)

Note that, because the line number correction hack is no longer needed, traversals over Sourceror’s AST is the same as traversals over regular AST, just don’t discard comments metadata

The multi alias expansion livebook was further simplified thanks to this change.

Also, now Sourceror.get_range/1 returns the actual start and end positions for any node(provided it has line and column metadata), making it suitable for things like “replace the contents between these two positions with this new content”. Column offsets are counted as UTF-8 offsets(like the regular Elixir AST), so tools that want to support the Language Server Protocol need to convert them to UTF-16 offsets.

Changelog:

1. Enhancements

• [Sourceror] - to_string no longer requires line number corrections to produce properly formatted code.
• [Sourceror] - Added prewalk/2 and prewalk/3.
• [Sourceror] - parse_string won’t warn on unnecesary quotes.
• [Sourceror.TraversalState] - Sourceror.PostwalkState was renamed to Sourceror.TraversalState to make it more generic for other kinds of traversals.

2. Removals

• [Sourceror] - get_line_span was removed in favor of using get_range and calculating the difference from the range start and end lines.
• [Sourceror.TraversalState] - line_correction field was removed as it is no longer needed.

3. Bug fixes

• [Sourceror] - get_range now properly returns ranges that map a node to it’s actual start and end positions in the original source code.