Regarding `end_of_expression` metadata

Questions / Help
1

The docs state:

  • :end_of_expression - denotes when the end of expression effectively happens. Available for all expressions except the last one inside a __block__ (when :token_metadata is true)

My question is about “except the last one inside a __block__”.

Is limitation based on the existing parser implementation or is there an “actual” (in quotes because I’m not sure the right wording here haha) reason for the exception?

cc @josevalim

2

Another question actually, is end_of_expression only included if the end of the expression cannot be computed from {line, column + length(token)}?

3

I think it is because the last expression end is marked inside the __block__ already. But we can add more metadata if necessary.

4

+1 for more metadata. No end_of_expression means editor tools need to reimplement this in every use case that involves going from AST to source code ranges. Here’s an example from ElixirLS elixir-ls/apps/language_server/lib/language_server/providers/document_symbols.ex at 7b3344cc845d0c608c291934bdc0e95b74a26da4 · elixir-lsp/elixir-ls · GitHub. Compare that to elixir-ls/apps/language_server/lib/language_server/providers/document_symbols.ex at 7b3344cc845d0c608c291934bdc0e95b74a26da4 · elixir-lsp/elixir-ls · GitHub

5

I’ll look into it more to confirm, thanks!

6

Given the following code and result from Code.string_to_quoted/2

iex(4)> Code.string_to_quoted(~S'''
...(4)>       defmodule Foo do
...(4)>         use AnotherMod.Nested,
...(4)>           some: :option
...(4)>
...(4)>         def run(arg) do
...(4)>           bar()
...(4)>           :ok
...(4)>         end
...(4)>       end
...(4)> ''', columns: true, token_metadata: true)
{:ok,
 {:defmodule,
  [do: [line: 1, column: 21], end: [line: 9, column: 7], line: 1, column: 7],
  [
    {:__aliases__, [last: [line: 1, column: 17], line: 1, column: 17], [:Foo]},
    [
      do: {:__block__, [],
       [
         {:use,
          [
            end_of_expression: [newlines: 2, line: 3, column: 24],
            line: 2,
            column: 9
          ],
          [
            {:__aliases__, [last: [line: 2, column: 24], line: 2, column: 13],
             [:AnotherMod, :Nested]},
            [some: :option]
          ]},
         {:def,
          [
            do: [line: 5, column: 22],
            end: [line: 8, column: 9],
            line: 5,
            column: 9
          ],
          [
            {:run, [closing: [line: 5, column: 20], line: 5, column: 13],
             [{:arg, [line: 5, column: 17], nil}]},
            [
              do: {:__block__, [],
               [
                 {:bar,
                  [
                    end_of_expression: [newlines: 1, line: 6, column: 17],
                    closing: [line: 6, column: 15],
                    line: 6,
                    column: 11
                  ], []},
                 :ok
               ]}
            ]
          ]}
       ]}
    ]
  ]}}

I am wondering what is the criteria that makes the use call and the bar() function call get an :end_of_expression metadata, but not any of the other expressions?

I am not hoping for any particular answer, just want to know exactly how it works so I can reproduce it in Spitfire.

Thanks @josevalim.

7

Actually maybe I have figured it out moments after posting.

Is it because the the tokens given to use are just AST, and not actually “expressions” at parse time, since use is a macro? Same thing for the arguments to def, and def and :ok being the last expression inside their do blocks?

8

They are all expressions. The current docs state though that end_of_expression is added only to expressions with a __block__ (except the last one). The purpose of end_of_expression was to format blocks properly, which is why it is limited in scope. We could add it to all constructs in a block though. Adding them to everything is a taller order (and probably not that useful).

9

Oh, this was a missing piece then for me.

The current docs don’t actually mention that it’s only expressions in a block

denotes when the end of expression effectively happens. Available for all expressions except the last one inside a __block__ (when :token_metadata is true)

I can easily incorporate this I think.

Thanks for helping me understand this! :bowing_man:t2:

PS: I don’t personally have a need to increase or decrease the metadata currently, just needed to know how the existing metadata works.

10

Yeah. I actually tried to say that but my english failed me. :smiley:

11

This is obvious now that I knew where to look in elixir_parser.yrl.

Thanks again for the help José!