That is because of macros. Compilers for languages with macros are usually pretty weird, because compilation is not a single direction pipeline any more, but rather something that goes back and forth between the compiler and the user-provided macros. The regular approach of a compiler seen as a neatly ordered set of passes breaks down when you introduce true macros.
Depends on what level the macro’s run at. I’ve made a few micro-lisp variants in a few languages that have both read-macro and normal macro support and they ran at parsing time, such as when the prefix character for a read-macro was encountered at the start of an expression then it passed the parser into the read-macro callback until it returned with the ast and the updated parser state, which would then expand the ast, and the ast for macro’s was expanded on the fly as ast elements were completed (so if a macro was called like via (my-macro 1 2 3 4) then it would recognize that the my-macro call was a macro, then finish parsing the elements then execute the my-macro callback passing in the rest of the AST to it and then expand what was returned as well before storing it inline, this all happened before anything after the closing ) was parsed). And yes, though not with the lisps but with some of the others you can still have out-of-order macro execution (using a function before it’s definition in a file) by just binding the function name to a use list that must be empty by the end of parsing, and if a macro is defined then you check that list and execute it over it’s component ast and replace as necessary.
I’ve always found context-sensitive parsing to be far superior to staged parsing, both in how easy they are to reason about (assuming you use a decent library) as well as being faster, plus you can even do crazy stuff like invent new syntax on the fly that the original parser was never designed to handle if you so wish to have such features (basically read-macro’s).
Just copying José’s response here to mark as answer:
It is not a breaking change because the previous behaviour was never expected nor specified. do/end blocks should either have expressions OR clauses but never both. I will be glad to document this behaviour more clearly to avoid such confusion in the future but, in this case, I am afraid you were relying on accidental behaviour.
I don’t think having a spec would have prevented the bug, because the behaviour was always clear to me when implementing this feature set, but the bug was still unnoticed for large periods of time. Maybe it would have been caught earlier though.
That said, I definitely wouldn’t want all parser bugs to be considered features. Over the years we have improved how we spec our syntax, including the document changed above, so I would continue treading this path.
My suggestion here would be to replace those operators by any of our custom operators.
2 Likes
Please try implementing it and let us know of any edge cases you run into!
You have no idea how much I want the time to do so! It would make a good base for an Elixir 2.0 I think since it would probably break special casings like for/with as they currently stand (unless that feature was opened up somehow, I could imagine a macro named blah being encoded as MACRO*-blah instead of MACRO-blah that takes a list of arguments…). First I think I’d want a spec. ^.^;
Hmm, though depending on how the system is written, the spec could potentially be extracted from it…
EDIT: As an aside, there are quite a number of translation languages that do just that, like take Lisp, it’s built on a very simple restricted set of primitives. 
Remember the discussion we had about Elixir 2.0 where we actually don’t plan to introduce breaking changes. I think starting Elixir 2.0 with the assumption that we are breaking the syntax and macros would be really bad. We might as well call it something else.
Redesigning the parser and compiler toolchain in hindsight is very straight-forward if we assume we do not have to abide to any of the rules that exist today. But back in the real world we need to balance all of those trade-offs. If you are not happy that we broke something that worked by accident… imagine what will happen when we break things that are working as designed.
Yeah that’s why I think a macro that takes an unbounded argument count would be useful, would keep backwards compat and open that feature up properly as well.
Yeah it can be done without bumping the major version, I think the special forms are the only oddities, just need to encode them properly to open up their features to the front-end so they can actually be designed within the language then.
Implementing an Elixir parser with NimbleParsec sounds fun. I might try it.
1 Like
