hst337

hst337

Elixir compiler and language specification

Purpose of the proposal

Elixir language is in mature state and no breaking or heavy changes are expected to happen. Therefore “now” is the right time to start thinking about creating somewhat specification of the language. Current proposal is supposed to be the place to talk about necessity, usefulness and implementation of the language specification

Language and compiler specification

What is it?

Language specification is a somewhat formal definition and description of how the language runtime must/should/can/shouldn’t behave to be valid Elixir runtime. It may also define properties of standard library (like “Kernel.+ is always a functions”), etc.
Compiler specification is also a somewhat formal definition and description

Why should we need one?

  1. In case someone (like me) takes an opportunity to implement compatible compiler or language runtime or both. Anyway, for the language which is mature enough, it is only a question of time when third-party compilers and runtimes will appear, and reference-oriented implementations are almost always doomed to be incompatible in some edge cases. This leads to UBs among different compilers and runtimes, which is bad.

  2. Specification prevents exploits and incorrect usage of undocumented or unexpected behavior. I can provide some examples of undocumented things, which can be exploited to hack module compilation order, create weak inter-module dependencies, detect function overrides and so on. https://www.hyrumslaw.com/

  3. Specification defines more clearer path of language improvement, so that users could expect that some parts of language may change in the future. For example, marking that protocol functions (like Enumerable.reduce) may never be called if the compiler can derive the type of the first argument, can create a room for type-inference oriented compile-time optimizations.

  4. Process specification of development is also a process of exploration of improvement possibilities

Why can’t we just document this stuff?

Documentation is intended for users of the language, not for compiler or runtime developers. Therefore, some topics just do not belong in documentation, while others just do not need deep explanation.

Won’t it take a ton of time?

Specifications may (and they definitely will) evolve over time. It is not necessary to define every aspect of the language beforehand, specification can be ad hoc and reactive, defining just the most important points.

Can’t we just discuss language details here?

Yes, we can, but these discussions can be lost, they can become outdated or they can just be incorrectly answered by mistake.

What do we need to do to create the specification?

  1. Define a process of specification development. For example, “contributors create RFC (whose format defined here), core team approves”

  2. Define specification format and rules. For example, “MUST defines that every Elixir compiler must behave this way to be considered compatible”

  3. Write the specification


Let’s discuss

First 10 of 15 Posts Switch mode

hst337

hst337 OP

Who might be interested in specification

Without taking into account Elixir core team and myself there are several third-party projects who might benefit in having language specification and

  • Elixir linters can detect unreliable and exploiting behaviors (cc @rrrene)
  • LSP can have better completion around what and where can be called (cc @axelson)
  • Tools leveraging reflection and elixir metaprogramming approaches (like sourceror (cc @dorgan) and patch (cc @mdnowack)) could have a better understanding of how Elixir modules must work and how AST must behave, how variable contexts behave and so on.
  • erts developers could have a better understanding of how beam generated by Elixir programs is supposed to be executed
  • Elixir compiler developers (like myself)
D4no0

D4no0

Personally I think that this is not worth the time, you should understand that elixir without erlang is pretty much an empty shell, because some of the more advanced features like pattern matching are part of erlang, not elixir.

hst337

hst337 OP

I understand that, but how does this go against the idea of language specification?

LostKobrakai

LostKobrakai

Shouldn’t it be the other way round? Beam code being the standard (which has defined specs for erts to follow) and elixir just like other beam languages is a consumer of that spec?

D4no0

D4no0

This most probably will never happen, the only exception is lumen and who knows how many years it is from completion.

The other thing is that there are talks about elixir 2.0, so there might be a lot of breaking changes.

In general I think that having specifications is a good thing, however I don’t see how the effort invested in making a specification for elixir will be paid off.

LostKobrakai

LostKobrakai

Where did you get that impression? Yes there’s things queued for an eventual 2.0, but afaik there are no plans at all to act on those at this point.

cmo

cmo

Elixir 2.0 is coming out soon :tm:

dimitarvp

dimitarvp

I just don’t see an already overworked core team working on this ever. And the community, while very practical and of high quality, is not big enough or with enough free bandwidth to tackle this. We seem to mostly address functional stuff that helps us do our jobs. Not too much foundational work (exceptions exist of course, but they kind of prove the rule).

I do agree with the rationale btw, I am just not seeing this idea happen.

One thing I’d explore if I had all the time and energy in the world: an alternative Beam VM implementation without hot code reloading. Maybe skipping that will allow for both more performance optimizations and a way to spec the whole thing better. Maybe.

…Or maybe we can collaboratively agree on e.g. AST format for code-generating libraries as a start. The value of that is up in the air though.

hst337

hst337 OP

It works in both ways. ERTS developers are interested in optimizing runtime for frequently used patterns, and Elixir developers are interested in generating efficient code. The specification would help to define what kind of features Elixir relies the most and what features are not that necessary.


There is also Tria, Eir project, and a lot of companies are interested in development of statically typed elixir compilers (@josevalim gave several talks about it)


It is neither scheduled, nor expected to happen yet.


It is actually a good point, thanks. I think that there are several approaches to this problem. As I’ve said, it is not required to have 100% complete spec of the language. And I think that it should not be the job of core team to gather specification requirements, specification format, specification implementation and the specification of language itself

I would suggest something like this:

  • Specification proposals are gathered in terms of RFC which can be accepted, declined or can have an update requested. Core team should not be responsible for writing these RFCs.
  • One time a year (perhaps synchronously with Elixir release) all RFC must be accepted, declined or must have an update requested. The decision to accept or decline RFC can be the only job core team must do.
  • Accepted RFCs and the current specification will form the new version of language specification
D4no0

D4no0

Yeah like this talk, and the conclusion of that talk is that having static types that are solved at compile-time don’t solve much and why elixir will never be typed like languages such as java.

This seems to be a big overstatement, but it is no news that people want their classic OOP language, either because they are too lazy to learn a new paradigm or because that is the only way they know how to architect and write software, a good example is this discussion.

Where Next? Top

Trending in Proposals: Ideas Top

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New

We're in Beta

About us Mission Statement