Oh I just caught that myself too! That is brilliant. ^.^
EDIT:
Vwoop:
defmacro defml(opts) when is_list(opts) do
case opts[:do] do
nil -> defml_impl(opts, [])
ast -> defml_impl(ast, opts)
end
end
I can probably live with requiring parenthesis around anonymousall function definitions, though would be nice if an infix operator like => (much as I stylistically dislike fat arrows) would work like normal infix operators (-> is really weird… not like a normal operator…).
EDIT: What no strikethrough markdown operator on this forum? o.O
EDIT: For note, I’m trying to translate a mini-language I made into Elixir’s AST to use straight in Elixir (right now it parses out text into elixir ast, which makes syntax coloring really bad when used like defml "let a = 42 in a", although it works, just… ugly, we need read-macro’s in elixir ^.^), where its function syntax is:
let myAdder = fun
| (a:int) (b:int b>=0) -> a + b
| (a:int) (b:int) -> a - b
| (a:binary) (b:binary) -> String.concat [a, b]
| a b -> error "unsupported"
in myAdder 10 (-4)
Types are optional if it can be inferred, though fully typed it would look more like:
let myAdder = fun
| (a : int) (b : int b>=0) : int a+b -> a + b (* Since a has a valid range of infinite here, the return type is infinite+{b's range, which is {0, infinite}}, which is still infinite regardless, it is not that smart yet *)
| (a : int) (b : int) : int a-b -> a - b (* int's extra information holds possible values allowed, simple math allowed on those values *)
| (a : string) (b : string) : string -> String.concat [a, b] (* string's can refine on length only, currently *)
| (a : any) (b : any) -> error "unsupported" (* any's do not have any refinement yet *)
in let curried = myAdder "Hello " (* A new erlang anonymous function is made here that curries myAdder, curried things and closures cannot be made into top-level module functions in this play language (type error) *)
in curried "world" (* Now I call it *)
Reading this thread has been such an emotional rollercoaster.
To sum up, -> is only allowed between do/end, fn/end and (/). But you need to be careful because the parens need to apply to the -> and not arguments. The reason why foo(fun x -> x) does not work becomes clearer if you add multiple arguments. If you have foo(fun x, y -> x), does it mean foo(fun x, (y -> x)) or foo((fun x, y -> x))?
The reason I wrote the document linked by Eric is exactly because we wanted to show it is less rules than most would expect. Especially because we don’t need to specify the rules for keywords like case, def, defmodule, receive, if, try, etc.
Heh, sorry, I never meant for it to be as such, just fighting AST (I’m more used to tokanization macros rather than ast macros ^.^) by purposefully seeing how much I can subvert Elixir macro’s to do things not originally intended. MLElixir is not the end-goal by far, rather something else is, a working hint:
# The extra type is to give a name to the type in simple, so the input and output become the same type.
# If the spec was `simple(any()) :: any()` then you could not state that the output type is based on the input type.
defmodulet TypedTest_Typed_Identity do
@type identity_type :: any()
@spec simple(identity_type) :: identity_type
def identity(x), do: x
end
Making an inference algorithm on something I’m more familiar with using such algorithms on (ML style AST’s) let me build it up and work out kinks. ^.^
The string parser version I made worked pretty well but I made some bad design choices in it, wanted to change it up a bit but I did not want to re-parse either (plus I was curious in the challenge of making it work in a macro), and this one ended up pretty clean, just needed to make one more design change (already done, just not in ‘it’) and have it working in the main project now. MLElixir is never designed to be used, just what I am going to play in. ^.^
All things considered, I was able to get a typed system with surprisingly few AST nodes, I’m half-tempted to parse out Elixir from text and build it out from the base level (plus with my new parser that is not based on leex/yecc in any form holds column information, so I can report significantly more accurate errors ^.^).
But yeah, foo(fun x, y -> x) is entirely ambiguous if -> were an operator, I’d have opted for foo(fun x y -> x) instead, or to have the fun support a block probably foo(fun x y -> do x end) and so forth, with matching semantics working like foo(fun {x, _} y when x>10 -> x) or so. I am exceedingly biased on getting rid of commas though as I consider them entirely superfluous with a decent syntax and they can easily get lost in the noise. The above example I’d personally want to write it as foo (fun x y -> x) where () surround arguments instead of the call as it makes them gone most of the time and keeps the block in-place, take for example some calls that I’ve seen ‘in-the-wild’ like blah = something(var1 + var2, {var3, var4, var5}, var6.var7, var8+var9), if read in a more ML’y language it would be blah = something (var1 + var2) {var3, var4, var5} var6.var7 (var8+var9), which I find more readable (coincedentally it also works in Elixir if not being piped or something), however that is still syntax and not really the purpose behind what I’m trying to do anyway. ^.^
I guess I mostly just expected operator-looking constructs to exist like operators in the AST, it just ended up being very different, which still feels like a special case. I’m guessing Ruby has all these odd constructs, and I did not come from the Ruby world (though to be fair I primarily came from the horror that is C++, no other language has such a sizable Spec… and I know it better than I probably wish, this might also contribute to my like of the simple syntax’s of functional languages that have such a simple syntax like Erlang and ML ^.^), so there are a lot of things assumed in that world that I do not have.
And yeah, I know dialyzer catches most of this already, but I accept more information than it that can soon be specified via other attributes, it is still mostly just for the challenge though, although this part may actually be something I use in production as it only compiles or not without changing any syntax beyond defmodule to defmodulet based on what it scans (it can pull type information from other modules decorated in the same way, I might have it parse BEAM files later as I originally tried to have it do but that is too much of a pain for a first version, so explicit decorations can be done instead, which is fine for me, plus it has more options this way :-)).
Tokanization macros could do the same, but without needing to special case things like -> or so although, just for convenience special casing all block forms would save the most effort in writing them, but even that would be optional at that step.
Like something like this to implement if via case:
# This would be called when an identifier token with `:if` was reached, and would consume the
# token stream until it reaches the end or it exits early (the end would be the end of the block
# that it was called within), assuming a 'block' form returned a map with the internal tokens of
# the usual `do`, `else`, etc... that is supported now.
deftokenmacro if(tokens) do
# The :if should already be eaten by this point, but if it was not you could do it by:
# [:if | tokens] = tokens
{tokens, condition_ast} = AST.parse_expression(tokens) # returns the expression_ast and the remaining untouched tokens
{tokens, clauses} = AST.parse_blocks(tokens, do: true, else: false) # Only support the do and else forms in a block here, do is required, else is optional, returns a map of the AST's
do_ast = clauses.do
else_ast = clauses[:else] # `nil` will be returned regardless if empty
if_ast = quote do # Quoting things could even run through the tokenmacro's too
case unquote(condition_ast) do
value when value in [false, nil] -> unquote(else_ast)
_ -> unquote(do_ast)
end
end
{tokens, if_ast} # Return un-eaten tokens and the ast we generated
end
Like say the case was handled in the ast for matchers and the AST is in the form that Elixir has now, you could make a tokenmacro parse that kind of like (although you’d probably want case built-in to the kernel instead of defined in the language, so it could be used in the earliest of deftokenmacros ^.^):
# `case` would probably be defined internally, but an example by using only function heads
# without using `case` at all:
deftokenmaco case(tokens) do
{tokens, value_ast} = AST.parse_expression(tokens) # You'd probably be passing an environment around somehow too to be honest...
{tokens, blocks} = AST.parse_blocks(tokens, &case_block_parser/2, do: true) # You could even make an `else` block easily to handle non-matches ^.^
{:case, [], [
value_ast,
[do: blocks.do]
]}
end
# defp case_block_parser(block_name, tokens) # The callback format
defp case_block_parser(:do, tokens, astSoFar \\ [])
defp case_block_parser(:do, [], astSoFar) do # successfuly used every token in this block, or the next head would have already errored
astSoFar
end
defp case_block_parser(:do, tokens, astSoFar) do
{tokens, matcher_ast} = AST.parse_matchspec(tokens)
# [:-> | tokens] = tokens # Probably want to report a decent error if missing though, a helper could be like:
tokens = Tokens.requireNextToken(tokens, :->, "Matchspecs must be followed by a `->`")
{tokens, body_ast} = AST.parse_expression(tokens) # Which could be a block, a block is just a set of expressions that returns the last one, but is delimited by do/end or (/) or so
ast = {:->, [], [matchspec_ast, body_ast]
case_block_parser(:do, tokens, astSoFar ++ [ast])
end
This version of case would not be parsed like Elixir’s is now (with special whitespace rules and such, though I guess you could add that by attempting to parse a matchspec followed by a :-> at each step and if it fails then parse another expression into the block), it would be more like this:
case blah do # A `do` is absolutely required in this version, not that it isn't already ^.^
x when is_int(x) -> x+42 # Single expression requires no block
x when is_binary(x) -> do # Multi-expression requires a block, whether do/end or (/) or whatever, I so love explicit blocks, implicit ones are harder to reason ^.^
x = String.trim(x)
"Test: " <> x
x -> x
end
And I could have easily made a version that supported this:
case blah do # A `do` is absolutely required in this version, not that it isn't already ^.^
x when is_int(x) -> x+42 # Single expression requires no block
x when is_binary(x) -> do # Multi-expression requires a block, whether do/end or (/) or whatever
x = String.trim(x)
"Test: " <> x
else
blah
end
Or made a version that supported this more different format:
case blah do # A `do` is absolutely required in this version, not that it isn't already ^.^
| x when is_int(x) -> x+42 # Make all cases delimited by `|`, first one could even be optional
| x when is_binary(x) -> # Multi-expression is fine, still delimited by `|` so just parse tokens until a `|` or the end of this block is reached at `end`
x = String.trim(x)
"Test: " <> x
| x -> x
end
That last one would be especially easy if certain tokens were not overridable like the block constructs do/end and (/) and maybe even others (though keeping it small is good) then the tokenizer could be block-aware and make parsing the tokens almost trivially easy. But then you could build entirely new forms of constructs while keeping things well scoped.
Even the case expression in Elixir now you have to separate cases with a ;, which is such an odd construct to use as ; is usually used to separate expressions, not cases (I’d have chosen | and put it in front of each case):
case condition do blah when is_int(blah) -> a=2; blah+a; bleep -> bleep end
# or in long form
case condition do
blah when is_int(blah) ->
a=2
blah+a
bleep -> bleep
end
# Compare to:
case condition do blah when is_int(blah) -> a=2; blah+a | bleep -> bleep end
# or in long-form:
case condition do # The first `|` is optional regardless, even here, but consistency...
| blah when is_int(blah) ->
a=2
blah+a
| bleep -> bleep
end
Using a ; as a case separator suddenly makes this ambiguous, so you have to try parsing a match then a :-> to see if it is a new case after every ;, or you could just check for |, which I also think makes cases more readable (I even un-indented the body as an example, though you still could). ^.^
But yeah, as for deftokenmacro a base set (like case probably) would probably be implemented in Erlang for initial parse-time. ^.^
Blergh, this ended up much longer than I wanted, syntax is often on my mind since I teach it and I often have to poke at such things… >.>
Feel free to ignore the above (except maybe the defmodulet example showing normal but typed elixir if you are curious), I’m working on typing things, not syntax. ^.^
Yes, yes! I didn’t mean it as a negative thing. More like: “will he make it?!”.
The reason we didn’t use tokenization macros is because they are quite more powerful than macros, allowing anyone to create their own syntax for Elixir. It would be wonders for what you are doing but could make Elixir a really hard language, as totally new syntax constructs could be introduced any time.
That is why I had the idea for scoped tokanization macro’s.
Once scoped based on non-overridable block tokens keep the entire tokanization macro context entirely defined and unable to leak out, so it is true that anyone could create their own DSEL but only within a specific block, which would be fantastic for math, parsing, or anything else, while not ‘leaking’ out. It is not a concept I’ve seen languages with token-macro’s support yet but its an idea that I’ve had for a long time and I’ve been considering implementing it in one of my play languages (although the only block context my recent ML one has is parenthesis (/) so far, been holding off on adding the traditionally ML begin/end blocks ^.^).
I do have to say that I took the Elixir AST as an idea on how to store the AST, it is very uniform and easy to go over, which would work very well with elixir-style macro’s in there too (making PPX’s in OCaml, its version of macro’s, is… you need to know compiler guts… ^.^;).
For the elixir macro MLElixir test I’m building up modules in reverse order, I have tests for it to become like this:
defml \
let identity = (fun x -> x) in
let blah = 42 in
module {
identity: identity,
blah: blah,
{
# Or you can do it inline too...
defml module {
identity: (fun x -> x),
blah: 42,
}
And what it does is parse my internal function format to build up a def elixir AST instead of an anonymous function AST, and set values like blah above becomes a 0-arg function. But this will also create the function on the module named __ml_open__/0 that returns an information structure that is currently of the form of (this is how I pull information at compile-time from other modules):
%{
types: %{ # Types defined and exported from this module
t: <TypeDef>, # This is an exported type named `t` (traditional 'module' default name type for many ML languages
},
funs: %{
+: <AnonFunc/3> # Defines the `+` function
},
}
The anonymous function for a function is passed the parsing environment (not the elixir environment, but this holds the type information and more), the meta-structure of the call AST node (to do better error reporting), and a list of the argument AST’s, which contain the type information, so it can unify the types as it wants, match it to multiple function heads, etc… For + (since it is built-in) it mandates that the types of all the arguments are the same type and of type integer or float, while updating the refined information and so forth, that way doing 2+2 always returns an int type and 1.1+2.2 always returns a float type yet 1+2.2 errors out because it cannot unify them. If I ever advance ‘far enough’ I want to put OCaml-style first-class modules in the type system (as a value it is just an atom after all, so easy to pass around) while adding implicit module support (been planning for that to be added) that way + could be called on any user-defined type that implements a module to handle it that is in scope, like typeclasses (but more powerful).
For the typed elixir that I’ve been doing as well (the defmodulet macro) it just uses normal Elixir code, so I may have to invent some new module attributes to support such advanced constructs, but I left it open in the parser to do just that, however it should be possible regardless.
/me really loves statically typed languages, they catch so many errors that leak through otherwise that even dialyzer cannot catch…
As an aside, I am hoping for the basic functionality of defmodulet to be so that you just import TypedElixir, add the t to the end of defmodule (I do not like overriding internal names) and just type your existing code, although it will rightfully yell at you if you call a function that it does not know the type of outside the module so you may need to decorate them as well (just normal @spec’s, you can even put them in another file that you can import so no changes are needed to the original module, then just import the types).
Eyup, and I’ve been commenting in their issues and discussions since it was known by its previous name. ^.^
They’ve done a few things I’m not a fan of, like they are going the route of 1-arg functions only with automatic currying at the function head, where I’d prefer to do (to fit the EVM better) N-arg heads by default and instead auto-curry at the call sites. That is how my in-elixir tests have been, following that style.
I’m also very iffy about their ideas of typing PID’s/mailboxes, I’d personally prefer to black-box message types and force tests to happen on it to deduce what it is as that seems far more Erlang’y to me as well while also being significantly easier to reason about in the EVM’s architecture (anything could send you a message of anything after all).
Little more work, a large revamp of how the code works (more shared code with Typed Elixir whoo!). Module types are a thing now, and consequently so are accessing remote type, and thus also made a defmlmodule call, all of this works and dies if the types do not match as expected:
import MLElixir
defmlmodule MLModuleTest do
type type_declaration
type type_definition = integer
def test_int_untyped = 42
def test_int_typed | integer = 42
def test_float_untyped = 6.28
def test_float_typed | float = 6.28
def test_defined | type_definition = 42
def identity_untyped(x) = x
def identity_typed(x | +id_type) | +id_type = x
def identity_int(x | integer, _f | float) | integer = x
def identity_float(_x | integer, f | float) | float = f
def test_block0 do 42 end
def test_blockN0() do 42 end
def test_blockT0() | integer do 42 end
def test_blockN1(x) do x end
def test_blockT1(x | +id_type) | +id_type do x end
end
defmlmodule MLModuleTest_Generalized do
type t
def blah(t | t) | t, do: t
end
defmlmodule MLModuleTest_Specific do
type t = MLModuleTest.type_definition
type Specific = MLModuleTest_Generalized.(t: float)
def testering0 | t = 42
end
Parameterized types and overloadable types are done via .(blah) syntax. So to refine a type (say a ‘Dict’ module has a type ‘t’ that is generic, you can refine it to a specific type that fits within generic (anything) and use the module with that kind of type) you specify it with the standard proplist style with the key as the id, so in the examples above that is the type specific = MLModuleTest_Generalized.(t: float) call, it refines the t type on MLModuleTest_Generalized to make a more refined type. You can then call on Specific to access it but with the refined type, however I’m not quite happy with this syntax (not able to ‘pass’ a module around), so going to change it sometime when I get time so that modules can be packed as a value like in OCaml, pretty easy to do on the BEAM. ^.^
However, the same syntax will also be used to apply types as well, so given a type like:
type result(ok, error) = enum # still unsure what to call this type, variant is long...
| ok = ok
| error = error
You can refine it like result.(String.t, nil) or like result.(ok: String.t, error: nil) for explicitness. ^.^
The . (dot) calling convention is used for applying types, the normal (or left out) parenthesis is for function application. Still up in the air of course.
EDIT: Using ‘def’ because syntax coloring, ‘let’ is supported too, and it supports 'let’ish style and elixir do/block style.
Also, a type name like integer refers to a built-in or pre-named type, a type name prefixed with + like +id_type is a named generic.
