# Dark Kernel

Link here: GitHub - tmbb/elixir_dark_kernel: Elixir functions that use dark magic
(not available on hex yet)

A couple years ago Andrea Leopardi and Chris Meyer created some pretty cool libraries to add ES6-style shorthand maps. In elixir notation, one would be able to write somthing like `%{a, b, c}` which would be equivalent to `%{a: a, b: b, c: c}`. It turns out the libraries didnβt get much use. Iβve always thought that they were very intersting libraries and came up with the idea to extend those concepts to keyword lists.

This library is the result of those ideas. It is very simple, and it was coded in an afternoon, but I think it has some pretty cool capabilities. The library embeds the source of ShorterMaps, which I plan to extend in the near future. @OvermindDL1 might enjoy this.

Despite having named this library the βDark Kernelβ, I actually think it could be useful in real-world projects. It does have some βmagicβ (it calls `Code.string_to_quoted!/1` at least once), but all (dark!) magic works at compile time, and the macroexpanded code is not that complex.

## Dark keyword lists

Meet dark keyword lists, the simplest form:

``````import DarkKernel

opts = [a: 1, b: 2, c: 3]
# Pattern match the keyword list against an existing variable
~k[a, b, c, d = opts]

assert a == 1
assert b == 2
assert c == 3
assert d == nil
``````

Dark keyword lists with constant defaults:

``````import DarkKernel

opts = [a: 1, b: 2]
# Pattern match the keyword list against an existing variable
~k[a, b, c: "a string", d: :an_atom, e: 42, f = opts]

assert a == 1
assert b == 2
assert c == "a string"
assert d == :an_atom
assert e == 42
assert f == nil
``````

Dark keyword lists with defaults which are arbitrary expressions:

``````import DarkKernel

opts = [a: 1, b: 2]
# Pattern match the keyword list against an existing variable
~k[a, b, c: 1 + 2, d: :rand.uniform(), e: 3.14 + :rand.uniform() = opts]

assert a == 1
assert b == 2
assert c == 3
assert is_float(d)
assert is_float(e)
``````

Instead of returning `nil` for missing keys, we can raise an error. To do it, just prefix the required key with a bang (!):

``````iex(1)> import DarkKernel
DarkKernel

iex(2)> opts = [a: 1, b: 2, c: 3]
[a: 1, b: 2, c: 3]

# This will bind the variable to `nil`
iex(3)> ~k[a, b, x = opts]
[a: 1, b: 2, x: nil]
iex(4)> x
nil

# This will raise an error
iex(5)> ~k[a, b, !y = opts]
** (KeyError) key :y not found in: [a: 1, b: 2, c: 3]
(elixir 1.14.1) lib/keyword.ex:595: Keyword.fetch!/2
iex:5: (file)
iex(5)>
``````

## Dark maps

Dark maps are currently implemented using ShorterMaps. Due to dificulties of using a macro inside another macro, and because I wanted all sigils to be importable from the same module, `dark_kernel` embeds the source code of `ShorterMaps`.

In the (near) future, `dark_kernel` will support more advanced pattern matching capabilities for dark maps, similar to the ones supported by dark keyword lists.

``````iex(1)> import DarkKernel
DarkKernel
iex(2)> ~M{a, b, c} = %{a: 1, b: 2, c: 3}
%{a: 1, b: 2, c: 3}
iex(3)> a
1
iex(4)> b
2
iex(5)> c
3
iex(6)>
``````
3 Likes

Some initial reactions to this syntax:

• the `=` inside the brackets feels strange, but that may be because it makes me think of Rubyβs default argument syntax.

• having a `d` in the βpatternβ but then not failing to match feels odd compared to matching the literal forms where those keys are required to match

A less vibes-based issue I encountered trying this out is that it fails if a default expression calls a function with two arguments:

``````iex(3)> opts = [a: 1, b: 2]
[a: 1, b: 2]
iex(4)> ~k{a, b, c: Integer.pow(2, 4) = opts}
** (DarkKernel.DarkKeywords.InvalidKeywordError) Invalid keyword: 4)
Keywords must be valid Elixir variable names and contain only ASCII characters.
They can be preceded by a bang (e.g. `!key`) or contain a default value (e.g. `key: default`).

(dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:116: anonymous fn/2 in DarkKernel.DarkKeywords.parse_keyword_match/1
(elixir 1.14.0) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
(dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:103: DarkKernel.DarkKeywords.parse_keyword_match/1
(dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:193: DarkKernel.DarkKeywords.do_match_keywords/1
(dark_kernel 0.1.0) expanding macro: DarkKernel.sigil_k/2
iex:4: (file)
``````

or if a default argument is a literal with a comma in it:

``````iex(4)> ~k{a, b, c: "Hello, world!" = opts}
** (DarkKernel.DarkKeywords.InvalidKeywordError) Invalid keyword: world!"
Keywords must be valid Elixir variable names and contain only ASCII characters.
They can be preceded by a bang (e.g. `!key`) or contain a default value (e.g. `key: default`).

(dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:116: anonymous fn/2 in DarkKernel.DarkKeywords.parse_keyword_match/1
(elixir 1.14.0) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
(dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:103: DarkKernel.DarkKeywords.parse_keyword_match/1
(dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:193: DarkKernel.DarkKeywords.do_match_keywords/1
(dark_kernel 0.1.0) expanding macro: DarkKernel.sigil_k/2
iex:4: (file)
``````

In at least these two cases, `Code.string_to_quoted!` could be helpful if you wrap the whole expression passed to `~k` in `[ ]`:

``````iex(5)> Code.string_to_quoted!("[a, b, c: \"Hello, world!\" = opts]")
[
{:a, [line: 1], nil},
{:b, [line: 1], nil},
{:c, {:=, [line: 1], ["Hello, world!", {:opts, [line: 1], nil}]}}
]

iex(6)> Code.string_to_quoted!("[a, b, c: Integer.pow(2, 4) = opts]")
[
{:a, [line: 1], nil},
{:b, [line: 1], nil},
{:c,
{:=, [line: 1],
[
{{:., [line: 1], [{:__aliases__, [line: 1], [:Integer]}, :pow]},
[line: 1], [2, 4]},
{:opts, [line: 1], nil}
]}}
]
``````

A little (ok, probably LOTS) of pattern matching should be able to split / transform these into the output AST. It could even support cases like two equals signs that currently raise an exception:

``````iex(8)> Code.string_to_quoted!("[!a, b: (_ = 1 + 2) = opts]")
[
{:!, [line: 1], [{:a, [line: 1], nil}]},
{:b,
{:=, [line: 1],
[
{:=, [line: 1], [{:_, [line: 1], nil}, {:+, [line: 1], [1, 2]}]},
{:opts, [line: 1], nil}
]}}
]
``````
3 Likes

Thanks for the feedback!

True, I think thatβs pretty strange too, but itβs still the most succint way Iβve found support this kind of matches with a sigil. For very good reasons, the sigil canβt influence whatβs outside the sigil! This means that all βmagicβ syntax must happen inside the sigil.

Ideally, it should be something like this:

``````~k[a, b, c, d] = opts
``````

but I donβt think I can make this work with Elixirβs macro system. I could use one of the undefined operators for something like this:

``````~k[a, b, c, d] <~ opts
``````

What do you think?

Ok, maybe a match is not the right word for this. The goal of these keyword βmatchesβ is actually to extract variables from keyword lists. Iβll think of a better name/better syntax

Youβre probably right. This library kinda grew up βorganicallyβ, in that I started with something very simple based on regexes (actually not even regexes, I just used `String.split()`) and then I started adding to it. But I should look into implementing it with `Code.string_to_quoted!/1`.

1 Like

Ok, to my great surprise, both:

``````[a, !b, c: default = opts]
``````

and

``````[a, !b, c = opts]
``````

are valid elixir syntax.

This means that using `Code.string_to_quoted!` will work to parse `~k[]` expressions. Now the only question is whether I should keep the equals sign inside the sigil or move it outside with something like `~k[a, b, c, d] <~ opts`. I guess I might prefer the `~k[a, b, c, d = opts]` because it suggests weβre actually using a new magic syntax instead of importing an operator out of the blueβ¦ Also, the Elixir assignment operator is `=` and not `<~`, so if weβre to extend the syntax, I guess we should try to keep the `=`.

Itβs a list with three elements, where youβre matching `opts` on the last element. Which is why `~k[a, b, c, d = opts]` that matches the list, not `d` on `opts` is rather odd syntax for Elixir.

One option you may consider is using `<-`. You should be able to do this inside the sigil without Elixir disliking it; and itβs kind of the language standard operator for enumerable on the right goes into left in some unique way in this context (see: comprehensions, with statements).

Yes, I understand what Elixir thinks that syntax is. It just surprises me that itβs valid. Elixirβs syntax is quite quirky after all. The newest version uses `Code.string_to_quoted!/1` to parse the list and operates on that.

Thatβs a nice idea. It took me about 15s to add support for `<-` as well as `=` inside the `~k[...]` expression. You can now do this:

``````opts = [a: 1, b: 2, c: 3]

assert Keyword.keys(~k[a, b <- opts]) == [:a, :b]
assert Keyword.keys(~k[a, b = opts]) == [:a, :b]
``````

I rather like the `<-` syntax. Which one do you prefer:

``````# Option 1)
~k[a, b <- opts]
``````

or

``````# Option 2)
~k[a, b] <~ opts
``````

I think I prefer the first one.

Iβd agree with @cmo:

So you prefer the `<~` operator outside, right? like this: `~k[a, b, c] <~ opts`

Yep, personally I would!

1 Like

Iβve already implemented that as an operator. When debugging is set to active it prints the generated code.

``````iex(43)> import DarkKernel
DarkKernel
iex(44)> ~k[a, b, c] <~ [b: 1, c: 2, a: 3]
[a: 3, b: 1, c: 2]
iex(45)>
18:32:00.996 [debug] iex:44
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β€· Code:
ββΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆ
β ~k[a, b, c] <~ [b: 1, c: 2, a: 3]
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β€· Generated code:
ββΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆ
β constant_kw_list = [b: 1, c: 2, a: 3]
β a = Keyword.get(constant_kw_list, :a, nil)
β b = Keyword.get(constant_kw_list, :b, nil)
β c = Keyword.get(constant_kw_list, :c, nil)
β [a: a, b: b, c: c]
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β€· Generated AST:
ββΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆβΆ
β {:__block__, [],
β  [
β    {:=, [],
β     [
β       {:constant_kw_list, [counter: -576460752303417279],
β        DarkKernel.DarkKeywords},
β       [b: 1, c: 2, a: 3]
β     ]},
β    {:=, [],
β     [
β       {:a, [], nil},
β       {{:., [], [{:__aliases__, [alias: false], [:Keyword]}, :get]}, [],
β        [
β          {:constant_kw_list, [counter: -576460752303417279],
β           DarkKernel.DarkKeywords},
β          :a,
β          nil
β        ]}
β     ]},
β    {:=, [],
β     [
β       {:b, [], nil},
β       {{:., [], [{:__aliases__, [alias: false], [:Keyword]}, :get]}, [],
β        [
β          {:constant_kw_list, [counter: -576460752303417279],
β           DarkKernel.DarkKeywords},
β          :b,
β          nil
β        ]}
β     ]},
β    {:=, [],
β     [
β       {:c, [], nil},
β       {{:., [], [{:__aliases__, [alias: false], [:Keyword]}, :get]}, [],
β        [
β          {:constant_kw_list, [counter: -576460752303417279],
β           DarkKernel.DarkKeywords},
β          :c,
β          nil
β        ]}
β     ]},
β    [a: {:a, [], nil}, b: {:b, [], nil}, c: {:c, [], nil}]
β  ]}
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ

nil
iex(46)> a
3
iex(47)> b
1
iex(48)> c
2
iex(49)>
``````
1 Like