DarkKernel - elixir functions which use dark magic

Dark Kernel

Link here: GitHub - tmbb/elixir_dark_kernel: Elixir functions that use dark magic
(not available on hex yet)

A couple years ago Andrea Leopardi and Chris Meyer created some pretty cool libraries to add ES6-style shorthand maps. In elixir notation, one would be able to write somthing like %{a, b, c} which would be equivalent to %{a: a, b: b, c: c}. It turns out the libraries didn’t get much use. I’ve always thought that they were very intersting libraries and came up with the idea to extend those concepts to keyword lists.

This library is the result of those ideas. It is very simple, and it was coded in an afternoon, but I think it has some pretty cool capabilities. The library embeds the source of ShorterMaps, which I plan to extend in the near future. @OvermindDL1 might enjoy this.

Despite having named this library the β€œDark Kernel”, I actually think it could be useful in real-world projects. It does have some β€œmagic” (it calls Code.string_to_quoted!/1 at least once), but all (dark!) magic works at compile time, and the macroexpanded code is not that complex.

Dark keyword lists

Meet dark keyword lists, the simplest form:

import DarkKernel

opts = [a: 1, b: 2, c: 3]
# Pattern match the keyword list against an existing variable
~k[a, b, c, d = opts]

assert a == 1
assert b == 2
assert c == 3
assert d == nil

Dark keyword lists with constant defaults:

import DarkKernel

opts = [a: 1, b: 2]
# Pattern match the keyword list against an existing variable
~k[a, b, c: "a string", d: :an_atom, e: 42, f = opts]

assert a == 1
assert b == 2
assert c == "a string"
assert d == :an_atom
assert e == 42
assert f == nil

Dark keyword lists with defaults which are arbitrary expressions:

import DarkKernel

opts = [a: 1, b: 2]
# Pattern match the keyword list against an existing variable
~k[a, b, c: 1 + 2, d: :rand.uniform(), e: 3.14 + :rand.uniform() = opts]

assert a == 1
assert b == 2
assert c == 3
assert is_float(d)
assert is_float(e)

Instead of returning nil for missing keys, we can raise an error. To do it, just prefix the required key with a bang (!):

iex(1)> import DarkKernel
DarkKernel

iex(2)> opts = [a: 1, b: 2, c: 3]
[a: 1, b: 2, c: 3]

# This will bind the variable to `nil`
iex(3)> ~k[a, b, x = opts]
[a: 1, b: 2, x: nil]
iex(4)> x                   
nil

# This will raise an error
iex(5)> ~k[a, b, !y = opts]
** (KeyError) key :y not found in: [a: 1, b: 2, c: 3]
    (elixir 1.14.1) lib/keyword.ex:595: Keyword.fetch!/2
    iex:5: (file)
iex(5)>

Dark maps

Dark maps are currently implemented using ShorterMaps. Due to dificulties of using a macro inside another macro, and because I wanted all sigils to be importable from the same module, dark_kernel embeds the source code of ShorterMaps.

In the (near) future, dark_kernel will support more advanced pattern matching capabilities for dark maps, similar to the ones supported by dark keyword lists.

iex(1)> import DarkKernel
DarkKernel
iex(2)> ~M{a, b, c} = %{a: 1, b: 2, c: 3}
%{a: 1, b: 2, c: 3}
iex(3)> a
1
iex(4)> b
2
iex(5)> c
3
iex(6)>
3 Likes

Some initial reactions to this syntax:

  • the = inside the brackets feels strange, but that may be because it makes me think of Ruby’s default argument syntax.

  • having a d in the β€œpattern” but then not failing to match feels odd compared to matching the literal forms where those keys are required to match

A less vibes-based issue I encountered trying this out is that it fails if a default expression calls a function with two arguments:

iex(3)> opts = [a: 1, b: 2]
[a: 1, b: 2]
iex(4)> ~k{a, b, c: Integer.pow(2, 4) = opts}
** (DarkKernel.DarkKeywords.InvalidKeywordError) Invalid keyword: 4)
Keywords must be valid Elixir variable names and contain only ASCII characters.
They can be preceded by a bang (e.g. `!key`) or contain a default value (e.g. `key: default`).

    (dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:116: anonymous fn/2 in DarkKernel.DarkKeywords.parse_keyword_match/1
    (elixir 1.14.0) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:103: DarkKernel.DarkKeywords.parse_keyword_match/1
    (dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:193: DarkKernel.DarkKeywords.do_match_keywords/1
    (dark_kernel 0.1.0) expanding macro: DarkKernel.sigil_k/2
    iex:4: (file)

or if a default argument is a literal with a comma in it:

iex(4)> ~k{a, b, c: "Hello, world!" = opts}
** (DarkKernel.DarkKeywords.InvalidKeywordError) Invalid keyword: world!"
Keywords must be valid Elixir variable names and contain only ASCII characters.
They can be preceded by a bang (e.g. `!key`) or contain a default value (e.g. `key: default`).

    (dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:116: anonymous fn/2 in DarkKernel.DarkKeywords.parse_keyword_match/1
    (elixir 1.14.0) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:103: DarkKernel.DarkKeywords.parse_keyword_match/1
    (dark_kernel 0.1.0) lib/dark_kernel/dark_keywords.ex:193: DarkKernel.DarkKeywords.do_match_keywords/1
    (dark_kernel 0.1.0) expanding macro: DarkKernel.sigil_k/2
    iex:4: (file)

In at least these two cases, Code.string_to_quoted! could be helpful if you wrap the whole expression passed to ~k in [ ]:

iex(5)> Code.string_to_quoted!("[a, b, c: \"Hello, world!\" = opts]")
[
  {:a, [line: 1], nil},
  {:b, [line: 1], nil},
  {:c, {:=, [line: 1], ["Hello, world!", {:opts, [line: 1], nil}]}}
]

iex(6)> Code.string_to_quoted!("[a, b, c: Integer.pow(2, 4) = opts]")
[
  {:a, [line: 1], nil},
  {:b, [line: 1], nil},
  {:c,
   {:=, [line: 1],
    [
      {{:., [line: 1], [{:__aliases__, [line: 1], [:Integer]}, :pow]},
       [line: 1], [2, 4]},
      {:opts, [line: 1], nil}
    ]}}
]

A little (ok, probably LOTS) of pattern matching should be able to split / transform these into the output AST. It could even support cases like two equals signs that currently raise an exception:

iex(8)> Code.string_to_quoted!("[!a, b: (_ = 1 + 2) = opts]")      
[
  {:!, [line: 1], [{:a, [line: 1], nil}]},
  {:b,
   {:=, [line: 1],
    [
      {:=, [line: 1], [{:_, [line: 1], nil}, {:+, [line: 1], [1, 2]}]},
      {:opts, [line: 1], nil}
    ]}}
]
3 Likes

Thanks for the feedback!

True, I think that’s pretty strange too, but it’s still the most succint way I’ve found support this kind of matches with a sigil. For very good reasons, the sigil can’t influence what’s outside the sigil! This means that all β€œmagic” syntax must happen inside the sigil.

Ideally, it should be something like this:

~k[a, b, c, d] = opts

but I don’t think I can make this work with Elixir’s macro system. I could use one of the undefined operators for something like this:

~k[a, b, c, d] <~ opts

What do you think?

Ok, maybe a match is not the right word for this. The goal of these keyword β€œmatches” is actually to extract variables from keyword lists. I’ll think of a better name/better syntax

You’re probably right. This library kinda grew up β€œorganically”, in that I started with something very simple based on regexes (actually not even regexes, I just used String.split()) and then I started adding to it. But I should look into implementing it with Code.string_to_quoted!/1.

1 Like

Ok, to my great surprise, both:

[a, !b, c: default = opts]

and

[a, !b, c = opts]

are valid elixir syntax.

This means that using Code.string_to_quoted! will work to parse ~k[] expressions. Now the only question is whether I should keep the equals sign inside the sigil or move it outside with something like ~k[a, b, c, d] <~ opts. I guess I might prefer the ~k[a, b, c, d = opts] because it suggests we’re actually using a new magic syntax instead of importing an operator out of the blue… Also, the Elixir assignment operator is = and not <~, so if we’re to extend the syntax, I guess we should try to keep the =.

It’s a list with three elements, where you’re matching opts on the last element. Which is why ~k[a, b, c, d = opts] that matches the list, not d on opts is rather odd syntax for Elixir.

One option you may consider is using <-. You should be able to do this inside the sigil without Elixir disliking it; and it’s kind of the language standard operator for enumerable on the right goes into left in some unique way in this context (see: comprehensions, with statements).

Yes, I understand what Elixir thinks that syntax is. It just surprises me that it’s valid. Elixir’s syntax is quite quirky after all. The newest version uses Code.string_to_quoted!/1 to parse the list and operates on that.

That’s a nice idea. It took me about 15s to add support for <- as well as = inside the ~k[...] expression. You can now do this:

opts = [a: 1, b: 2, c: 3]

assert Keyword.keys(~k[a, b <- opts]) == [:a, :b]
assert Keyword.keys(~k[a, b = opts]) == [:a, :b]

I rather like the <- syntax. Which one do you prefer:

# Option 1)
~k[a, b <- opts]

or

# Option 2)
~k[a, b] <~ opts

I think I prefer the first one.

I’d agree with @cmo:

So you prefer the <~ operator outside, right? like this: ~k[a, b, c] <~ opts

Yep, personally I would!

1 Like

I’ve already implemented that as an operator. When debugging is set to active it prints the generated code.

iex(43)> import DarkKernel
DarkKernel
iex(44)> ~k[a, b, c] <~ [b: 1, c: 2, a: 3]
[a: 3, b: 1, c: 2]
iex(45)>
18:32:00.996 [debug] iex:44
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β€· Code:
β”œβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Ά
β”‚ ~k[a, b, c] <~ [b: 1, c: 2, a: 3]
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β€· Generated code:
β”œβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Ά
β”‚ constant_kw_list = [b: 1, c: 2, a: 3]
β”‚ a = Keyword.get(constant_kw_list, :a, nil)
β”‚ b = Keyword.get(constant_kw_list, :b, nil)
β”‚ c = Keyword.get(constant_kw_list, :c, nil)
β”‚ [a: a, b: b, c: c]
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β€· Generated AST:
β”œβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Άβ•Ά
β”‚ {:__block__, [],
β”‚  [
β”‚    {:=, [],
β”‚     [
β”‚       {:constant_kw_list, [counter: -576460752303417279],
β”‚        DarkKernel.DarkKeywords},
β”‚       [b: 1, c: 2, a: 3]
β”‚     ]},
β”‚    {:=, [],
β”‚     [
β”‚       {:a, [], nil},
β”‚       {{:., [], [{:__aliases__, [alias: false], [:Keyword]}, :get]}, [],
β”‚        [
β”‚          {:constant_kw_list, [counter: -576460752303417279],
β”‚           DarkKernel.DarkKeywords},
β”‚          :a,
β”‚          nil
β”‚        ]}
β”‚     ]},
β”‚    {:=, [],
β”‚     [
β”‚       {:b, [], nil},
β”‚       {{:., [], [{:__aliases__, [alias: false], [:Keyword]}, :get]}, [],
β”‚        [
β”‚          {:constant_kw_list, [counter: -576460752303417279],
β”‚           DarkKernel.DarkKeywords},
β”‚          :b,
β”‚          nil
β”‚        ]}
β”‚     ]},
β”‚    {:=, [],
β”‚     [
β”‚       {:c, [], nil},
β”‚       {{:., [], [{:__aliases__, [alias: false], [:Keyword]}, :get]}, [],
β”‚        [
β”‚          {:constant_kw_list, [counter: -576460752303417279],
β”‚           DarkKernel.DarkKeywords},
β”‚          :c,
β”‚          nil
β”‚        ]}
β”‚     ]},
β”‚    [a: {:a, [], nil}, b: {:b, [], nil}, c: {:c, [], nil}]
β”‚  ]}
└────────────────────────────────────────────────────────────────────────────────────────────────────


nil
iex(46)> a
3
iex(47)> b
1
iex(48)> c
2
iex(49)>
1 Like