TaggedTupleShorthand - Field Punning in Elixir

christhekeele · December 23, 2024, 12:54am

Hey all!

I’m releasing TaggedTupleShorthand today as an experiment with Elixir syntax.

Field Punning

The aim of the library is to support a commonly-requested Elixir feature¹ ² ³ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ¹⁰ ¹¹ ¹² ¹³ ¹⁴ ¹⁵ ¹⁶ ¹⁷:

Short-hand for constructing, deconstructing, and pattern matching on key/value pairs in associative data structures, interacting with variable names in the current scope.

This is called “field punning”—essentially trying to turn this

def handle_in(
      event,
      %{
        "chat" => chat,
        "question_id" => question_id,
        "data" => data,
        "attachment" => attachment
      },
      socket
    )
    when is_binary(chat) do...

into this:

def handle_in(event, %{chat, question_id, data, attachment}, socket)
    when is_binary(chat) do...

This syntax sugar exists with many variations in many other languages. It is popular for several reasons:

This syntax saves on visual noise, expressing destructuring key/value data tersely in the common case of the key making for a sufficient variable name.
This syntax calls attention to the cases where we are intentionally not re-using the key as a variable name, placing emphasis on a subtle decision a developer decided was important for readability or understanding.
This syntax prevents common typos, and ensures that variable names match keys throughout refactors when that is the desired behaviour.

Generally speaking, Elixir’s pattern matching and destructuring functionality greatly surpasses other languages, so people are often surprised to discover that no syntax for field punning exists!

Why not?

Well, for a lot of reasons. I encourage you to explore discussions around previous proposals, but in my opinion it boils down to:

We have two different common associative data structures, Keyword lists and Maps
We have two different common key types, Atoms and Strings
We have two different common syntaxes for key/value associativity, arbitrary => value (maps only) and atom: value (atom keys only)
We have another common container type, Tuples, that look a lot like Maps, and many proposed syntaxes blur the line between them visually
It is hard to invent a syntax that spans all of these considerations consistently
Everyone has a different, incompatible preference as to which permutations of these considerations to support and which to ignore completely
Consensus is hard, syntax sugar is subjective, and a good compromise leaves everyone dissatisfied
Therefore, a working field punning proposal for the language that satisfies everyone has remained elusive

Anyways, that’s field punning…

...and now, for something completely different.

Tagged Tuple Shorthand

“Tagged Tuples” describe a pattern where you have a value, and want to “tag” it with information as to what it represents. In Elixir, this is generally accomplished with a two-tuple, where the first element of the tuple is an atom “tag”, and the second element is the value itself:

case try_to_procure_an_integer() do
  result when is_integer(result) -> {:ok, result}
  other ->
    reason = "not an integer: #{inspect other}"
    {:error, reason}
end

Here, {:ok, result} and {:error, reason} are the tagged tuples. This idiom is overwhelmingly common across the ecosystem; you’ve probably wrote similar Elixir code yourself before you learned the term for it.

What if there was a way of constructing, deconstructing, and pattern matching on tagged tuples that tied into variable names in the current scope? An operator for it, even?

Introducing `TaggedTupleShorthand.@/1`

Let’s devise an operator that takes either an atom or a string, and expands at compile-time to a tagged tuple, with a variable reference mirroring that atom/string as the second element. We’ll use the unary @ operator since it is already overridable via Elixir macros, and has the correct operator arity, precedence, and associativity for this use-case.

use TaggedTupleShorthand

foo = 1
@:foo # aka {:foo, foo}
#=> {:foo, 1}

bar = 2
@"bar" # aka {"bar", bar}
#=> {"bar", 2}

Essentially,

Form	Expands To
`@:atom`	`{:atom, atom}`
`@^:atom`	`{:atom, ^atom}`
`@"string"`	`{"string", string}`
`@^"string"`	`{"string", ^string}`
`@anything_else`	Fallback to `Kernel.@/1`

This is… certainly a thing? A novel bit of syntax, hard to acclimate to, but perhaps justifiable in a language with such widespread idiomatic usage of tagged two-tuples?

We could refactor the previous example by choosing our variable names carefully:

use TaggedTupleShorthand

case try_to_procure_an_integer() do
  ok when is_integer(ok) ->
    @:ok # {:ok, ok}
  other ->
    error = "not an integer: #{inspect other}"
    @:error # {:error, error}
end

We could even craft string-tagged tuples, for some reason, why not:

case try_to_procure_an_integer() do
  ok when is_integer(ok) ->
    @"ok" # {"ok", ok} <- nobody wants this
  other ->
    error = "not an integer: #{inspect other}"
    @"error" # {"error", error} <- get this right out of here
end

But why?

I’m sure you see where this is going.

Tagged Tuple Shorthand as Field Punning

It turns out, this simple TaggedTupleShorthand.@/1 macro is all we need to get field punning in Elixir, with expressive power inline with our robust pattern matching and exceeding other languages’ capabilities.

The Abstract Syntax Tree for Elixir—its AST that macros can operate on—already represents key/value pairs in map and keyword list literals as tagged tuples. That is, in any literal map or list we might want to use field punning in, we can use a macro that expands into a tagged tuple and a variable reference, without losing any other expressive powers of Elixir’s pattern matching syntax. A macro like TaggedTupleShorthand.@/1.

Put another way, if we can acclimate to this:

foo = 1
@:foo
#=> {:foo, 1}

and learn to swallow/write a linter to forbid this:

ok when is_integer(ok) ->
  @"ok" # {"ok", ok} <- nobody wants this

We can get rich field punning in places like these, and anywhere else two-tuples are used in Elixir AST:

destructure_map = fn %{@:foo, @"bar"} ->
  {foo, bar}
end

map = %{"bar" => 2, foo: 1}
destructure_map.(map)
#=> {1, 2}

Phoenix Channels

Before:

def handle_in(
      event,
      %{
        "chat" => chat,
        "question_id" => question_id,
        "data" => data,
        "attachment" => attachment
      },
      socket
    )
    when is_binary(chat) do...

After:

def handle_in(event, %{@"chat", @"question_id", @"data", @"attachment"}, socket)
    when is_binary(chat) do...

Diff:

-def handle_in(
-      event,
-      %{
-        "chat" => chat,
-        "question_id" => question_id,
-        "data" => data,
-        "attachment" => attachment
-      },
-      socket
-    )
+def handle_in(event, %{@"chat", @"question_id", @"data", @"attachment"}, socket)
     when is_binary(chat) do...

Phoenix Controller Actions

Before:

def show(conn, %{"id" => id, "token" => token}) do
  case Phoenix.Token.decrypt(conn, "file", token, max_age: :timer.minutes(1)) do
    {:ok, %{id: ^id, vsn: 1, size: _size}} ->
     path = MediaLibrary.local_filepath(id)
     do_send_file(conn, path)

    _ ->
      send_resp(conn, :unauthorized, "")
  end
end

After:

def show(conn, %{@"id", @"token"}) do
  case Phoenix.Token.decrypt(conn, "file", token, max_age: :timer.minutes(1)) do
    {:ok, %{@^:id, vsn: 1, size: _size}} ->
     path = MediaLibrary.local_filepath(id)
     do_send_file(conn, path)

    _ ->
      send_resp(conn, :unauthorized, "")
  end
end

Diff:

-def show(conn, %{"id" => id, "token" => token}) do
+def show(conn, %{@"id", @"token"}) do
   case Phoenix.Token.decrypt(conn, "file", token, max_age: :timer.minutes(1)) do
-    {:ok, %{id: ^id, vsn: 1, size: _size}} ->
+    {:ok, %{@^:id, vsn: 1, size: _size}} ->
      path = MediaLibrary.local_filepath(id)
      do_send_file(conn, path)

What do you think?

Have you ever wanted field punning in Elixir? Is it worth introducing a general tagged tuple syntax to get it?

Maybe... But not by enough

Worth noting that repurposing the module attribute operator @ for field punning is not my preference, nor do I encourage the use of the tagged tuple shorthand outside of field punning—but this is the only way to deliver this feature in library form.

TaggedTupleShorthand is compatible with normal module attribute usage, so you can use it everywhere and still @doc everything to your heart’s content. If this gains popularity and merits a language proposal, we can always introduce a new distinct unary operator as needed and restrict the scope of where this expression is allowed.

Try it out!

Grab the library and play with it in your codebase or console today!

I’m still not sure how I feel about this syntax myself, but now that I have a library as a proof-of-concept, I’m going to start playing with it in personal projects, and I invite you to do the same.

Well then, I'm going to put them on and make you watch me have fun. See?

cmo · December 23, 2024, 3:21am

I don’t mind it for de-structuring maps but you lost me with the pin syntax and magic tuples.

I wish we could use the dot syntax through the function without paying the performance penalty for using Access.

christhekeele · December 23, 2024, 4:22am

Yep, it’s not really intended to be used generally for “magic” two-tuples wherever. But I am laying the mechanism behind how it enables map destructuring bare, intentionally. Without first-class language support to reign in the scope of where it’s allowed, and improving match and variable warnings/errors accordingly, at the end of the day it’s just a magic two-tuple syntax.

That’s kind of the idea behind naming it use TaggedTupleShorthand. I want people trying this syntax on to understand the underlying mechanism when they sign up for it, because without first-class language support you’re going to see confusing messages and have to think a little more about the ordering of expressions pattern matching in certain edge-cases.

(Also, as this is an experiment, I don’t want to squat on a more valuable library name that someone’s likely to want in the future. This is exactly what it says: a tagged tuple shorthand! It just gets us terse k/v destructuring by merit of how Elixir map literals work under the hood.)

Eiji · December 23, 2024, 7:28am

How about something like that?

defmodule MyLib do
  defmacro sigil_p({:<<>>, _meta, [string]}, flags) do
    left_to_atom = ?a in flags

    case String.split(string, ":", parts: 2) do
      ["^" <> left] -> decompose_pun(left, left_to_atom, true)
      [left] -> decompose_pun(left, left_to_atom, false)
      [left, "^" <> right] -> decompose_pun(left, right, left_to_atom, true)
      [left, right] -> decompose_pun(left, right, left_to_atom, false)
    end
  end

  defp decompose_pun(left, left_to_atom, right_to_pin) do
    left
    |> ast_left(left_to_atom)
    |> then(&ast_decompose_pun(&1, &1, right_to_pin))
  end

  defp decompose_pun(left, right, left_to_atom, right_to_pin) do
    left
    |> ast_left(left_to_atom)
    |> then(&ast_decompose_pun(&1, right, right_to_pin))
  end

  defp ast_left(left, true), do: String.to_existing_atom(left)
  defp ast_left(left, false), do: left

  defp ast_decompose_pun(left, right, true), do: {left, ast_pin_variable(right)}
  defp ast_decompose_pun(left, right, false), do: {left, ast_variable(right)}

  defp ast_pin_variable(right), do: {:^, [], [ast_variable(right)]}

  defp ast_variable(right) when is_atom(right), do: Macro.var(right, nil)
  defp ast_variable(right), do: right |> String.to_existing_atom() |> ast_variable()
end

defmodule Example do
  import MyLib

  def with_pin do
    atom = :old_atom
    string = "old string"
    result = "Old Result"
    value = result

    ~p"^atom"a = {:atom, :old_atom}
    %{~p"^atom"a} = %{atom: :old_atom}
    [~p"^atom"a] = [atom: :old_atom]

    ~p"^string" = {"string", "old string"}
    %{~p"^string"} = %{"string" => "old string"}
    [~p"^string"] = [{"string", "old string"}]

    ~p"ok:^result"a = {:ok, "Old Result"}
    %{~p"key:^value"a} = %{key: "Old Result"}
    [~p"key:^value"a] = [key: "Old Result"]

    result = :new_result
    value = result
    ~p"ok:^result" = {"ok", :new_result}
    %{~p"key:^value"} = %{"key" => :new_result}
    [~p"key:^value"] = [{"key", :new_result}]
  end

  def without_pin do
    ~p"atom"a = {:atom, :atom}
    true = atom == :atom

    %{~p"atom"a} = %{atom: :atom}
    true = atom == :atom

    [~p"atom"a] = [atom: :atom]
    true = atom == :atom

    ~p"string" = {"string", "string"}
    true = string == "string"

    %{~p"string"} = %{"string" => "string"}
    true = string == "string"

    [~p"string"] = [{"string", "string"}]
    true = string == "string"

    ~p"ok:result"a = {:ok, 5}
    true = result == 5

    ~p"ok:result" = {"ok", 10}
    true = result == 10

    %{~p"key:value"a} = %{key: 5}
    true = value == 5

    %{~p"key:value"} = %{"key" => 10}
    true = value == 10

    [~p"key:value"a] = [key: 5]
    true = value == 5

    [~p"key:value"] = [{"key", 10}]
    true = value == 10
  end
end

Example.with_pin()
Example.without_pin()