Proposal: Add field puns/map shorthand to Elixir

proposal

#1

Moderation note: this is a language proposal. Please keep the discussion on topic and try to read the following discussion for any updates when joining in. If you agree or disagree with the proposal, do your best to give precise and clear feedback to the author.

New polls have been added for a struct only proposal

TL;DR Add field punning to maps in Elixir

Background

What is Field Punning?

Field punning allows a programmer to use a variable name as the key in a map literal expression. For example,
in javascript:

import {foo, bar} from 'common-names';

is equivalent to:

import {foo: foo, bar: bar} from 'common-names';

Here, the variable name, foo was used as the key, and the value was assigned to that variable.

The purpose of field punning is to reduce redundant information when reading and writing code.

It has the added benefit of reducing code verbosity. In Elixir where pattern matching is common, this
would lead to complex pattern matches being simpler to read. This is especially true in function heads
which can get long and are less readable when split over multiple lines.

Field Punning in Elixir

Field punning in Elixir is a very common request (1, 2, 3, 4, 5). I created a spreadsheet to help me summarize the differing viewpoints. In my review of previous conversations, more people support adding this feature to the language than those who oppose it. Even people who did oppose the feature agreed that there would be many cases where they have code that would be able to take advantage of this feature. People who did oppose it expressed the following concerns:

  1. It can be implemented as a package.
  2. Atom and string key confusion.
  3. Proposed syntax similar to tuples.
  4. The variable name becomes meaningful.
  5. It is less explicit. Seems like magic.
  6. Create and update of maps is cryptic with field punning.

1, I will address next. 2 and 3, I will attempt to address in the proposal by using different syntax than what has previously been suggested. The rest I will address last.

Package Implementations and Shortcomings

short_maps and shorter_maps both use sigils to express a pattern match of a map with field punning. The problem with the sigil approach is that the macro is passed a string. For example, ~m(foo bar) would pass "foo bar" as a string to the ~m macro. That means any feature supported by the sigil has to be manually parsed and then converted into the desired code. So:

~m(%Foo bar ^baz)a = some_map

would require parsing %Foo, bar, ^, and baz and turning that into

%Foo{bar: bar, baz: ^baz} = some_map

short_maps does all that, but stops short of implementing nesting and allowing non-punned fields. shorter_maps takes things a bit farther and implements more features such as nesting, non-punned fields, and map updates. The result is two libraries that reimplement a subset of the elixir syntax, but can’t use the AST. This makes the implementation difficult and full of pitfalls and caveats.

shorthand and synex do not use sigils and can take advantage of the Elixir AST. The caveat with shorthand is that in order to provide syntax like m(foo, bar) it has to generate the m/1, m/2, m/3, m/N versions of the macros because each variable passed is a separate argument in Elixir. So, if you wanted to create a map of N+1 elements, you would get an error. The workaround in the library is to wrap the elements in a list. So, you would have something like:

m([foo, bar]) = my_map

synex has the same constraints as shorthand. One benefit is that it is able to utilize more of the standard Elixir syntax, by requiring things to be wrapped in a map. For example:

map = keys(%{map | a, b, c: 100, d: 200})
keys(%{^a, ^c}) = %{a: 1, b: 2, c: 3}

All the libraries, in my opinion, suffer from not looking like a map. This makes it harder to understand what is going on. Sigils look slightly more like the syntax you might expect, but suffer from having to reimplement Elixir’s AST. Macro’s are able to take advantage of the AST, but make pattern matching much harder to understand, because it looks like a function call and it is not clear what the macro does. All libraries also have the disadvantage that you must import them everywhere you wish to use this feature. Additionally, I believe (I haven’t tested this yet), ExUnit can’t display useful error messages when using any of these libraries in an assertion.

Proposal

I propose we add two modifiers to the map syntax a and s for atom and string respectively. The presence of the modifier specifies the type of key used for field puns.

x = 1
y = 2

%{x, y}a #=> %{x: 1, y: 2}
%{x, y}s #=> %{"x" => 1, "y" => 2}

When no modifier is specified, the default modifier a is assumed as the default. However, using punning without a modifier will produce a warning suggesting a modifier be specified. The warning would mention both modifiers in case string keys were intended. This would help people who just try to use field punning to discover the correct usage in Elixir. The code formatter would fix this warning for the user by adding a, when no modifier is specified. Atom keys are chosen as the default because they are the dominant key type in Elixir.

When field punning is not being used, but a modifier is specified, Elixir will emit a warning saying that the modifier is not necessary. The code formatter will remove the modifier.

Field puns can be used with normal key value pairs. However field puns must come first. This makes field puns behave similar to Keyword lists and function arguments.

map = %{x, y: 2}a #=> %{x: 1, y: 2}

# This error could likely be improved to be more helpful.
map = %{y: 2, x}a #=> ** (SyntaxError) syntax error before: x

This change will require the elixir syntax be changed to parse the modifiers and the %/2 and %{}/1 kernel special forms be updated to support field punning and the modifiers. Additionally, the code formatter would be updated to consider the new syntax.

The following sections go into more detail about specific map constructs in Elixir and how they will be adapted to support field punning.

Creating Maps

Existing map declarations will work the same with no warnings or modifications required.

x = 1
y = 2
# No punning
%{x: x, y: y} #=> %{x: 1, y: 2}

In order to declare a map with atom keys, you should add the a modifier. If you do not, the code will still work, but produce a warning suggesting the adding of a modifier.

# Punning with atom keys
%{x, y}a #=> %{x: 1, y: 2}

# Punning with atom keys (default)
%{x, y, z: 3} #=> Warning: field punning requires ...
              #=> %{x: 1, y: 2, z: 3}

The a modifier does not prevent you from specifying key value pairs with string keys.

%{x, y, "z" => 3}a #=> %{x: 1, y: 2, "z" => 3}

In order to declare a map with string keys, the s modifier must be specified. Otherwise, the default behavior described above occurs.

# Punning with string keys
%{x, y, "z" => 3}s #=> %{"x" => 1, "y" => 2, "z" => 3}

The s modifier does not prevent you from specifying key value pairs with atom keys.

# Punning with string keys
%{x, y, z: 3}s #=> %{"x" => 1, "y" => 2, z: 3}

Maps with punning can be nested. The modifier on a map only applies to that level of nesting. For example, using the a modifier on the top level map does not make the a apply to nested maps. They must each specify their modifier in order to not receive a warning about specifying a modifier. Likewise, there is no requirement that nested maps use the same modifier as their parent.

%{x, nested: %{y}a}a #=> %{x: 1, nested: %{y: 2}}

%{x, nested: %{y}}a #=> Warning about nested map not having modifier
                    #=> %{x: 1, nested: %{y: 2}}

%{x, nested: %{y}s}a #=> %{x: 1, nested: %{"y" => 2}}

Creating Structs

Existing struct declarations will work the same with no warnings or modifications required. The a modifier will be the default with structs and Elixir will not emit a warning when punning is used in structs. If the a modifier is specified, Elixir will emit a warning saying the a modifier is not necessary. The code formatter will remove the a modifier if it is present. This is intended to help people who convert a map to a struct. The s modifier will not be valid for structs, since structs can only have atom keys.

%Point{x, y} #=> %Point{x: 1, y: 2}

%Point{x, y}a #=> Warning: ...
              #=> %Point{x: 1, y: 2}

%Point{x, y}s #=> ** (ArgumentError) ...

The properties described for nested maps applies to structs as well and either may be nested inside one another.

Updating Maps

Maps can be updated by using the map update syntax %{map | key: value}. Field punning will also be possible in map updates. When field puns are used, you should specify a modifier as described above and the field puns must precede any key value pairs.

point = %{x: 1, y: 2}
point2 = %{"x" => 1, "y" => 2}
x = 3

%{point | x} #=> Warning: ...
             #=> %{x: 3, y: 2}

%{point | x}a #=> %{x: 3, y: 2}
%{point2 | x}s #=> %{"x" => 3, "y" => 2}

%{point | x, y: 4}a #=> %{x: 3, y: 4}
%{point2 | x, "y" => 4}s #=> %{"x" => 3, "y" => 4}

%{point | y: 4, x}a #=> ** (SyntaxError) syntax error before: x
%{point2 | "y" => 4, x}s #=> ** (SyntaxError) syntax error before: x

If the key is not already present in the map, it will fail with the same error message that exists today.

Updating Structs

When using the map update syntax for structs, the rules for maps also apply. Because structs can only have atom keys, you should use the a modifier. Map update will be unaware that it is updating a struct, so it will not error on an invalid s modifier being present. Instead the update will fail on inserting the string keys.

point = %Point{x: 1, y: 2}
x = 3

%{point | x} #=> Warning: ...
             #=> %Point{x: 3, y: 2}

%{point | x}a #=> %Point{x: 3, y: 2}
%{point | x}s #=> ** (KeyError) key "x" not found in: ...

%{point | x, y: 4}a #=> %Point{x: 3, y: 4}

%{point | y: 4, x}a #=> ** (SyntaxError) syntax error before: x
%{point | "y" => 4, x}s #=> ** (SyntaxError) syntax error before: x

When using the struct update syntax on a struct, no modifier needs to be present. The a modifier will issue a warning that the a modifier is unnecessary and will be removed by the code formatter. The s modifier will produce an error.

point = %Point{x: 1, y: 2}
x = 3

%Point{point | x} #=> %Point{x: 3, y: 2}

%Point{point | x}a #=> Warning: ...
                   #=> %Point{x: 3, y: 2}
%Point{point | x}s #=> ** (ArgumentError) ...

%Point{point | x, y: 4}a #=> %Point{x: 3, y: 4}

%Point{point | y: 4, x}a #=> ** (SyntaxError) syntax error before: x
%Point{point | "y" => 4, x}s #=> ** (SyntaxError) syntax error before: x

Pattern Matching Maps

In order to support field puns, the a and s map modifiers may also be used anywhere a pattern match may occur (left of =, function head, case, etc). In addition, the pin operator (^) will also be allowed with field puns. When the pin operator is used, the value of the variable will be pinned and the variable name will be used as a literal. In order to pin a variable for match with a key, field puns cannot be used.

x = 1
point = %{x: 1, y: 2}

%{^x, y} = point #=> Warning: ...
                 #=> x is matched and y is bound to 2

%{^x, y}a = point #=> x is matched and y is bound to 2

When a pinned field pun variable is not present in the scope, it will raise an error.

%{^z}a = %{z: 1} #=> unknown variable ^z. No variable "z" has been defined before the current pattern

When a match error occurs with a pinned field pun, it will raise the same error used today for key value pairs.

z = 2
%{^z}a = %{z: 1} #=> ** (MatchError) no match of right hand side value: %{z: 1}

%{^z} = %{z: 1} #=> Warning: ...
                #=> ** (MatchError) no match of right hand side value: %{z: 1}

When the key type is mismatched in a pattern match, the same error that is produced for key value errors will be used.

z = 1
%{^z}s = %{z: 1} #=> ** (MatchError) no match of right hand side value: %{z: 1}

Ideally in the future Elixir could add a more helpful error message about string and atom key mismatches for both field puns and key value pairs.

Pattern Matching Structs

Structs will behave similar to maps, but the a modifier will not be necessary.

x = 1
point = %Point{x: 1, y: 2}

%Point{^x, y} = point #=> x is matched and y is bound to 2

%Point{^x, y}a = point #=> Warning: ...
                       #=> x is matched and y is bound to 2

%Point{^y} = point #=> unknown variable ^y. No variable "y" has been defined before the current pattern

x = 2
%Point{^x} = point #=> ** (MatchError) no match of right hand side value: %Point{x: 1, y: 2}

Objections and Responses

This Feature is About Saving Key Strokes

The narrowing of this proposal to structs only, in my opinion, reduces the strength of this argument. See the poll below.

The Name of the Variable and Key are Tied Together

I would make the argument that the key name and the variable name are already coupled, in that both are describing the value that they point to. So, if the value they point to changes, such that the name no longer makes sense, I would want to change both. I personally would make that sort of change regardless of field puns being present.

It is possible the names semantically are similar, they are changed to be the same. In this case, you could take the conservative approach and just change the variable name and use a key value pair to give them separate names, because it reduces the things that must change. However, if someone chooses to make the names the same universally I don’t believe that makes the code worse. It is up to the author to decide. This would also be true regardless of field puns.

Polls

How would people feel about a structs only approach? This would mean maps are not supported and there are no modifiers at all. Additionally, only the %MyStruct{struct | field_pun} update syntax would allow puns. The %{struct | field_pun} syntax would not work, you would have to use key value pairs. I’ve specified two flavors. One where %MyStruct{struct | field_pun} is supported and one where it is not.

I will assume that people who have voted that they don’t support this feature are continuing to not support it.

  • I would support a structs only proposal with struct update support
  • I would support a structs only proposal without struct update support
  • I don’t support this feature, but prefer a structs only proposal with struct update support
  • I don’t support this feature, but prefer a structs only proposal without struct update support

0 voters

Also, I’d like to understand the general preference on requiring field puns first:

  • field puns first, then key value pairs
  • field puns intermixed with key value pairs

0 voters


Has Map shorthand syntax in other languages caused you any problems?
How to use the Suggestions & Proposals section
New Suggestions & Proposals section for Elixir and other community projects
#2

I would like:

  • this proposal added to Elixir
  • this proposal added with modifications to Elixir
  • a different proposal added to Elixir
  • this feature to not be added to Elixir

0 voters


#4

@yawaramin Do you have an alternate proposal in mind?


#5

For those who don’t want this in the language. Pretend for a moment this was going to be added to Elixir anyway. Is there something in this proposal you would want to suggest be changed to make it more palatable?


#6

As I said in the mailing list. I would like this for structs only. It is most similar to the other three languages I pointed out (Haskell, OCaml and Rust) in that you define an actual datatype with specific keys.

I personally don’t see what is gained from doing this on a bare map other than typing a couple less characters.


#7

I’d be happy if we only seen this on function headers, I think that’s where most of the pain comes from (IMO)

def some_function(%{name: name, age: age, blah: blah}) do
...
end

Should be:

def some_function(%{name, age, blah}) do
...
end

#8

I’m not convinced of the ultimate utility of this feature, but if it’s put in, only having in function headers would be a mistake, imo. Currently you can take a case and a function head and convert between the two freely. That would put an artificial restriction there that would require you to do manual conversions and would be a sharp corner people would regularly bump into while refactoring. If available, it should be available for all places maps can be key/value matched currently. I would say that we should have a lot of examples of troublesome code that this would be cleaned up by. My gut feeling is that I will rarely, if ever, use this, which is not true of many language features currently.

I see new Elixir programmers confusing the map and tuple syntax (I think this is a JS thing because of their map syntax?), and while {name, address, friends} is a valid tuple, %{name, address, friends} would be a valid map here, but completely different semantics from that single character. I imagine that will result in hard to track down bugs when refactoring, especially for people still learning the syntax.


#9

There’s already quite the confusion around %{a: :b} being sugar for %{:a => :b} and therefore people not even realizing that maps can take any term as key not just atoms or strings. Therefore if such a feature would make it’s way into the core I’d only like to see it for atom keys (or structs) and not for any others. So that all the syntax sugar is specific to only one type of key and not some sugar for just atoms and some for strings and atoms. I recently had to work a bit with js and after lot’s of elixir it just felt unnatural and unnecessarily indirect.


#10

IMHO redundancy is not always bad. In this particular case I think the current syntax is easier for onboarding people in the language and it is not a lot of typing we are saving here (maps keys). I think this would help and hurt at the same time. There might be cases where one would change the name of a variable just make it fit in the map structure. So, I am against this proposal.

I’d like to add that this is a very well written proposal and I enjoyed reading it. I know you’ve dedicated yourself to writing this and appreciate all the effort you put in this. Even though I don’t agree with this being added to the language I’d like all future proposals to follow a similar approach to yours.


#11

I already said this on the mailing list but since it seems like we’re now having this conversation in 2 places its worth posting here as well :stuck_out_tongue_winking_eye:.

I appreciate the time, energy, and thought you’ve put into this but I have to say I’m strongly against this proposal.

The only benefit of the proposed syntax is brevity and it sacrifices clarity and adds complexity to achieve this brevity. Having magic letters at the end of a map that change how that map is constructed strikes me as pretty beginner hostile. It introduces yet another surprising piece of syntax that you need to hold in your head. It introduces new, non-obvious syntax rules like requiring that your “shorthand” keys come before non-shorthand keys. None of those problems exist with maps or structs as they currently exist at the expense of a little more typing. Special casing this syntax to only structs, only maps with atom keys, or some other rule only serves to introduce inconsistencies to the language.

If Elixir didn’t support both string and atom keys in maps or if tuples used () instead of {} or if we didn’t distinguish between the strings and atoms or if the language was different in some other fundamental ways then I’d probably feel differently. But with the Elixir we have today I don’t think this proposal is a good fit.


#12

Of course, it will scare beginners away!


#13

Actually, I preferred %{a, b} = %{a: a, b: b} i.e. Jason S.'s proposal, but José has ruled it out so since it’s not possible I don’t have a solid suggestion right now. The only thing I can say is that I don’t want to add more symbols or corner-case syntax to map (including struct) construction. A little verbosity doesn’t hurt and can only help beginners. For power users Shorthand looks pretty good.


#14

@rawkode, I’m not sure that would be possible. But as @summers mentioned, it would probably be detrimental.


#15

I didn’t mean I only want it there; I meant that function headers are my biggest pain in this area and if it came to it, I’d be happy to only see it there - really I want it everywhere.

Perhaps my comment was too ambiguous, sorry.


#16

Are people aware of this library, which uses sigils to add shorthand for maps?

I think I’d be in favor of having a ~M{} sigil in the standard library (if it were namespaced, importing it by default seems like too much namespace pollution). Using a sigil tells the user: “Careful here! This is magic and can break all minds of guarantees for your code”. I think the normal syntax (aside from sigils should be as predictable as possible). But I’m very much against the proposal as it stands


#17

@tmbb I mentioned the libraries providing similar features and their limitations in the proposal. See Package Limitations and Shortcomings


#18

I’ve misses that section, sorry. As I said, to me, looking like magic is a feature here, not a bug.


#19

@ankhers I could be happy with just structs. I obviously prefer my full proposal. The reason I didn’t just do structs is because right now, things that work for maps tend to work for structs and vice versa. The few exceptions being non-atom keys for maps, no access protocol defined for structs, defaults for for structs, and restricted set of keys for structs. Only the access protocol one seems to be noticeable in practice though.


#20

I would have liked to cover this in the Objections and Responses section, because other people have brought up this point. In another thread I wrote:

I would make the argument that the key name and the variable name are already coupled, in that both are describing the value that they point to. So, if the value they point to changes, such that the name no longer makes sense, I would want to change both. I personally would make that sort of change regardless of shorthand syntax being present.

I suppose you could mean, when the names semantically are similar, they are changed to be the same. I think in this case, you could take the conservative approach and just change the variable name and use a key value pair to give them separate names, because it reduces the things that must change. However, if someone chooses to make the names the same universally, are you saying the code is worse off because of that?

Thanks for the kind words! I’m not sure everything needs to be this detailed, though.


#21

After reading this thread and the proposal I’d like to say that the proposal is very well-written thoughtful. However, I am against this proposal. My thoughts on the implementation of the feature are as follows:

  • The strict ordering of punned vs. non-punned fields is weird and unintuitive.
  • Elixir already gets criticism because of odd syntax things like calling an anonymous function vs. a named function, etc. I don’t think we need more of that
  • I think adding postfix modifiers to pretty much anything is a bad idea and sets a bad precedent. The syntax is inconsistent with the rest of the language (besides maybe sigils but those are prefix).
  • The length of this document itself is a testament to how this feature would add a bunch of unnecessary behavior and rules to structs and maps, which I don’t like. It’s too much.

In terms of the proposal itself and the future of Elixir in regards to this, I also have some thoughts:

  • The proposal should include examples of long pattern matches, etc. which could be improved. Proposals like this should be use case/pain point driven. Otherwise, if there aren’t any common enough use cases or pain points (as I believe is the case with this proposal), the feature shouldn’t be implemented.
  • If libraries which emulate this functionality become ubiquitous and as essential as installing NPM when you install Node, I think it would be worth revisiting this topic and this proposal and deciding how to add this to the language. As of now, I don’t think enough people are pulling in libraries for this purpose to justify adding it to the language.
  • If this is implemented, it should not just be for structs. I want to adhere to the “structs are just like maps” mantra as closely as possible.