Elixir allows Map functions to manipulate structs

We ran into an issue today with structs that I found a bit surprising.

The essence of the issue is:

  • A struct is a Map under the hood
  • you can manipulate a struct with Map functions, deleting keys that should be there and adding keys that shouldn’t
  • as long as the __struct__ key is still there, the Map is treated as a struct including in Elixir pattern matching

Here’s some code which demonstrates this:

defmodule Sandbox do    
  @enforce_keys [:mandatory]
  defstruct [:mandatory, :optional]

  def struct_foo do
    s1 = %Sandbox{mandatory: "must have this"}
    do_something("s1", s1)

    s2 = Map.delete(s1, :mandatory)
    do_something("s2", s2)

    s3 = Map.put(s1, :bogus, "blah")
    do_something("s3", s3)


  end

  def do_something(s, %Sandbox{} = thing) do
    IO.puts "#{s} #{inspect thing} is a %Sandbox"
  end

  def do_something(s, thing) do
    IO.puts "#{s} #{inspect thing} is just a thing"
  end

end
iex(24)> Sandbox.struct_foo()
s1 %Sandbox{mandatory: "must have this", optional: nil} is a %Sandbox
s2 %{__struct__: Sandbox, optional: nil} is a %Sandbox
s3 %{__struct__: Sandbox, bogus: "blah", mandatory: "must have this", optional: nil} is a %Sandbox

One interesting thing is that IO.inspect seems to know the difference between the original struct and one that’s been hacked into a Map, but Elixir pattern matching doesn’t.

So you almost certainly should not be manipulating a struct in this way, and you certainly shouldn’t be deleting keys from structs. The question is: should (could) Elixir do anything to stop you doing this? Should Map functions check for __struct__ and behave differently if they are asked to do something nefarious to a struct? Should a Map function which manipulates a struct at the very least also remove the __struct__ key so it returns a Map that is not treated as a struct any more?

1 Like

If you know you’re working with a struct and want to preserve its integrity use struct/2 or struct!/2.

1 Like

That’s a really weird code though. It’s known that you shouldn’t try adding/deleting keys to/from a struct.

Why would you have code like your above?

1 Like

Well I don’t have code like that – that’s just sample code to demonstrate the underlying behaviour: that Elixir allows you to manipulate a struct into something that looks nothing like the struct, but it still treats it as a struct.

(For the curious, the issue was that somewhere in a codebase someone had, with good intentions, deleted from a struct all keys whose value was nil. This caused issues elsewhere in the codebase – specifically when relying on the default @derive implementation of Jason.Encoder – where code assumed that if something was a struct then all the keys should be there.)

I know you shouldn’t write code like that; my question was whether Elixir should - or indeed could - be doing something to stop you.

So if you are dealing with a larger code base and more developers then I’d say it makes sense to convert the strict structs to loose maps on their way out of the code you control. Let the other modules / projects do whatever with them and then try and convert them back to strict structs via the struct function on their way in back to your code.

I’m surprised dialyzir didn’t catch it, this seems like a case it would catch?

If the type is not opaque what could dialyzer catch here? It is not even aware of structs. Making a struct opaque is possible but then you have to write accessors for every piece of data you might want to use.

But dialyzer does know that access on a struct’s/map’s fields that don’t exist is an error.

I don’t think any error will be raised in the above code. The Map.put option succeeds, so what is there for dialyzer to complain about? The elixir compiler does complain about access or pattern match of non-existing fields, but dialyzer will never even see that code. It should catch cases that must lead to an error in exactly the same circumstances that it would for a bare map, but no others unless user-defined types are supplied.

One of the things I find odd is that IO.inspect somehow knows the difference between a struct and the Map with a __struct__ key that is returned from e.g. Map.delete:

struct: %Sandbox{mandatory: "must have this", optional: nil} is a %Sandbox
map with a __struct__ key: %{__struct__: Sandbox, optional: nil} is a %Sandbox

Correct.

It could but it would make all maps operations more expensive, which is mostly why we don’t. If we had a static type system, doing Map.put/3 on a struct would certainly fail.

5 Likes

If there is a typespec in the function, dialyzer will fail :slight_smile:

Example:

defmodule Test do
  defstruct a: 0, b: nil, c: nil

  @type t :: %__MODULE__{a: non_neg_integer(), b: nil | String.t(), c: atom()}

  @spec foo() :: t
  def foo do
    %Test{
      a: 1,
      b: "aha",
      c: :foo
    }
    |> Map.put(:bar, "bar")
  end
end
$ mix dialyzer


lib/test.ex:6:invalid_contract
The @spec for the function does not match the success typing of the function.

Function:
Test.foo/0

Success typing:
@spec foo() :: %Test{:a => 1, :b => <<_::24>>, :bar => <<_::24>>, :c => :foo}
________________________________________________________________________________
2 Likes

Adding to this point wanted to say I found this library to be awesome for succinctly defining typespecs, enforced keys, and default values on structs so I now use it for every struct… https://github.com/ejpcmac/typed_struct

It doesn’t solve all the runtime concerns of the OP but it helps dialyzer help you without too much ceremony!

3 Likes

As I’ve written above, dialyzer errors properly on Map.put/3, guarding structs if you do use typespecs…

…However noticed that Map.delete/2 would still pass, even if the required key defined in the struct typespec is missing :thinking:

Example, the below returns an empty map after all, and still passes as it’s returning a t:

  @spec foo() :: t
  def foo do
    %__MODULE__{
      a: 1,
      b: "aha",
      c: :foo
    }
    |> Map.delete(:a)
    |> Map.delete(:b)
    |> Map.delete(:c)
    |> Map.delete(:__struct__)
  end

It isn’t a struct specific issue, a simple map also has the same issue (no errors):

  @spec foo() :: %{required(:a) => non_neg_integer()}
  def foo do
    %{a: 1}
    |> Map.delete(:a)
  end

I think it is a dialyzer bug, let’s see with the OTP guys what this behaviour turns out to be, bug or feature :stuck_out_tongue:

https://bugs.erlang.org/browse/ERL-1002

3 Likes