Our setup is covered in some detail in this talk: https://www.youtube.com/watch?v=cVQUPvmmaxQ
darkmarmot
darkmarmot
jeremyjh
I think this versioning problem is a really interesting problem actually. In Haskell there is a library called SafeCopy that attempts to solve the same problem; it is used for restoring binary serialized data in an updated version of the “same” logical type. Basically you have a separate type for each version that has existed, and a typeclass (similar to protocol) that is responsible for migrating the data from the previous version to some later version. I think something conceptually similar could be done in an Elixir library though the details would be completely different (there is no multi-type protocol in Elixir for one thing, but you could manipulate structs as raw maps in a migration function).
It is a very small fraction of the Elixir community that even uses hot code loading so I wouldn’t expect language changes to address it, but I think it could be done in a library and even be pretty usable with the right macros.
5
wojtekmach
Hex Core Team
I must be missing something because I don’t see where the problem is. Please bare with me. This Elixir code:
def f(%Date{} = date) do
date
end
compiles to this Erlang code:
f(#{'__struct__' := 'Elixir.Date'} = _date@1) ->
_date@1.
so at runtime there are no checks on anything besides the __struct__ field.
Of course, if you choose to write f(%Date{year: year}) and you pass something other than a map
with %{__struct__: Date, year: year} then that’s gonna be a match error.
This functions:
def f(%Date{} = date) do
date.year
end
compiles to:
f(#{'__struct__' := 'Elixir.Date'} = _date@1) ->
case _date@1 of
#{year := _@1} -> _@1;
_@1 when erlang:is_map(_@1) ->
erlang:error({badkey, year, _@1});
_@1 -> _@1:year()
end.
so there are no additional checks either.
If you don’t want this behaviour of date.year then call Map.get(date, :year)? Or implement
Access for your structs? (cannot implement it for things you don’t own.)
Is it less about your code where you can choose not to use structs and more about libraries that you want to use which do use them, and thus you run into a danger of a runtime errors when things change?
I’m really curious, what dd you exactly mean by structs becoming more like duct typed maps. Which semantics of the structs are you proposing to change? How should they behave?
13
cevado
darkmarmot
I probably should’ve gotten with my team first to put together a list of places where stuff has gone badly for us here. I think we’ve gotten a bit of PTSD about using them at this point, maybe undeservedly in some places.
3
al2o3cr
I have a vague recollection of problems with :erlang.binary_to_term and structs with different definitions, but they don’t seem to happen on 1.12 + OTP24:
iex(1)> defmodule Foo do
...(1)> defstruct [:foo]
...(1)> end
iex(2)> %Foo{foo: "bar"}
%Foo{foo: "bar"}
iex(3)> %Foo{foo: "bar"} |> :erlang.term_to_binary()
<<131, 116, 0, 0, 0, 2, 100, 0, 10, 95, 95, 115, 116, 114, 117, 99, 116, 95, 95,
100, 0, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 100, 0, 3, 102,
111, 111, 109, 0, 0, 0, 3, 98, 97, 114>>
---RESTART IEX---
iex(1)> defmodule Foo do
...(1)> defstruct [:foo, :bar]
...(1)> end
iex(2)> binary = <<131, 116, 0, 0, 0, 2, 100, 0, 10, 95, 95, 115, 116, 114, 117, 99, 116, 95, 95,
...(2)> 100, 0, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 100, 0, 3, 102,
...(2)> 111, 111, 109, 0, 0, 0, 3, 98, 97, 114>>
iex(3)> :erlang.binary_to_term(binary)
%{__struct__: Foo, foo: "bar"}
---RESTART IEX---
iex(1)> defmodule Foo do
...(1)> defstruct [:bar]
...(1)> end
iex(2)> binary = <<131, 116, 0, 0, 0, 2, 100, 0, 10, 95, 95, 115, 116, 114, 117, 99, 116, 95, 95,
...(2)> 100, 0, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 100, 0, 3, 102,
...(2)> 111, 111, 109, 0, 0, 0, 3, 98, 97, 114>>
iex(3)> :erlang.binary_to_term(binary)
%{__struct__: Foo, foo: "bar"}
Even adding @enforce_keys didn’t change that last result 
0
hauleth
Well, the explanation is simple - binary_to_term/1 knows nothing about structs. When you use %Foo{foo: "bar"} what Elixir logically does is to check in the compiler cache whether it is allowed construct and then create map with all values. When you do something like struct(Foo, foo: "bar") then it will “behind the scenes” call the hidden “magical” function that will validate all that data. binary_to_term/1 is “dumb” deserialiser and doesn’t know anything about Elixir structs, so it happily decode the encoded map. There is no way to circumvent that, as that would additionally make it really hard to use Distributed Erlang with structs. So it is all law of leaky abstractions.
@darkmarmot only case when the stuff could go ugly after upgrade is if you have renamed or removed keys from the struct, otherwise it would work perfectly fine, as it is the reason why Elixir uses map-backed structs instead of tuple-based records as it is done in Erlang. Whole reason for that is that you can “painlessly” add new or reorder fields in the struct without need to change all pattern matching and recompile modules that use this structure. So you either done some real magic that broken that, or you have done something that would broke the invariants independently whether you would use other constructs. So show us what change have caused that mayhem, because now we are flying dark with explanations.
9
paulstatezny
@darkmarmot It seems in this thread there’s some disagreement or at least confusion about in what ways things can break due to changing struct shapes.
Would you be willing to share a specific piece of code (or distilled example) and elaborate on in what sense it broke? (The specific runtime error that happened, or whatever else happened.)
0
darkmarmot
Yeah, thanks! I’m talking to my team now to see if we can build a project to reproduce what we thought we had been experiencing (to make sure we weren’t just deluding ourselves and misinterpreting data as well as to provide feedback here since it sounds like there are technical reasons we shouldn’t have experienced what we thought we were). And it’s totally possible we were wrong.
It might take a bit, though, as we’re in the middle of a bunch of stuff – but the basic setup will need to send structs between 2 nodes on different versions of a codebase using OTP 22 cluster, rpc and gzip. I’ll update this thread when we have something definitive.
Thanks!
12