roflbobl

roflbobl

Changing module of saved binary data

I run a system, where some structs are saved as binary with :erlang.term_to_binary and then deserialized with :erlang.binary_to_term. The problem is, that if I alter the struct, it does not match the saved binary, and it will then not be deserialized correctly.

I have a test where i add a key, after saving some data, and of course, the deserialized data is missing that key. Is was hoping it will just give the new key the value nil.

Has anyone solved this problem?

Most Liked

al2o3cr

al2o3cr

If you’re storing long-lived term data (versus caching, etc where old versions can be discarded) here are two approaches that can help:

  • explicit migration when changing the underlying struct. You’d write a one-off task to load each term, adjust its structure, then save it back
  • explicit versioning in the data. Add a version field to the struct with a default value. Bump it every time you add/remove fields. The code that says binary_to_term will need to check version and handle converting the old shape into the new shape

IMO the motivation to pick one versus the other is how often “old” data is read; if it’s used constantly then updating it once in a batch will be faster (the first approach) - but if it’s mostly historical and only read occasionally then doing it on-demand (the second approach) will avoid doing computations that might never be used.

NobbZ

NobbZ

Do not serialize the struct, but just the map without the __struct__ key.

Roughly:

data
|> Map.from_struct()
|> :erlang.term_to_binary()

For the deserializing something like the following should do:

binary
|> :erlang.binary_to_term()
|> then(&struct(YourStruct, &1))

This should work, as drafted in the following iex session:

iex(1)> defmodule S do defstruct ~w[a b c d e f g]a end
iex(2)> struct(S, %{a: 1})
%S{a: 1, b: nil, c: nil, d: nil, e: nil, f: nil, g: nil}

On top of that though, I suggest to finally stabilize the struct quickly. Or to version your data format and use explicit conversion or “upcasters”.

gregvaughn

gregvaughn

You could probably even get fancy and use the built-in @vsn Module attribute

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Other popular topics Top

jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 40042 209
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44139 214
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement