MarcusRiemer

MarcusRiemer

(De)serializing JSON documents into Structs: Just include `__struct__`? Seemingly not quite

I have loads of structs defined via typed_struct and would like to serialize and deserialize these structs to and from JSON. I am currently using Jason, but I am not tied to that at all. Serializing any of my structs into JSON is simple, deriving Jason.Encoder works. An example structure that (hopefully) contains everything I need could look like this:

require Protocol

defmodule Example do
  use TypedStruct
  @type id :: Ecto.UUID.t()

  typedstruct module: Presence do
    field :joined_since, DateTime.t(), enforce: false
    field :reason, String.t()
  end

  typedstruct enforce: true do
    field :id, id
    field :name, String.t()
    field :presences, %{optional(Presence.id) => Presence.t()}
  end

  def build() do
    %__MODULE__{
      id: Ecto.UUID.generate(),
      name: "Some Guest",
      presences: %{
        "#{Ecto.UUID.generate()}": %Presence{
          joined_since: DateTime.utc_now,
          reason: "Creation"
        }
      }
    }
  end
end

Protocol.derive(Jason.Encoder, Example)
Protocol.derive(Jason.Encoder, Example.Presence)

Implementing serialization and deserialization for this however comes with two problems:

  1. By default I am losing the type when serializing as JSON: From my point of view the easiest thing would be to tag each JSON object with the type. As far as I understand it, this is also the way a “normal” map is differentiated from a struct: By having a :__struct__ field present in the map. If I could just serialize this field alongside the normal data, the resulting map would be properly treated as a struct. I am already using the :__struct__ field to know on which module I want to call functions, so to me this seems like a natural thing to do. But it at least doesn’t seem to be encouraged by Jason: I didn’t stumble over any way of doing this except for repeating all keys as part of the only option to the protocol or providing a manual implementation.

    Protocol.derive(Jason.Encoder, Example, only: [:__struct__, :id, :name, :presences])
    Protocol.derive(Jason.Encoder, Example.Presence, only: [:__struct__, :id, :joined_since, :reason])
    
  2. I do fear however that whatever I am doing is misguided. Because when serializing and deserializing such a struct (using the derived protocol from 1.) I made some observations:

    iex(14)> Example.build |> Jason.encode!() |> Jason.decode!(keys: :atoms)
    %{
        id: "21be5f01-eef1-4ff1-afd5-a09919a9f26f",
        name: "Some Guest",
        __struct__: "Elixir.Example",
        presences: %{
          "bfa4f256-d1d9-416c-a08f-8a04e3b7cc06": %{
            id: "b0e55b2f-1541-4987-8b99-b57563584558",
            reason: "Creation",
            __struct__: "Elixir.Example.Presence",
            joined_since: "2025-03-27T12:08:18.791171Z"
          }
        }
      }
    
    1. At first I was pleasently surprised to discover, that the keys: :atoms option did not turn the UUID that is part of the presences key into an atom. That saves me from a self-inflicted atom exhaustion I guess, but I don’t understand why the UUID wasn’t converted.
    2. It however also didn’t “recover” the DateTime, but left it as a string. I guess this is to be expected, as there is no “proper” JSON representation for date and time.
    3. The resulting map suddenly has the :__struct__ “visible” as part of my iex representation. So something about the map I recovered is not quite a “proper” struct.

This leaves me with the following questions:

  1. What is a better way to tell Jason to include the __struct__ when serializing? So far I would probably write a manual implementation and re-use that. Or is there a reason I really should not do this?
  2. What is different from a “proper” struct about the map I get from calling encode! and decode!?
  3. Should I just abandon the idea of serializing to JSON and “simply” write the binary representation into my PostgreSQL database?

Marked As Solved

Nicd

Nicd

The UUID key in your output is an atom. If it was a string, the syntax would be "bfa..." => %{ instead of "bfa...": %{. As you suspected, this is risky.

Your __struct__ key has been included in the JSON as a string, and thus retrieved back as a string. But the value in an Elixir struct is an atom (the module name). So your struct is broken and Elixir shows it as a regular map. Jason doesn’t support decoding data into structs so you’ll need to do that yourself.

I’d decode with Jason without supplying keys: :atoms (because it’s dangerous) and then look for some library to validate/transform the data forward from that point. I think Ecto can also be used for this but I don’t have experience with that.

Also Liked

LostKobrakai

LostKobrakai

Tbh instead of questioning json vs binary to term I’d question if it is a good idea to try to store structs – imo the answer is no.

Structs are a datatype whose lifetime is coupled to the code running. Each time code is changed the potential exists for a struct definition to have changed. Data stored in a db usually has a lifetime longer than that. Therefore you want to lower values to a simpler format and explicitly transform to and from those higher level values.

That transformation layer is is then the place you can use to upcast or downcast between distinct versions of a struct.

16
Post #3
venkatd

venkatd

I want to give a shoutout to the library Flint (GitHub - acalejos/flint: Declarative Ecto embedded schemas for data validation, coercion, and manipulation. · GitHub) which we have been using extensively in our app.

You can lean on Ecto to declare schemas and go to and from JSON.

garrison

garrison

I think this is the crux of the issue.

In order to rectify this you could create a simple mapping atom => module i.e. :my_struct => MyApp.Example.MyStruct and store a type: :my_struct in each JSON blob. That will at least save you if you ever need to rename your modules.

Then you could implement default values in your struct definitions when you add new fields for backwards compatibility. You could also drop fields which are no longer found in the struct. You could add type checking…

And of course by the time you are finished you will have reimplemented Ecto on top of JSON :slight_smile:

There are two real, underlying problems here: first, Ecto has no polymorphic embeds (which is what you’re really trying to do with the structs). And second, relational databases fail to properly model polymorphic relations because foreign key references are too rigid (restricted to a single table).

For us, the former problem is probably easier to solve (though personally I am far more interested in the latter).

Where Next?

Popular in Questions Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New

Other popular topics Top

AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54120 245
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement