(De)serializing JSON documents into Structs: Just include `__struct__`? Seemingly not quite

The UUID key in your output is an atom. If it was a string, the syntax would be "bfa..." => %{ instead of "bfa...": %{. As you suspected, this is risky.

Your __struct__ key has been included in the JSON as a string, and thus retrieved back as a string. But the value in an Elixir struct is an atom (the module name). So your struct is broken and Elixir shows it as a regular map. Jason doesn’t support decoding data into structs so you’ll need to do that yourself.

I’d decode with Jason without supplying keys: :atoms (because it’s dangerous) and then look for some library to validate/transform the data forward from that point. I think Ecto can also be used for this but I don’t have experience with that.

Tbh instead of questioning json vs binary to term I’d question if it is a good idea to try to store structs – imo the answer is no.

Structs are a datatype whose lifetime is coupled to the code running. Each time code is changed the potential exists for a struct definition to have changed. Data stored in a db usually has a lifetime longer than that. Therefore you want to lower values to a simpler format and explicitly transform to and from those higher level values.

That transformation layer is is then the place you can use to upcast or downcast between distinct versions of a struct.

Not a perfect solution to your issue, but I’ve been using data_schema v0.5.0 — Documentation for quite some time now and very happy with it. We have extended it to include a lot more functionality to fit our business case better

You might want to take a look at estructura library and specifically to Estructura.User example which is solving exactly the issues you’ve described:

• loading nested map data into structs
• coercion of types (like ISO-8601 binary into DateTime struct)
• input validation
• and generation of StreamData “instances” for property-based testing.

Nitpick: build here is not building the type that you’ve declared for presences - it produces a map with atom keys, but the type specifies Ecto.Uuid.t() keys.

You’d write it as (not tested, apologies if it’s wrong):

  def build() do
    %__MODULE__{
      id: Ecto.UUID.generate(),
      name: "Some Guest",
      presences: %{
        Ecto.UUID.generate() => %Presence{

I think this is the crux of the issue.

In order to rectify this you could create a simple mapping atom => module i.e. :my_struct => MyApp.Example.MyStruct and store a type: :my_struct in each JSON blob. That will at least save you if you ever need to rename your modules.

Then you could implement default values in your struct definitions when you add new fields for backwards compatibility. You could also drop fields which are no longer found in the struct. You could add type checking…

And of course by the time you are finished you will have reimplemented Ecto on top of JSON

There are two real, underlying problems here: first, Ecto has no polymorphic embeds (which is what you’re really trying to do with the structs). And second, relational databases fail to properly model polymorphic relations because foreign key references are too rigid (restricted to a single table).

For us, the former problem is probably easier to solve (though personally I am far more interested in the latter).

Shameless plug, you could do something like that:

Mix.install([:typed_struct, :jason, :ecto, :jsv], consolidate_protocols: false)

require Protocol

defmodule Example do
  use TypedStruct
  @type id :: Ecto.UUID.t()

  typedstruct module: Presence do
    field(:joined_since, DateTime.t(), enforce: false)
    field(:reason, String.t())
  end

  typedstruct enforce: true do
    field(:id, id)
    field(:name, String.t())
    field(:presences, %{optional(Presence.id()) => Presence.t()})
  end

  def build() do
    %__MODULE__{
      id: Ecto.UUID.generate(),
      name: "Some Guest",
      presences: %{
        Ecto.UUID.generate() => %Presence{
          joined_since: DateTime.utc_now(),
          reason: "Creation"
        }
      }
    }
  end
end

:ok = Protocol.derive(Jason.Encoder, Example)
:ok = Protocol.derive(Jason.Encoder, Example.Presence)

defmodule ExampleSerializer do
  import JSV

  defschema_for(Example, %{
    type: :object,
    properties: %{
      id: %{type: :string},
      name: %{type: :string},
      presences: %{
        type: :object,
        propertyNames: %{type: :string, format: :uuid},
        additionalProperties: PresenceSerializer
      }
    }
  })
end

defmodule PresenceSerializer do
  import JSV

  defschema_for(Example.Presence, %{
    type: :object,
    properties: %{
      joined_since: %{type: :string, format: :"date-time"},
      reason: %{type: :string}
    }
  })
end

ExUnit.start()
import ExUnit.Assertions

deserializer = JSV.build!(ExampleSerializer, formats: true)
original_data = Example.build()
json = Jason.encode!(original_data, pretty: true)
{:ok, new_data} = JSV.validate(Jason.decode!(json), deserializer, cast_formats: true)
assert original_data == new_data

I want to give a shoutout to the library Flint (GitHub - acalejos/flint: Declarative Ecto embedded schemas for data validation, coercion, and manipulation. · GitHub) which we have been using extensively in our app.

You can lean on Ecto to declare schemas and go to and from JSON.

Thank you all, what an amazingly welcoming place producing a plethora of useful information this forum is.

In hindsight I should have clarified that the data that is meant to be stored is comparatively short lived. It’s a long lived process for an asynchronous turn based game and I was mainly looking for a way to make it survive server restarts and possibly periodically storing it away just in case.

I sort of liked the idea to have the data queryable in PostgreSQL as it has some amazing JSON capabilities. But I guess I will go with the YAGNI route and start with the simplest solution (binary storage) and worry about the implications of version updates if I ever make it that far. But I will at least tag the data with some kind of version at the root and hope that I am smart enough to keep in mind bumping that version if I make incompatible changes.

Thanks also for the corrections regarding the atom keys, I missed the fact that it’s the delimiter between key and value that tells me the type of the key. The fact that I missed the __struct__ value is not an atom (representing the module) seems a little embarassing in hindsight, but at least now I really know. And the nitpick regarding the type of the presences key was most welcome as well.

Last but not least: Thanks alot for all the library suggestions. It seems like there is a solution for desired level of complexity if I ever make it far enough. That’s reassuring

You can use a jason library to go from json binary to a graph of Elixir data structures (maps and lists) and then define Ecto embedded schemas to take that as “params” and generate a graph of structs (all Ecto schemas are structs too) using all the cast power Ecto types offer. I gave a talk a few years back, “Ecto without a DB” exploring that when I worked at a company transforming a lot of json from 3rd party APIs.