Re-casting json-encoded structs gracefully

nikokozak

Hi all - I might have dug my own grave by going down this path, but now I’m just curious… What is the best way to re-cast data after decoding a struct that has been encoded as json in order to be persisted? I’m working with embedded_schemas nested in a map that gets encoded when persisted via Ecto/PSQL. A very simplified version looks like this:

defmodule App.InnerSchema do

  embedded_schema do
    field :a_field, :string
    field :date_scheduled, :utc_datetime
  end

end

defmodule App.ParentSchema do

  schema "parents" do

    # a series of App.InnerSchema-like structs, all slightly different,
    # get stored here like so:
    # %{ plugin_1: %InnerSchema{}, plugin_2: %ADifferentInnerSchema{}, etc.. }
    field :stored_here, :map

  end

end

I thought about defining an Ecto.Type for :embedded_here, but the problem I have is that per the nature of the program, I can never be entirely sure what embedded_structs have been included in the encoded map, and while I could check fields against a list of probable structs when loading, it doesn’t seem graceful. I also tried forcing Jason.Encoder to encode the __struct__ field from the original InnerSchema in order to rescue it when decoding: the idea being, I could then cast the values I extract from the decoded json map using Ecto.Changeset.cast(the_decoded_json, apply(S.to_e_A(the_decoded_struct_field), :__struct__, []), ~w(the_fields), _opts), as per this old thread but cast keeps giving me an error saying it expects a :map instead of the %InnerSchema{}.

This might be a bit convoluted all in all, but ultimately all I want is a way to easily cast/decode each field’s type (namely for more complex stuff, like datetimes and custom Types), according to the struct that field is nested in after Jason encoding. Now that I’m typing it out, I realize it’s also about rescuing as much as possible from the json encoding (i know, i know). Any nice ways to do this?

4 comments

#ecto #data-structures #jason #embedded_schema

0 1996 4

2021-02-03 13:44:42 UTC

Showing Posts 1 to 4

Show Best Posts
Show All (oldest first)
Show All (newest first)

dimitarvp

Making your own Ecto.Types and just recursively cast – which happens automatically when you declare your schemata well – I found is the best way, both in professional and hobby projects.

What’s troubling you? Yeah it’s a bit manual… the first time. You gain stronger data consistency guarantees though.

Also, where does the data come from? Should there be a tolerance for malformed inputs?

Post #1

nikokozak

Thanks @dimitarvp - you’re right about the data consistency guarantees, it just felt a bit hacked together the first time I tried it (particularly the act of pulling from the encoded :__struct__ in order to create a new instance for casting). Having said that, it’s also pretty cool - thank you for confirming it’s a sane strategy.

The data itself is sanitized and passed through our own changesets before insertion and encoding, so there’s relatively little risk of malformed inputs.

Just as a side note, the only way I’ve been able to persist the :__struct__ field with Jason is by explicitly mentioning it in the only: option of the derive, like so:

@derive {Jason.Encoder, only: [:__struct__, etc.]} 
embedded_schema do...

Is there any other way to force the encoding of “hidden” fields without necessarily implementing Jason from scratch?

Thanks again.

Post #2

dimitarvp

There’s two sides to this:

You might not be experienced enough with the technology today so these things aren’t coming naturally to you… yet.
You might be coming from other technologies where the frameworks are more automatic and hands-off and the cognitive dissonance of “do I have to do it myself?!” might be getting to you.

Both are possible to overcome.

None that I am aware of but that solution is quite fine – you are opting out of a sane default behaviour and saying “I know what I am doing, please serialize these fields exactly as I say”.

Post #3