ExAws.Dynamo.Encoder and Ecto.Schema do not work together

michaelwa

My mix.exs dependencies (partial list):
{:ex_aws, “~> 2.1”},
{:ex_aws_dynamo, “~> 4.0”},
{:hackney, “~> 1.9”},
{:configparser_ex, “~> 4.0”}I created an Ecto.Schema:

I can create dyanmo tables, insert, update, and delete records just fine as long as the data records are derived from generic structs.

However, when I create an Ecto.Schema

defmodule IntandemWww.Users.User do
use Ecto.Schema
import Ecto.Changeset

@derive [ExAws.Dynamo.Encodable]

schema “users” do
field :name, :string, virtual: true
field :first_name, :string
field :middle_name, :string
field :last_name, :string
field :condition, :string
field :email, :string
field :phone, :string
field :status, :string
field :mentor, :string
field :enrollment_date, :string

timestamps()

end
end

Instance looks like this (note the “meta” attribute):

%Users.User{
meta: #Ecto.Schema.Metadata<:built, “users”>,
id: nil,
name: nil,
first_name: nil,
middle_name: nil,
last_name: nil,
condition: nil,
email: nil,
phone: nil,
status: nil,
mentor: nil,
enrollment_date: nil,
inserted_at: nil,
updated_at: nil
}

Then when I try to Encode it (with some actual data set though):

Dynamo.Encoder.encode!(new_user)

I get the following error:

** (Protocol.UndefinedError) protocol ExAws.Dynamo.Encodable not implemented for #Ecto.Schema.Metadata<:built, “users”> of type Ecto.Schema.Metadata (a struct). This protocol is implemented for the following type(s): Atom, BitString, Float, HashDict, Users.User, Integer, List, Map, MapSet, User
(ex_aws_dynamo 4.2.1) lib/ex_aws/dynamo/encodable.ex:1: ExAws.Dynamo.Encodable.impl_for!/1
(ex_aws_dynamo 4.2.1) lib/ex_aws/dynamo/encodable.ex:5: ExAws.Dynamo.Encodable.encode/2
(ex_aws_dynamo 4.2.1) lib/ex_aws/dynamo/encodable.ex:87: anonymous fn/2 in ExAws.Dynamo.Encodable.Map.do_encode/1
(stdlib 4.0.1) maps.erl:411: :maps.fold_1/3
(ex_aws_dynamo 4.2.1) lib/ex_aws/dynamo/encodable.ex:64: ExAws.Dynamo.Encodable.Map.encode/2

I looked through the encoder.ex in the ex_aws_dynamo dependency and the “meta” field is obviously not accounted for during encoding.

Is anyone using Ecto.Schema with Dynamo and if so how? Is this an oversight or were they not meant to ever work together?

I do not see any usage of the “options” that are passed to “encode” functions. Would a valid solution be to add “ignore_fields” as an array of fields to skip over?

5 comments

#ecto #dynamodb #data_schema

0 711 5

2023-03-03 16:58:06 UTC

First 5 of 5 Posts

RudManusachi

I never used those libs by myself, however, I see that ExAws.Dynamo.Encodable accepts option :only
https://github.com/ex-aws/ex_aws_dynamo/blob/2e523c9efb81e41c13da8b62859296b5b60d3991/lib/ex_aws/dynamo/encodable.ex#L38-L44

So perhaps something like:

@derive {ExAws.Dynamo.Encodable, only: [:name, :email, ...]}

schema "users" do
  field :name ...

would do the trick?

Post #1

michaelwa

Thank you!! Thank you! This gets me moving forward again!!

This worked!

@derive {ExAws.Dynamo.Encodable, only: [:name, :first_name, :middle_name, :last_name, :condition, :email, :phone, :status, :mentor, :enrollment_date, :inserted_at, :updated_at]}

I note that right below:

def do_encode(map, only: only) do

That there is an “except”:

def do_encode(map, except: except) do

I tried the three variants below. They all compile but no happiness!

@derive {ExAws.Dynamo.Encodable, except: [Ecto.Schema.Metadata]}
@derive {ExAws.Dynamo.Encodable, except: [:__meta__]}
@derive {ExAws.Dynamo.Encodable, except: ["__meta__"]}

I also see in the Ecto.Schema module the following:

Module.put_attribute(__MODULE__, :ecto_struct_fields, {:__meta__, meta})

So I would have guessed the :atom version should have worked.

Post #2

michaelwa

added note: the forum altered my second post. the “:meta” is surrounded by double underscores in my code.

Post #3

al2o3cr

(Note: I edited your post to use ``` to preserve code-formatting, underscores etc)

The encode function accepts an except option, but that’s distinct from the code generated when you say @derive {ExAws.Dynamo.Encodable, etc etc etc}.

https://github.com/ex-aws/ex_aws_dynamo/blob/2e523c9efb81e41c13da8b62859296b5b60d3991/lib/ex_aws/dynamo/encodable.ex#L38-L52

What you want would be something like:

extractor =
  cond
    only = options[:only] ->
      quote(do: Map.take(struct, unquote(only)))

    except = options[:except] ->
      quote(do: :maps.without(unquote(except), struct))

    true ->
      quote(do: :maps.remove(:__struct__, struct))

and then you could say:

# NOTE TO FUTURE GOOGLERS: THIS LINE REQUIRES THE CHANGE ABOVE!
@derive {ExAws.Dynamo.Encodable, except: [:__struct__, :__meta__]}

This could make a good PR to ExAws.Dynamo, if you added some tests. Maybe :except should automatically add :__struct__

Post #4

michaelwa

That is a good idea. I am kind of a newbie but i do want to contribute. I’ll look at it this weekend and see if I think can come up with the solution.

Once again, thank you for you support!!

Post #5

— All posts loaded —