How to convert Ecto structs/changesets <=> maps with :source options and custom types?

For simple maps and schemas, I understand how to convert maps into Ecto schemas.
Now, I want to convert a map into a Ecto struct which does not have the same fields, and have custom types.


The Schema I have

defmodule AuthInfo do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key false
  schema "UserTable" do
    field :user_id, :string, source: :pk
    field :info, ProviderAccountID, source: :sk
defmodule ProviderAccountID do
  @moduledoc """
  Struct that holds provider and its account_id.
  @encode_prefix "AUTHINFO"
  @type provider :: :twitter | :facebook | :google
  @type t :: %__MODULE__{provider: provider, id: String.t()}
  defstruct [:provider, :id]

  @spec new(any, any) :: {:ok, t} | {:error, reason :: String.t()}
  def new(provider, _id) when not (provider in [:twitter]), do: {:error, "unacceptable provider: #{provider}"}
  def new(provider, id), do: {:ok, %__MODULE__{provider: provider, id: id |> to_string()}}

  use Ecto.Type

  def type, do: :string

  def cast(%__MODULE__{} = data), do: {:ok, data}
  def cast({provider, id}), do: new(provider, id)
  def cast(nil), do: nil
  def cast(_), do: :error

  def load(@encode_prefix <> "#" <> body) do
    case body |> String.split("#") do
      [provider | [id | _]] -> new(provider |> String.to_existing_atom(), id)
      _ -> :error

  def load(""), do: nil
  def load(nil), do: nil

  def dump(%__MODULE__{provider: p, id: id}), do: {:ok, "#{@encode_prefix}##{p}##{id}"}
  def dump(nil), do: {:ok, ""}
  def dump(_), do: :error

and what I want to do:

iex> %{pk: "user_1", sk: "AUTHINFO#twitter#12345678"}
...> |> convert_to_struct()
%AuthInfo{__meta__: #Ecto.Schema.Metadata<:built, "UserTable">,
   info: %ProviderAccountID{provider: :twitter, id: "123456878"},
   user_id: "user_1"}
iex> %AuthInfo{__meta__: #Ecto.Schema.Metadata<:built, "UserTable">,
   info: %ProviderAccountID{provider: :twitter, id: "123456878"},
   user_id: "user_1"}
...> |> convert_to_map()
%{pk: "user_1", sk: "AUTHINFO#twitter#12345678"}

or maybe convert Ecto.Changeset into maps like above directly.

What I get currently

def parse(map) do
  |> cast(map, [:pk, :sk])
  |> validate_required([:user_id, :info])
  |> apply_changes()
iex> .AuthInfo.parse(%{pk: "user_1", sk: "AUTHINFO#twitter#12345678"})    
** (ArgumentError) unknown field `:pk` given to cast. Either the field does not exist or it is a :through association (which are read-only). The known fields are: :info, :user_id
    (ecto 3.5.8) lib/ecto/changeset.ex:551: Ecto.Changeset.cast_type!/2
    (ecto 3.5.8) lib/ecto/changeset.ex:520: Ecto.Changeset.process_param/7
    (elixir 1.11.3) lib/enum.ex:2193: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto 3.5.8) lib/ecto/changeset.ex:506: Ecto.Changeset.cast/6
    lib/auth_info.ex:37: AuthInfo.parse/1 

It’s okay to implement the whole conversion myself (maybe by using __schema__(:field_source, field)), but if I’m missing predefined features or any open source products, I’m happy to know.

def convert_to_struct(struct_name, map) do
  struct(struct_name, map)

def convert_to_map(%model{} = schema) do
  Map.take(schema, model.__schema__(:fields))

Your changeset will not work, as in cast/3 you need to use schema field names, not backend field names (so it should be [:user_id, :info] instead of [:pk, :sk]). If you want to use “backend field names” then you probably should look at Ecto.Repo.load/2.

1 Like

Thanks hauleth,

For Ecto.Repo.load, I think I need to create a custom adapter which takes maps as a backend, maybe?

Since Ecto.Repo has no dump function, and seems a bit off for just converting between maps and schema structs, and also it is a huuuge work to create an adapter, I’ve landed creating a function using __schema__/2 and Ecto.Type.load/3.

For people who came to see afterwards:

def __to_map__(%__MODULE__{} = data) do
    &{__MODULE__.__schema__(:field_source, &1), &1, __MODULE__.__schema__(:type, &1)}
  |> {map_key, str_field, type} ->
    {map_key, data |> Map.fetch!(str_field), type}
  |> {map_key, str_value, type} ->
    {map_key, Ecto.Type.dump(type, str_value)}
    {map_key, {:ok, value}} -> {map_key, value}
  |> Enum.into(%{})

def __from_map__(map) do
  map =
      &{&1, __MODULE__.__schema__(:field_source, &1), __MODULE__.__schema__(:type, &1)}
    |> {str_field, map_key, type} -> {str_field, map[map_key], type} end)
    |> {str_field, orig_value, type} ->
      {str_field, Ecto.Type.load(type, orig_value)}
      {str_field, {:ok, value}} -> {str_field, value}
    |> Enum.into(%{})
  struct!(__MODULE__, map)

This doesn’t handle Ecto.Type.load/3 and Ecto.Type.dump/2's errors yet, but I think the principle is correct;
And I assume embedded schemas will not work also.

I inserted these functions via the __before_compile__ macro and use it from a utility module

def dump(%mod{} = data), do: data |> mod.__to_dynamo_map__()
def load(map, mod), do: map |> mod.__from_map__()

and now I’m happy!

iex(1)> %{pk: "user_1", sk: "AUTHINFO#twitter#1234qwer"} |> Helper.Schema.load(AuthInfo)
  __meta__: #Ecto.Schema.Metadata<:built, "UserTable">,
  info: %ProviderAccountID{
    id: "1234qwer",
    provider: :twitter
  inserted_at: nil,
  updated_at: nil,
  user_id: "user_1"
iex(2)> v() |> DynamoHelper.Schema.dump
  inserted_at: nil,
  pk: "user_1",
  sk: "AUTHINFO#twitter#1234qwer",
  updated_at: nil

(Doesn’t anyone else does this though? I couldn’t reach out any libraries for something like this)

still reaching out for enhancements!