Complex data mapping and transformation

Questions / Help
1

Hi all,

I’ve just read the Ecto.Schema docs. While I appreciate the tools at our disposal to define the various structs to map to different sources and destination, I fail to understand how to do complex data mapping.

For example, imagine that José purchased something with my frontend which then sends this to my backend:

%{
    customer: %{
        email: "jose@valim.com",
        firstname: "José",
        lastname: "Valim",
        address: %{
            first_line: "1 Picadilly Circus",
            second_line: "",
            city: "London",
            postcode: "SE1GE2",
            country: "UK"
        }
    },
    payment: %{
        type: "credit_card",
        status: "success",
        amount: 5403,
        transaction_id: "f8usdefkljnds902lfsdjkljjj20"
    }
    basket: %{
        products: [
            %{
                id: 123,
                quantity: 3,
                discount_code: "J05E_R0CK5_2019"
            },
            %{
                id: 29,
                quantity: 1,
                discount_code: ""
            }
        ]
    }
}
```
One of the duties of my backend is to update our CRM with some of this data. Following is the data I need to extract, transform and load into the CRM based on the previous example:

```
%{
    name: "José Valim",
    address: "1 Picadilly Circus, London SE1GE2, UK",
    purchase_amount: 5403,
}
```

I'm thinking that Ecto could shine here. I could define an embedded schema for the second shape and cast the received data into it but
1. how is Ecto helping to extract `customer.firstname` and `customer.lastname`, concat them and put the result into `name`?
2. how is Ecto helping to take all address fields from the purchase and join them into a single string?
3. how is Ecto helping to put `payment.amount` into `purchase_amount`?

As you can see now, there are multiple issues: the keys don't map because the name don't match and because they are also sometimes nested differently, sometimes a transformation is necessary (concat X Y and Z).
2

Once you’ve casted your scary external parameters to a nicely validated shape with embedded_schemas it can be passed into a constructor function of another embedded schema or just a struct definition that describes your CRM fields.

defmodule Shopping.CRMCustomer do
  alias Shopping.Events.OrderPurchased
  defstruct [
    :name,
    :address,
    :purchase_amount,
  ]

  def new(%OrderPurchased{} = event) do
    %__MODULE__{
      name: "#{event.customer.first_name} #{event.customer.first_name}",
      address: event.customer.address,
      purchase_amount: event.transaction.amount,
    }
  end
end

I’ve mapped it from an OrderPurchased struct type but it really doesn’t matter so long as the data from the front end has been casted and validated into a known shape you can match on.

Also check out this article for some more examples: https://medium.com/coingaming/elixir-structs-for-the-order-5c45be57c5c9

3 Likes
3

Ecto is great in casting external data, but it doesn’t really do structural changes of data. So either cast things in the format you receive them in and do the mapping to a different structure afterwards or you first cleanup the data you receive without ecto and then cast things into schemas.

3 Likes
4

I am not sure I understand how this relates to Ecto. It never promises to do structural transformations for you.

A quick and naive approach I’d take is this:

defmodule Util do
  def nil_if_empty(x) when is_binary(x) do
    case String.length(x) do
      0 -> nil
      _ -> x
    end
  end

  def join_strings(list, sep) when is_list(list) do
    list
    |> Enum.map(&nil_if_empty/1)
    |> Enum.reject(&is_nil/1)
    |> Enum.join(sep)
  end
end

defmodule Customer do
  def full_name(first, last), do: Util.join_strings([first, last], " ")
end

defmodule Address do
  def full(%{
        first_line: l1,
        second_line: l2,
        city: city,
        postcode: zip,
        country: country
      }) do
    [
      Util.join_strings([l1, l2], " "),
      Util.join_strings([city, zip], " "),
      country
    ]
    |> Util.join_strings(", ")
  end
end

defmodule Order.ERP do
  def from_order(%{
        customer: %{
          firstname: first,
          lastname: last,
          address: addr
        },
        payment: %{
          amount: amount
        }
      }) do
    %{
      name: Customer.full_name(first, last),
      address: Address.full(addr),
      purchase_amount: amount
    }
  end

  def new(),
    do: %{
      name: "Jose Valim",
      address: "1 Picadilly Circus, London SE1GE2, UK",
      purchase_amount: 5403
    }

  def old(),
    do: %{
      customer: %{
        email: "jose@valim.com",
        firstname: "Jose",
        lastname: "Valim",
        address: %{
          first_line: "1 Picadilly Circus",
          second_line: "",
          city: "London",
          postcode: "SE1GE2",
          country: "UK"
        }
      },
      payment: %{
        type: "credit_card",
        status: "success",
        amount: 5403,
        transaction_id: "f8usdefkljnds902lfsdjkljjj20"
      },
      basket: %{
        products: [
          %{
            id: 123,
            quantity: 3,
            discount_code: "J05E_R0CK5_2019"
          },
          %{
            id: 29,
            quantity: 1,
            discount_code: ""
          }
        ]
      }
    }
end

You can then try this in iex:

Order.ERP.from_order(Order.ERP.old()) == Order.ERP.new()

It yields true on my machine.

5

Thanks all