svarlet

Complex data mapping and transformation

Hi all,

I’ve just read the Ecto.Schema docs. While I appreciate the tools at our disposal to define the various structs to map to different sources and destination, I fail to understand how to do complex data mapping.

For example, imagine that José purchased something with my frontend which then sends this to my backend:

%{
    customer: %{
        email: "jose@valim.com",
        firstname: "José",
        lastname: "Valim",
        address: %{
            first_line: "1 Picadilly Circus",
            second_line: "",
            city: "London",
            postcode: "SE1GE2",
            country: "UK"
        }
    },
    payment: %{
        type: "credit_card",
        status: "success",
        amount: 5403,
        transaction_id: "f8usdefkljnds902lfsdjkljjj20"
    }
    basket: %{
        products: [
            %{
                id: 123,
                quantity: 3,
                discount_code: "J05E_R0CK5_2019"
            },
            %{
                id: 29,
                quantity: 1,
                discount_code: ""
            }
        ]
    }
}
```
One of the duties of my backend is to update our CRM with some of this data. Following is the data I need to extract, transform and load into the CRM based on the previous example:

```
%{
    name: "José Valim",
    address: "1 Picadilly Circus, London SE1GE2, UK",
    purchase_amount: 5403,
}
```

I'm thinking that Ecto could shine here. I could define an embedded schema for the second shape and cast the received data into it but
1. how is Ecto helping to extract `customer.firstname` and `customer.lastname`, concat them and put the result into `name`?
2. how is Ecto helping to take all address fields from the purchase and join them into a single string?
3. how is Ecto helping to put `payment.amount` into `purchase_amount`?

As you can see now, there are multiple issues: the keys don't map because the name don't match and because they are also sometimes nested differently, sometimes a transformation is necessary (concat X Y and Z).

4 comments

#ecto

0 1542 4

2019-02-03 09:01:25 UTC

Most Liked

MrDoops

Once you’ve casted your scary external parameters to a nicely validated shape with embedded_schemas it can be passed into a constructor function of another embedded schema or just a struct definition that describes your CRM fields.

defmodule Shopping.CRMCustomer do
  alias Shopping.Events.OrderPurchased
  defstruct [
    :name,
    :address,
    :purchase_amount,
  ]

  def new(%OrderPurchased{} = event) do
    %__MODULE__{
      name: "#{event.customer.first_name} #{event.customer.first_name}",
      address: event.customer.address,
      purchase_amount: event.transaction.amount,
    }
  end
end

I’ve mapped it from an OrderPurchased struct type but it really doesn’t matter so long as the data from the front end has been casted and validated into a known shape you can match on.

Also check out this article for some more examples: https://medium.com/coingaming/elixir-structs-for-the-order-5c45be57c5c9

Post #2

LostKobrakai

Ecto is great in casting external data, but it doesn’t really do structural changes of data. So either cast things in the format you receive them in and do the mapping to a different structure afterwards or you first cleanup the data you receive without ecto and then cast things into schemas.

Post #3