Data_schema - declarative schemas for data transformations

Adzz

Data schemas are declarative descriptions of how to create a struct from some input data. You can set up different schemas to handle different kinds of input data. By default we assume the incoming data is a map, but you can configure schemas to work with any arbitrary data input including XML and json.

Data is selected from the input data and passed to a casting function before being set as a value under a key on the struct you want to build.

Check out the docs / guides and README for more detailed information on how it works but below is a flavour of what you can do.

A simple struct

First, let’s assume that your input data is a map with string keys. DataSchemas really shine when working with APIs because we can quickly convert an API response into trusted elixir data:

input = %{
  "content" => "This is a blog post",
  "comments" => [%{"text" => "This is a comment"},%{"text" => "This is another comment"}],
  "draft" => %{"content" => "This is a draft blog post"},
  "date" => "2021-11-11",
  "time" => "14:00:00",
  "metadata" => %{ "rating" => 0}
}

Now let’s define a schema to create a BlogPost struct from the above input data:

defmodule BlogPost do
  import DataSchema, only: [data_schema: 1]

  data_schema([
    field: {:content, "content", &BlogPost.to_okay_string/1},
  ])
  
  def to_okay_string(value) do
     {:ok, to_string(value)}
  end
end

The above is equivalent to:

defmodule StringType do
  @behaviour DataSchema.CastBehaviour

  @impl true
  def cast(value) do
    {:ok, to_string(value)}
  end
end

defmodule BlogPost do
  import DataSchema, only: [data_schema: 1]

  data_schema([
    field: {:content, "content", StringType},
  ])
end

Now you have defined your schema you can simple call DataSchema.to_struct/2:

DataSchema.to_struct(input, BlogPost)
# => %BlogPost{content: "This is a blog post"}

A more complex example

You can define a few kinds of fields, see the docs for more info but here is a more complex example introducing more field types:

  defmodule DraftPost do
    import DataSchema, only: [data_schema: 1]
    data_schema(field: {:content, "content", StringType})
  end

  defmodule Comment do
    import DataSchema, only: [data_schema: 1]
    data_schema(field: {:text, "text", StringType})
  end

  defmodule BlogPost do
    import DataSchema, only: [data_schema: 1]

    @mapping [
      field: {:date, "date", &Date.from_iso8601/1},
      field: {:time, "time", &Time.from_iso8601/1}
    ]
    data_schema(
      field: {:content, "content", &DataSchemaTest.to_stringg/1},
      has_many: {:comments, "comments", Comment},
      has_one: {:draft, "draft", DraftPost},
      list_of: {:list_of, "comments", &{:ok, &1["text"]} },
      aggregate: {:post_datetime, @mapping, &BlogPost.to_datetime/1}
    )

    def to_datetime(%{date: date, time: time}) do
      NaiveDateTime.new(date, time)
    end
  end

DataSchema.to_struct(input, BlogPost)
# The above returns:
{:ok, %DataSchemaTest.BlogPost{
  list_of: ["This is a comment", "This is another comment"],
  comments: [
    %DataSchemaTest.Comment{text: "This is a comment"},
    %DataSchemaTest.Comment{text: "This is another comment"}
  ],
  content: "This is a blog post",
  draft: %DataSchemaTest.DraftPost{content: "This is a draft blog post"},
  post_datetime: ~N[2021-11-11 14:00:00]
}}

Different Input Data - aka Are these not just embedded_schemas from ecto?

The examples so far have shown functionality that is very similar to what you can get from Ecto’s embedded schemas and data casting capabilities. However, in DataSchema we can also provide different data accessors. This allows us to defines schemas that can be casted from different input data, for example…

XML Schemas

Let’s imagine that we have some XML that we wish to turn into a struct. What would it require to enable that? First a new Xpath data accessor:

defmodule XpathAccessor do
  @behaviour DataSchema.DataAccessBehaviour
  import SweetXml, only: [sigil_x: 2]

  @impl true
  def field(data, path) do
    SweetXml.xpath(data, ~x"#{path}"s)
  end

  @impl true
  def list_of(data, path) do
    SweetXml.xpath(data, ~x"#{path}"l)
  end

  @impl true
  def has_one(data, path) do
    SweetXml.xpath(data, ~x"#{path}")
  end

  @impl true
  def has_many(data, path) do
    SweetXml.xpath(data, ~x"#{path}"l)
  end
end

Let’s define our schemas like so:

defmodule DraftPost do
  import DataSchema, only: [data_schema: 1]

  @data_accessor XpathAccessor
  data_schema([
    field: {:content, "./Content/text()", StringType}
  ])
end

defmodule Comment do
  import DataSchema, only: [data_schema: 1]

  @data_accessor XpathAccessor
  data_schema([
    field: {:text, "./text()", StringType}
  ])
end

defmodule BlogPost do
  import DataSchema, only: [data_schema: 1]

  @data_accessor XpathAccessor
  @datetime_fields [
    field: {:date, "/Blog/@date", &Date.from_iso8601/1},
    field: {:time, "/Blog/@time", &Time.from_iso8601/1},
  ]
  data_schema([
    field: {:content, "/Blog/Content/text()", StringType},
    has_many: {:comments, "//Comment", Comment},
    has_one: {:draft, "/Blog/Draft", DraftPost},
    aggregate: {:post_datetime, @datetime_fields, &NaiveDateTime.new(&1.date, &1.time)},
  ])
end

And now we can transform as above:

source_data = """
<Blog date="2021-11-11" time="14:00:00">
  <Content>This is a blog post</Content>
  <Comments>
    <Comment>This is a comment</Comment>
    <Comment>This is another comment</Comment>
  </Comments>
  <Draft>
    <Content>This is a draft blog post</Content>
  </Draft>
</Blog>
"""

DataSchema.to_struct(source_data, BlogPost)

# This will output:

{:ok, %BlogPost{
   comments: [
     %Comment{text: "This is a comment"},
     %Comment{text: "This is another comment"}
   ],
   content: "This is a blog post",
   draft: %DraftPost{content: "This is a draft blog post"},
   post_datetime: ~N[2021-11-11 14:00:00]
 }}

Data Accessor - An Access example.

Let’s look back at our map version.

input = %{
  "content" => "This is a blog post",
  "comments" => [%{"text" => "This is a comment"},%{"text" => "This is another comment"}],
  "draft" => %{"content" => "This is a draft blog post"},
  "date" => "2021-11-11",
  "time" => "14:00:00",
  "metadata" => %{ "rating" => 0}
}

We could define a data accessor that looks like this:

defmodule AccessDataAccessor do
  @behaviour DataSchema.DataAccessBehaviour

  @impl true
  def field(data, path) do
    get_in(data, path)
  end

  @impl true
  def list_of(data, path) do
    get_in(data, path)
  end

  @impl true
  def has_one(data, path) do
    get_in(data, path)
  end

  @impl true
  def has_many(data, path) do
    get_in(data, path)
  end
end

Now we can define our schema:

defmodule Blog do
  import DataSchema, only: [data_schema: 1]

  @data_accessor AccessDataAccessor
  data_schema([
    list_of: {:comments, ["comments", Access.all(), "text"], &{:ok, to_string(&1)}},
  ])
end

And create a struct from this:

input = %{
  "content" => "This is a blog post",
  "comments" => [%{"text" => "This is a comment"},%{"text" => "This is another comment"}],
  "draft" => %{"content" => "This is a draft blog post"},
  "date" => "2021-11-11",
  "time" => "14:00:00",
  "metadata" => %{ "rating" => 0}
}
DataSchema.to_struct(input, Blog)
# Returns:
{:ok, %Blog{comments: ["This is a comment", "This is another comment"]}}

This is still an early version. There are some planned upcoming features before a v1 but it is certainly useable as is.

17 comments

#api #library #maps #structs #xml #json #data #data_schema

15 3907 17

2022-09-29 21:51:46 UTC

Showing Posts 1 to 10

Show Best Posts
Show All Posts (oldest first)
Show All Posts (newest first)

Adzz

Update:

Livebooks added to the repo.

Post #1

Adzz

I’m also now realising I don’t think I ever actually linked to the repo:

https://github.com/Adzz/data_schema

Post #2

the_wildgoose

This looks interesting.

Can I run validations on my data?
Can I pass these structures to a phoenix form as a changeset compatible struct?

Post #3

Adzz

Great questions!

Validations

Can I run validations on my data?

Right now the focus is on parsing over validation. What I mean by that is instead of doing something like this:

input = %{"name" => ""}

input
|> DataSchema.to_struct(User)
|> validate_name_not_blank()

Or even:

input = %{"name" => ""}

input
|> validate_name_not_blank()
|> DataSchema.to_struct(User)

we can define our casting function to return an :error if it receives an empty string:

defmodule NonBlankString do
  @behaviour DataSchema.CastBehaviour

  @impl true
  def cast(""), do: {:error, "Field was blank!"}
  def cast(value), do: {:ok, to_string(value)}
end

defmodule User do
  import DataSchema, only: [data_schema: 1]

  data_schema([
    field: {:user, "user", NonBlankString}
  ])
end

My current take on validations is that they are for when you can’t design away the need for them (via making illegal states unrepresentable). So the idea is that the schema defines what is valid.

HOWEVER - as you can see in the above examples you could define your own functions before / after struct creation if you felt the need.

It’s possible that some validations can’t be expressed per field, in which case we could add some in the future.

Phoenix Forms

There is nothing specially added yet for phoenix forms, but off the top of my head there are a few ways you could approach it. One way is to use a schemaless changeset in the form:

  def index(conn, _params) do
    types = %{name: :string}
    user = %User{}
    changeset = Ecto.Changeset.change({user, types}, %{})
    render(conn, "index.html", changeset: changeset)
  end

# With a form like this
<%= form_for @changeset, Routes.user_path(@conn, :create), fn f -> %>
  <label>
    Name: <%= text_input f, :name %>
  </label>
  <%= submit "Submit" %>
<% end %>

Then when you post the form:

def create(conn, %{"user" => user_input}) do
  case DataSchema.to_struct(user_input, User) do
     {:error, error} -> ...
     {:ok, struct} -> ...
  end
end

We could possibly make this easier by supplying a function something like DataSchema.schemaless_changeset_from_schema(User):

  def index(conn, _params) do
    changeset = DataSchema.schemaless_changeset_from_schema(User)
    render(conn, "index.html", changeset: changeset)
  end

You’d also have to do the work of converting the error to a changeset error, which we could probably write some functions to help with, but it might be as easy as:

def create(conn, %{"user" => user_input}) do
  case DataSchema.to_struct(user_input, User) do
     {:error, %{errors: [{field,  message}]}} ->
       changeset =
         {%User{}, %{name: :string}}
         |> Ecto.Changeset.change(user_input)
         |> Ecto.Changeset.add_error(field, message)
         
         render(conn, changeset: changeset)
     {:ok, struct} ->
       render(...)
  end
end

My feel is that ecto might feel more natural, but open to the use case.

Post #4

Brainiac

Looks nice, though to add to the discussion here are some alternative libraries also in this space:

Post #5

Adzz

Thanks for sharing.

Like I say DataSchema could be used to help with phoenix forms but that isn’t where it shines because params in Phoenix forms are always maps with string keys.

A really good use case for DataSchema is talking to APIs. If the API is XML then we get ecto-like features for parsing that XML.

with {:ok, %{body: body, status: 200}} <- HTTPoison.post(request) do
  DataSchema.to_struct(body, MySchema)
end

Post #6

the_wildgoose

Just putting this out there, but… What I REALLY want is something similar to Ecto for JSON/XML…

Meaning I find myself needing to consume some JSON structure. eg:

{
    ....,
    "firewall": {
        "dnat": {
            "nat-in": [
                {
                    "dest": "loc:192.168.111.4",
                    "dport": "7",
                    "proto": "tcp",
                    "source": "net"
                },
                {
                    "dest": "loc:192.168.111.4:80",
                    "dport": "8080",
                    "origdest": "&ppp0",
                    "proto": "tcp",
                    "source": "net"
                }
            ]
        }
    },
    ....,
}

So this is a map of maps of maps, which contains an array of maps.

Now this snippet is part of a much larger JSON structure which has config for other stuff, ie there are other keys at the top level with their own trees under.

Now I want to parse chunks of this into Elixir structures, check that it’s valid before starting, present those to the user as some kind of phoenix/Liveview form, accept back the updated params and validate them (so that I can do instant errors on screen). Finally I want to be able to diff what changed from the original and re-apply it to the current JSON structure

Whilst I’ve been a little over specific on some of my own use case, I don’t think this is so different from a use case you likely have in mind: consume some API end point, present the details to the user, allow them to edit stuff, send the changes back to the API end point?

Things which might not be obvious from the above:

I need to validate fields in combination with each other, so certain params may only be valid if the :action is something specific
It’s REALLY boring mapping a map of maps of lists of maps into Ecto format… Ecto can only cope with representing database tables, so your map of maps ends up needing to become a list structure where you copy the keys in and out (think how you would represent it in an SQL database). It would be SOOO much easier if Ecto could understand something like a map structure in it’s “has_many” fields (yes you can do custom data types like {:map, string}, but then you lose the ability to use schemas and changesets on those fields)
I need to also validate the keys of the maps. They need to be sanitised and controlled for length, etc (as they may map to UI elements, etc)
I need some level of round trip ability
I need to “diff” the changes (so I can apply only the changes back upstream - I want to be granular if simultaneous changes were made to separate parts of the document. However, ideally I want to be able to re-run my “is it valid” after re-apply those changes as two edits might individually not clash, but there might be dependencies between the key values, eg we might have a section for the IP range of the local network and another section for the DHCP parameters, but we have a schema validation between the two as we enforce that one is within the same range as the other.

I solve this at the moment using ecto changesets. The shape of a changeset can cross chunks of the whole json document if needed to enforce cross schema changes. ie a changeset “plucks out” a bunch of fields from the JSON input, and kind of flattens them into the structures allowed within ecto (lists). Then we can run our nested validations, etc. Then unfortunately this needs another function to reverse this process as it’s not necessarily purely mechanical to reverse the original extraction. It’s also painful to represent maps of maps as these need flattening into lists with an id column to represent the map key names (and this reversing later)

What I desire is something like a JSON parser, coupled to a generic structure validator. Which in turn can be used in phoenix forms with functioning error handling (the phoenix error function is something you define, so it can work with any library which produces a validation output including some per field error term)

Does this sound like a direction you are heading in?

Post #7

Adzz

It’s tricky to know for sure without getting my head round your use case more but it feels like you can get a fair bit of what you want. I’d recommend having a play!

I would say that when validations come from a combination of fields, you can still wrap this up into a casting function. Let’s take a simple example, imagine you have to be over 18 to be an adult:

input = %{ age: 10, type: "adult" }

defmodule User do
  import DataSchema
  
  @type_fields [
     field: {:age, :age, SchemaInteger},
     field: {:type, :type, SchemaString}
  ]
  data_schema([
    aggregate: {:type, @type_fields, &User.type/1},
  ])

  def type(%{age: age, type: :adult}) when age < 18, do: :error
  def type(%{age: _, type: :adult}), do: {:ok, :adult}
end

Post #8

Adzz

New version released: V 0.2.3

0.2.3

Bug fix

Ensures we call Code.ensure_loaded? before checking if function is exported. This was causing problems when running tests.

0.2.2

Bug fix

We were not creating the nested errors correctly for has_many and has_one, now we do. We also were removing nils when they were allowed for :list_of, we now don’t.

0.2.1

Bug fix

Previously we could not use a :list_of field on an inline :aggregate field. This fixes that.

Post #9

Adzz

New Version Released!

Version 0.2.4:

Features

This release adds runtime schemas. Runtime schemas are schemas that are defined at runtime and allow for casting to existing structs or to a bare map instead of a struct. This makes it really easy to integrate with Ecto for example to save an XML response into a db.

See the livebook for more details: data_schema/livebooks/runtime_schemas.livemd at main · Adzz/data_schema · GitHub

Here is a small example of what is possible:

defmodule User do
  use Ecto.Schema

  schema "users" do
    field :name, :string
    field :age, :integer
  end

  def update_details_from_xml(user_id, xml) do
    schema = [
      field: {:name, "/Response/User/@name", &{:ok, &1}},
      field: {:age, "/Response/User/@age", &parse_in/1t}
    ]

    with {:ok, changes} <- DataSchema.to_struct(xml, %{}, schema, XpathAccessor),
      %User{} = user <- Repo.get(user_id, User),
      %{valid?: true} = changeset <- Ecto.Changeset.change(%User{}, changes) do
      Repo.update(changeset)
    end
  end

  defp parse_int(string) do 
    case Integer.parse(string)  do
      {int, _} -> {:ok, int}
      _error -> :error
    end
  end

end

xml = """
<Response>
  <User name="Jeff" age="12" />
</Response>
"""
User.update_details_from_xml("123", xml)

Post #10

Load more posts (7 remaining)