JSON input validation and `:source` on a `field` - Ecto without a Database

Questions / Help
1

Understanding that Ecto and Changesets were useful tools for input validation, I thought I would try to use them to validate some JSON data that my application receives. Here is an example of the JSON data after it is parsed:

%{
  "name" => "some_archive.tar.gz",
  "path" => ".",
  "repo" => "happy_fun_repo",
  "sha256" => "fb83ab7c2d4d90026a30b9479addae289aaf30ce53779205f65ddc72f23bcc2b",
  "updated" => "2019-03-01T15:05:10.807-06:00"
}

To handle this data I created this embedded_schema:

  defmodule MyApp.RepoItem do

  @primary_key false
  embedded_schema do
    field :name, :string
    field :repo, :string
    field :path, :string
    field :hash, :string, source: :sha256
  end

   …

Note the use of the source: option on the hash field. My intention here is that the "sha256" field in the JSON should map to the hash field in the resulting struct. I’m trying to use these values like this:

json_map = #<some code here that gets json stuff>

changeset = %MyApp.RepoItem{}
    |> Ecto.Changeset.cast(json_map, [:name, :repo, :path, :hash])

The resulting changeset has the :name, :repo, and :path fields, but it does not have the :hash field.

Is it the case that Ecto.Changeset.cast doesn’t take the source mapping into account (and that mapping is applied only when a database is involved) or do I have a problem with my code that I’ve not been able to track down?

2 Likes
2

According to the doc here this filed is only for db mapping. I believe you want something called Input Forms but schema is for data representation and source is just an utility to cover bad names in db.
Personally I just pass attrs though an extra function to transform the input to schema.

3

Yes, I just verified locally that the source param to field does not affect a cast operation. It would only affect the loading of a schema struct from the database. That makes it of no use for embedded_schemas. As @vlad.grb said, you should have a separate mapping step for inputs into Ecto.Changeset.cast

But I wonder if a PR would be accepted by the Ecto team to enable this mapping. That brings up some interesting possibilities such as a list of source fields in case you are mapping multiple external sources of data into the same embedded schema.

4

For my particular use case (and I don’t know how well this would work for Ecto in general) it would be nice if I could pass a source that was a function so it could do something like pull a value out of the nested JSON structure. I suppose that can be done just as well with a preprocessing step in the same pipeline though.

2 Likes
5

I have the same need and AFAICT setting the source option still has no effect on an embedded schema. Here’s the module with embedded schema:

defmodule MyMapper do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key false
  embedded_schema do
    field :some_id, :string, source: "id"
  end

  def from_json(data) do
    %__MODULE__{}
    |> cast(data, [:some_id])
    |> apply_changes
  end
end

Here’s how I’m testing it:

  test "maps field key from source option" do
    json = ~s({"id": "z3z3"}) |> Jason.decode!()
    payload = MyMapper.from_json(json)
    assert %MyMapper{some_id: "z3z3"} == payload
  end

This fails with %MyMapper{some_id: nil}.

Have there been any updates to best practices for mapping keys in an embedded_schema?