How to normalize function params

If you are up to a declarative approach and dependency inversion towards your application core’s data types, you can use Domo library. That generates validators and constructor functions from a t() type spec of the struct, and the type spec is validated for syntax correctness by elixir during the compilation.

So the request can be accepted into the struct that looks like the following:

defmodule Request do
  use Domo, ensure_struct_defaults: false

  defstruct [:email, :password, :date_of_birth]

  @type t :: %__MODULE__{
    email: email(),
    password: password(),
    date_of_birth: Date.t() | nil
  }

  @type email :: String.t()
  precond email: &String.match?(&1, ~r|.+\@.+\..+|)

  @type password :: String.t()
  precond password: &String.length(&1) > 7

  # Domo adds new/1, new!/1, ensure_type/1, ensure_type!/1 here automatically
end

And the validation can be done like that:

iex(1)> Request.new(%{email: "user@test.com", password: "some_password"})
{:ok, %Request{date_of_birth: nil, email: "user@test.com", password: "some_password"}}

iex(2)> Request.new(%{email: "usertestcom", date_of_birth: "none"})      
{:error,
 [
   password: "Invalid value nil for field :password of %Request{}. Expected the value matching the <<_::_*8>> type.",
   email: "Invalid value \"usertestcom\" for field :email of %Request{}. Expected the value matching the <<_::_*8>> type. And a true value from the precondition function \"&String.match?(&1, ~r|.+\\\\@.+\\\\..+|)\" defined for Request.email() type.",
   date_of_birth: "Invalid value \"none\" for field :date_of_birth of %Request{}. Expected the value matching the %Date{} | nil type."
 ]}

The dependency inversion can be done by sharing email() and password() in the shared module to have the same rules for emails and passwords across the whole app.

Domo plays nicely with Ecto schemas because they are structs too. See Domo.Changeset for this kind of integration.

IMO, the thing we’re validating is the input itself. In the simplest case (which is in my experience also the most frequent one), we can do most of the validations before even hitting the database, so that’s simple. Occasionally I may need to hit the database to validate some constraint (most often uniqueness). In most cases I’ve had this is also simple, since the field being validated usually directly corresponds to the input field. Combining these two, in most cases all I needed was a single validation+store changeset where all possible field errors corresponded to the input fields.

I can’t recall a single situation where this didn’t fit the bill, but vaguely speaking the options I’d consider in such cases would be:

1. Separate input validation changeset from db store, performing most of validations on the input.
2. On db store error, take the errors, change the keys if needed (e.g. replace db field :foo with input field :bar).

It would help if you had some specific situation in mind, then we could discuss it.

if you want to use something more standards-compliant, I have this library which does compile-time generation of strictly validation functions from Jsonschemas:

It doesn’t normalize your parameters, though, so if you need a translation layer between things with different “names” (e.g. camelCase → snake_case) you might have to cook up something on your own. We do this at work with a “Codec” module which I may do a webcast on sometime.

With Ecto, though, you can just peddle in strings and it figures out the pesky strings/atoms stuff for you.

Thanks! This seems to be exactly what I was looking for. I’ll test it and possibly mark it as the solution. The quest of building/understanding this was an interesting one thou.

I’ve not used it myself, but the tarams library looks like it might handle your use case if you want to go with a library instead of a custom solution GitHub - bluzky/tarams: Cast and validate external data and request parameters for Elixir and Phoenix · GitHub

I’d be intersted in how you handled the return value of the core, especially for errors. It might be easy if db schema and normalized schema are similar, but becomes tricky when that’s not the case.

Good catch. I dindn’t know about that function

I’d change String.to_atom by String.to_existing_atom, since this data comes from untrusted source.

Noted. Definitely it’s a little cryptic.

Oh yeah, the code looks much better.

I’m not convinced of this point. I’ve taken most of your ideas, but I kinda like normalize. To me parse means the data is going to change type.

Anyways, here is the revisited version with my take on @stefanchrobot and @tomekowal suggestions

defmodule Timetask.Helpers.Normalizer do
  import Ecto.Changeset

  @doc """
  Normalizes and validates that `params` is formed according to `schema`.

  ## Examples

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{name: "only required field"})
      {:ok, %{count: 10, name: "only required field"}}

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{"name" => "Also accepts strings as key"})
      {:ok, %{count: 10, name: "Also accepts strings as key"}}

      iex> normalized_params = Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{description: "has no name"})
      ...> {:error, %{errors: errors}} = normalized_params
      ...> assert Keyword.has_key?(errors, :name)

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: "I'm a string"
      ...>  }, %{name: "Use atom or tuple"})
      ** (ArgumentError) Bad formed schema

  """
  @spec normalize(map, map) :: {:error, Ecto.Changeset.t()} | {:ok, map}
  def normalize(%{} = schema, %{} = params) do
    normalized_schema =
      for {field_name, type_spec} <- schema,
          do: {field_name, apply_default_opts(type_spec)}

    defaults =
      for {field_name, {_type, opts}} <- normalized_schema,
          default = Keyword.get(opts, :default),
          into: %{},
          do: {field_name, default}

    types =
      for {field_name, {type, _opts}} <- normalized_schema, into: %{}, do: {field_name, type}

    normalized_params =
      for {param_name, param_value} <- params, into: %{}, do: {to_atom(param_name), param_value}

    fields = for {field_name, _} <- normalized_schema, do: field_name

    required_fields =
      for {field_name, {_type, opts}} <- normalized_schema,
          Keyword.get(opts, :required),
          do: field_name

    {defaults, types}
    |> cast(normalized_params, fields)
    |> validate_required(required_fields)
    |> apply_action(:normalize)
  end

  defp to_atom(name) when is_atom(name), do: name
  defp to_atom(name) when is_bitstring(name), do: String.to_atom(name)
  defp to_atom(_), do: raise(ArgumentError, "Bad formed schema")

  @default_opts [required: false]
  defp apply_default_opts(type) when is_atom(type), do: {type, @default_opts}
  defp apply_default_opts({type, opts}), do: {type, Keyword.merge(@default_opts, opts)}
  defp apply_default_opts(_), do: raise(ArgumentError, "Bad formed schema")
end

I wanted to reply on Twitter but I saw there is already a solution in here. I’d like to share my anyway
Since we need to traverse the schema many times, I normalize it first and then use list comprehensions like this:

  def parse(params, schema) do
    # First I want to have entire schema in one format [{key, {type, opts}}]
    normalized_schema = for {key, type_spec} <- schema, do: {key, apply_default_opts(type_spec)}
    keys = for {key, _} <- normalized_schema, do: key
    types = for {key, {type, _opts}} <- normalized_schema, into: %{}, do: {key, type}
    required_fields = for {key, {_type, opts}} <- normalized_schema, Keyword.get(opts, :required), do: key
    defaults = for {key, {_type, opts}} <- normalized_schema, default = Keyword.get(opts, :default), into: %{}, do: {key, default}

    {defaults, types}
    |> cast(params, keys)
    |> validate_required(required_fields)
    |> apply_action(:normalize)
  end

  @default_opts [required: false]
  defp apply_default_opts(type) when is_atom(type), do: {type, @default_opts}
  defp apply_default_opts({type, opts}), do: {type, Keyword.merge(@default_opts, opts)}

The line computing defaults might be tricky to understand because it is long and introduces a variable inside the comprehension filter.
Also, I’d vote for naming that function “parse” instead of normalize. Parsing is an action of taking a bunch of data and trying to transform it into a structure that is understandable. In the spirit of Parse, don’t validate

It would be possible to push the schema specification even more to introduce validations like:
email: {:string, format: ~r/@/}
and then call Changeset.validate_format(changeset, key, format) for each. But that would require another option for each Ecto.Changeset validatior. It might be an overkill but the schema format is very readable so it is tempting