How to normalize function params

I believe this is what you’re looking for: https://medium.com/very-big-things/towards-maintainable-elixir-the-core-and-the-interface-c267f0da43#1572

From my understanding, he just extracted the previous version of register into the normalize function.

As mentioned in the article, the author is suggesting using Ecto’ schemaless changesets for this. I think the normalize function can be written in a pretty generic way.

Thank @vrcca & @stefanchrobot for your pointers. I was able to recreate this function. I’ll leave it here for anyone who may find this post later.

If you see anyway I can improve this code, please let me know

defmodule Timetask.Helpers.Normalizer do
  import Ecto.Changeset

  @doc """
  Normalizes and validates that `params` is formed according to `schema`.

  ## Examples

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{name: "only required field"})
      {:ok, %{count: 10, name: "only required field"}}

      iex> normalized_params = Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{description: "has no name"})
      ...> {:error, %{errors: errors}} = normalized_params
      ...> assert Keyword.has_key?(errors, :name)

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: "I'm a string"
      ...>  }, %{name: "Use atoms of tuple"})
      ** (ArgumentError) Bad formed schema

  """
  @spec normalize(map, map) :: {:error, Ecto.Changeset.t()} | {:ok, map}
  def normalize(%{} = schema, %{} = params) do
    required_fields =
      Enum.reject(schema, fn {_, v} ->
        with true <- is_tuple(v),
             {_, opts} = v,
             true <- Keyword.has_key?(opts, :required) do
          false
        else
          _ -> true
        end
      end)
      |> Enum.map(fn {k, _} -> k end)

    defaults =
      schema
      |> Enum.filter(fn {_, v} ->
        with true <- is_tuple(v),
             {_, opts} = v,
             true <- Keyword.has_key?(opts, :default) do
          true
        else
          _ -> false
        end
      end)
      |> Enum.into(%{}, fn {k, {_, opts}} ->
        {k, Keyword.get(opts, :default)}
      end)

    normalized_schema =
      schema
      |> Enum.map(fn {k, v} ->
        cond do
          is_tuple(v) -> {k, elem(v, 0)}
          is_atom(v) -> {k, v}
          true -> raise(ArgumentError, "Bad formed schema")
        end
      end)
      |> Enum.into(%{})

    {defaults, normalized_schema}
    |> cast(params, Map.keys(normalized_schema))
    |> validate_required(required_fields)
    |> apply_action(:insert)
  end
end

Yes, on both accounts. I can’t share the code (it’s owned by the clients), but the basic take is something like

def normalize(data, types) do
  {%{}, types}
  |> Ecto.Changeset.cast(data, Map.keys(types))
  |> Ecto.Changeset.apply_action(:insert)
end

This is probably not enough (e.g. you may want to handle nils and missing keys), but it’s a solid start.

Just a few remarks if you don’t mind:

Enum.reject(schema, fn {_, v}

I’d go with something longer and less generic than k (key?) and v (value?). Maybe {field, schema}? Plus in this case I think it makes sense to use {_k, v} to explain the meaning of the first element.

cond do
  is_tuple(v) -> {k, elem(v, 0)}
  is_atom(v) -> {k, v}
  true -> raise(ArgumentError, "Bad formed schema")
end

Using more pattern matching would be more idiomatic:

case v do
  {type, _opts} when is_atom(type) -> {k, type}
  type when is_atom(type) -> {k, type}
  _ -> raise ArgumentError, "bad schema: #{inspect(v)}"
end

And one more thing:

schema |> Enum.map(fn {k, v} -> ... end) |> Enum.into(%{})

can be replaced with:

Map.new(schema, fn {k, v} -> ... end)

I wanted to reply on Twitter but I saw there is already a solution in here. I’d like to share my anyway
Since we need to traverse the schema many times, I normalize it first and then use list comprehensions like this:

  def parse(params, schema) do
    # First I want to have entire schema in one format [{key, {type, opts}}]
    normalized_schema = for {key, type_spec} <- schema, do: {key, apply_default_opts(type_spec)}
    keys = for {key, _} <- normalized_schema, do: key
    types = for {key, {type, _opts}} <- normalized_schema, into: %{}, do: {key, type}
    required_fields = for {key, {_type, opts}} <- normalized_schema, Keyword.get(opts, :required), do: key
    defaults = for {key, {_type, opts}} <- normalized_schema, default = Keyword.get(opts, :default), into: %{}, do: {key, default}

    {defaults, types}
    |> cast(params, keys)
    |> validate_required(required_fields)
    |> apply_action(:normalize)
  end

  @default_opts [required: false]
  defp apply_default_opts(type) when is_atom(type), do: {type, @default_opts}
  defp apply_default_opts({type, opts}), do: {type, Keyword.merge(@default_opts, opts)}

The line computing defaults might be tricky to understand because it is long and introduces a variable inside the comprehension filter.
Also, I’d vote for naming that function “parse” instead of normalize. Parsing is an action of taking a bunch of data and trying to transform it into a structure that is understandable. In the spirit of Parse, don’t validate

It would be possible to push the schema specification even more to introduce validations like:
email: {:string, format: ~r/@/}
and then call Changeset.validate_format(changeset, key, format) for each. But that would require another option for each Ecto.Changeset validatior. It might be an overkill but the schema format is very readable so it is tempting

Noted. Definitely it’s a little cryptic.

Oh yeah, the code looks much better.

I’m not convinced of this point. I’ve taken most of your ideas, but I kinda like normalize. To me parse means the data is going to change type.

Anyways, here is the revisited version with my take on @stefanchrobot and @tomekowal suggestions

defmodule Timetask.Helpers.Normalizer do
  import Ecto.Changeset

  @doc """
  Normalizes and validates that `params` is formed according to `schema`.

  ## Examples

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{name: "only required field"})
      {:ok, %{count: 10, name: "only required field"}}

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{"name" => "Also accepts strings as key"})
      {:ok, %{count: 10, name: "Also accepts strings as key"}}

      iex> normalized_params = Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{description: "has no name"})
      ...> {:error, %{errors: errors}} = normalized_params
      ...> assert Keyword.has_key?(errors, :name)

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: "I'm a string"
      ...>  }, %{name: "Use atom or tuple"})
      ** (ArgumentError) Bad formed schema

  """
  @spec normalize(map, map) :: {:error, Ecto.Changeset.t()} | {:ok, map}
  def normalize(%{} = schema, %{} = params) do
    normalized_schema =
      for {field_name, type_spec} <- schema,
          do: {field_name, apply_default_opts(type_spec)}

    defaults =
      for {field_name, {_type, opts}} <- normalized_schema,
          default = Keyword.get(opts, :default),
          into: %{},
          do: {field_name, default}

    types =
      for {field_name, {type, _opts}} <- normalized_schema, into: %{}, do: {field_name, type}

    normalized_params =
      for {param_name, param_value} <- params, into: %{}, do: {to_atom(param_name), param_value}

    fields = for {field_name, _} <- normalized_schema, do: field_name

    required_fields =
      for {field_name, {_type, opts}} <- normalized_schema,
          Keyword.get(opts, :required),
          do: field_name

    {defaults, types}
    |> cast(normalized_params, fields)
    |> validate_required(required_fields)
    |> apply_action(:normalize)
  end

  defp to_atom(name) when is_atom(name), do: name
  defp to_atom(name) when is_bitstring(name), do: String.to_atom(name)
  defp to_atom(_), do: raise(ArgumentError, "Bad formed schema")

  @default_opts [required: false]
  defp apply_default_opts(type) when is_atom(type), do: {type, @default_opts}
  defp apply_default_opts({type, opts}), do: {type, Keyword.merge(@default_opts, opts)}
  defp apply_default_opts(_), do: raise(ArgumentError, "Bad formed schema")
end

I’d change String.to_atom by String.to_existing_atom, since this data comes from untrusted source.

Good catch. I dindn’t know about that function

I’d be intersted in how you handled the return value of the core, especially for errors. It might be easy if db schema and normalized schema are similar, but becomes tricky when that’s not the case.