Goal - A parameter validation library based on Ecto

[martinthenth]

Hi fellow Elixirists!

About 8 months ago, I wrote a blog post on validating Phoenix controller parameters using Ecto Changesets in combination with Ecto Embedded Schemas (Using Ecto changesets for API request parameter validation).

Since then, I found that writing embedded schemas for validating parameters to be quite tedious. It required setting up new modules, a lot of boilerplate code, and complex file organisation.

So I decided to write a library that replaces the boilerplate code with a simple syntax, that can be written inside controllers, and is still based on Ecto Changesets to leverage its validation capabilities.

It’s called Goal, and it offers those capabilities with a nice syntax:

defmodule MyApp.SomeController do
  import Goal
  import Goal.Syntax

  def create(conn, params) do
    with {:ok, attrs} <- validate_params(params, schema()) do
      ...
    end
  end

  defp schema do
    defschema do
      required :uuid, :string, format: :uuid
      required :name, :string, min: 3, max: 3
      optional :age, :integer, min: 0, max: 120
      optional :gender, :enum, values: ["female", "male", "non-binary"]

      optional :data, :map do
        required :color, :string
        optional :money, :decimal
        optional :height, :float
      end
    end
  end
end

It features:

1. Compatibility with all primitive types from Ecto.Schema
2. Customisable regexes for email, password, and url formats, so you can maintain compatibility with your production system
3. Unlimited nested maps and lists of maps
4. trim and squish to trim and collapse whitespaces
5. optional and required arguments for defining optional and required fields
6. An extended version of Ecto.Changeset.traverse_errors/2 (aptly called Goal.Changeset.traverse_errors/2) that works with Goal and Ecto’s existing functionality

You can use Goal for validating Phoenix controller action parameters, but it will work with any data source as long as it’s represented in a map.

I aim for Goal to cover all parameter validations that Ecto.Changeset offers for database fields. I think I covered most with the 0.1.1 version; but if you’d like another validation, then please contribute or open an issue on GitHub.

It was inspired by Ruby’s dry-schema, and I had to borrow code from Ecto. Thank you for making such awesome libraries.

Feedback and suggestions are very welcome . For example, I was thinking about generating the chain of Ecto.Changeset functions at compile-time, to crunch out some additional usecs

[stefanchrobot]

This looks great! How about supporting this syntax:

defschema schema do
  required ...
end

I’d love to see the functionality of Goal incorporated into Ecto. Did you consider that? If so, there’s one thing to think about: the Goal schema seems to be controller-specific so it makes sense to have required, min, max, etc. as options. On the other hand Ecto schemas are reused across the changeset functions (each function has its own set of required/optional/validations). I still think it would make sense to support what Goal is using, maybe with a different macro (there’s schema and embedded_schema, so maybe validation_schema?).

[fuelen]

required :uuid, :string, format: :uuid

optional :gender, :enum, values: ["female", "male", "non-binary"]

if the library is based on Ecto, then why not to use built-in types? I mean this

required :uuid, Ecto.UUID

optional :gender, Ecto.Enum, values: ["female", "male", "non-binary"]

[martinthenth]

I felt it would be more intuitive to decouple the controller context from the database context (like, most people know uuid, but do they know Ecto.UUID?). That said, it can definitely be made to work like that in the library

Edit: I originally said using Ecto.UUID works, but it only works when using the validation syntax directly; with defschema it doesn’t work automatically.

[martinthenth]

Cool idea! Maybe the argument could be a function reference so the defp schema do wouldn’t be required

With Ecto I often find I am using custom validations and parsing, like with update_change/2. I suppose the schema fields could receive a function as an argument, it’s worth a thought

[fuelen]

to decouple the controller context from the database context

Ecto is not only about the database

people know uuid, but do they know Ecto.UUID?

people will know if guide them

[hlx]

Let me offer you a different opinion:

Let’s say, next to the controllers, you’d need to create something else, a GraphQL API for example. With this approach you’d have to implement the validating logic twice if I’m not mistaken.

What I usually do, in the controllers, is just check for the required fields and filter out fields that should not be there, and keep the validation like length and what not in the context. More of a normalize instead of validation.

That way your controllers and GraphQL schemas and etc, just need to check that the fields exists and the rest of the validation is shared.

Saša explains it better here: Towards Maintainable Elixir: The Core and the Interface | by Saša Jurić | Very Big Things | Medium

[martinthenth]

Goal 0.1.3 released

I recently switched to LiveViews and found that I have the same need to validate incoming parameters as with JSON APIs. Since LiveViews depend on changesets for validation, I decided to expose the Goal.build_changeset(params, schema) function in the library.

I find using database Ecto.Schemas in LiveViews, like the examples in the LiveView docs do, a bit problematic because I often need fields that aren’t defined in the Ecto.Schema. Embedded schemas are great, but it’s a lot of boilerplate…

Plus, often my changesets have required fields like foreign keys that I cannot validate in the LiveView so the changeset is never valid; a state that I use in my forms to enable/disable form submit buttons.

Goal solves all that with a sweet syntax.

Using Goal for LiveViews is super easy:

defmodule MyApp.SomeLiveView do
  import Goal.Syntax

  def handle_event("validate", %{"some" => some_params}, socket) do
    changeset =
      some_params
      |> Goal.build_changeset(schema())
      |> Map.put(:action, :validate)

    {:noreply, assign(socket, :changeset, changeset)}
  end

  defp schema do
    defschema do
      required :id, :uuid
      required :name, :string, max: 20
      optional :age, :integer, min: 0, max: 120
      optional :gender, :enum, values: ["female", "male", "non-binary"]
      optional :address, :string

      optional :data, :map do
        required :color, :string
        optional :money, :decimal
        optional :height, :float
      end
    end
  end
end

[martinthenth]

Goal 0.2.0 released today!

This version contains some breaking, but very convenient changes to the library.

1. It adds the defparams macro which makes defining parameter validation schemas easier than before, because it encapsulates the schemas in a function that you can call elsewhere.
2. It adds conveniences like validate/1 and /2, and changeset/1 and /2 that can be like MySchema.validate(:new, params) and MySchema.changeset(:new, params)

Let’s take a look at the new defparams macro:

Example with a schema in a separate module

Define a schema in a separate module:

defmodule MyAppWeb.MySchema do
  use Goal

  defparams :create do
    required :id, :uuid
    optional :name, :string
    optional :age, :integer, min: 0
  end
end

And then use it in a controller or LiveView:

defmodule MyApp.SomeController do
  use MyApp, :controller
  
  alias MyAppWeb.MySchema

  def create(conn, params) do
    with {:ok, attrs} <- MySchema.validate(:create, params)) do
      ...
    else
      {:error, changeset} -> {:error, changeset}
    end
  end
end

You can have the schema inside the controller, it will work the same.

Example with LiveView

Version 0.2.0 adds conveniences for working with LiveViews, called changeset/2 and validate/2. They are used to build and validate a changeset that can be rendered in a LiveView (also works in HTML-controllers):

defmodule MyApp.SomeLiveView do
  use MyApp, :live_view

  alias MyAppWeb.MySchema

  def mount(params, _session, socket) do
    changeset = MySchema.changeset(:new, %{})
    socket = assign(socket, :changeset, changeset)

    {:ok, socket}
  end

  def handle_event("validate", %{"some" => params}, socket) do
    changeset = MySchema.changeset(:new, params)
    socket = assign(socket, :changeset, changeset)

    {:noreply, socket}
  end

  def handle_event("save", %{"some" => params}, socket) do
    with {:ok, attrs} <- MySchema.validate(:new, params)) do
      ...
    else
      {:error, changeset} -> {:noreply, assign(socket, :changeset, changeset)}
    end
  end
end

With these additions, it looks like Goal is running out of major functional improvements. The library’s behavior is very similar to Ruby’s DRY params and other mature parameter validation libraries. So I think Goal may reach v1.0.0 soon

GitHub - martinthenth/goal: A parameter validation library for LiveView and...

[martinthenth]

Goal 0.2.1 released!

This update adds a neat feature called recase_keys/3, which recases parameter keys from e.g. camelCase to snake_case (i.e. what was defined in the defparams schema).

It’s an optional feature that can be enabled by passing the :recase_keys option to validate/3 or validate_params/3, or by setting :recase_keys in the application config:

config :goal,
  recase_keys: [from: :camel_case]

The supported cases are camelCase, PascalCase, kebab-case and snake_case. The common use-case is for JSON APIs where frontend and backend applications may have different parameter key formats.

Example:

MySchema.validate(:show, %{"firstName" => "Jane"})
{:ok, %{first_name: "Jane"}}

I added Recase as a dependency to handle the string recasing

[martinthenth]

I decided to add a function for outbound key recasing, because I need it in my apps and it seems intuitive that if you recase incoming parameters you would like to recase outbound parameters as well.

An example (with Phoenix 1.7 views):

config :goal,
  recase_keys: [to: :camel_case]

defmodule MyAppWeb.UserJSON do
  import Goal

  def show(%{user: user}) do
    %{data: %{first_name: user.first_name}}
    |> recase_keys()
  end

  def error(%{changeset: changeset}) do
    errors =
      changeset
      |> Goal.Changeset.traverse_errors(&translate_error/1)
      |> recase_keys()

    %{errors: errors}
  end
end

iex(1)> UserJSON.show(%{user: %{first_name: "Jane"}})
%{data: %{firstName: "Jane"}}
iex(2)> UserJSON.error(%Ecto.Changeset{errors: [first_name: {"can't be blank", [validation: :required]}]})
%{errors: %{firstName: ["can't be blank"]}}

The recase_keys/1 function should only be used when you are in control of the data that is being recased.

The updated docs are available on Hex: Goal 0.2.3