💦 Drops - Simplifying Data Validation in Elixir

Hello!

Today I released the first version of Elixir Drops :slight_smile: The announcement can be found on my blog but I’m also just pasting it here:

A few years ago my Ruby friends asked me if it would be possible to port some of the dry-rb libraries to Elixir. I remember some early community attempts at porting dry-validation specifically, I did my best to support the effort but back then my Elixir knowledge was too basic and I was too busy with other work.

Fast forward to today and I’m happy to announce the very first release of Elixir Drops! :tada: In this article I’ll introduce you to the concepts of the library and show you how you could use it.

Contracts with schemas

One of the core features of Drops is its Contract extension with the Schema DSL. It allows you to define the exact shape of the data that your system relies on and data domain validation rules that are applied to type-safe data that was processed by the schema.

There are multiple benefits of this approach:

  • Casting and validating data using schemas is very precise, producing detailed error messages when things go wrong

  • Schemas give you type safety at the boundaries of your system - you can apply a schema to external input and be 100% sure it’s safe to work with

  • It’s very easy to see the shape of data, making it easier to reason about your system as a whole

  • You can restrict and limit larger data structures, reducing them to simpler representations that your system needs

  • It’s easy to convert schemas to other representations ie for documentation purposes or export them to JSON Schema format

  • Schemas capture both the structure and the types of data, making it easy to reuse them across your codebase

  • Your domain validation rules become simple and focused on their essential logic as they apply to data that meets type requirements enforced by schemas

This of course sounds very abstract, so here’s a simple example of a data contract that defines a schema for a new user:

defmodule Users.Signup do
  use Drops.Contract

  schema do
    %{
      required(:name) => string(:filled?),
      optional(:age) => integer()
    }
  end
end

Users.Signup.conform(%{
  name: "Jane Doe",
  age: 42
})
# {:ok,
#  %{
#    name: "Jane Doe",
#    age: 42
#  }}

{:error, errors} = Users.Signup.conform(%{})

Enum.map(errors, &to_string/1)
# ["name key must be present"]

{:error, errors} = Users.Signup.conform(%{
  name: "",
  age: "42"
})

Enum.map(errors, &to_string/1)
# ["age must be an integer", "name must be filled"]

Let’s take a closer look at what we did here:

  • The contract defines a schema with two keys:

    • required(:name) means that the input map is expected to have the key :name

    • string(:filled?) means that the :name must be a non-empty string

    • optional(:age) means that the input could have the key :age

    • integer() means that the :age must be an integer

Even though this is a very basic example, notice that the library does quite a lot for you - it processes the input map into a new map that includes only the keys that you specified in the schema, it checks both the keys and the values according to your specifications. When things go wrong - it gives you nice error messages making it easy to see what went wrong.

Type-safe Casting

One of the unique features of Drops Schemas is type-safe casting. Schemas breaks down the process of casting and validating data into 3 steps:

  1. Validate the original input values

  2. Apply optional casting functions

  3. Validate the output values

It’s better to explain this in code though:

defmodule Users.Signup do
  use Drops.Contract

  schema do
    %{
      required(:name) => string(:filled?),
      optional(:age) => cast(string(match?: ~r/\d+/)) |> integer()
    }
  end
end

Users.Signup.conform(%{
  name: "Jane Doe",
  age: "42"
})

{:error, errors} = Users.Signup.conform(%{
  name: "Jane Doe",
  age: ["oops"]
})

Enum.map(errors, &to_string/1)
# ["cast error: age must be a string"]

{:error, errors} = Users.Signup.conform(%{
  name: "Jane Doe",
  age: "oops"
})

Enum.map(errors, &to_string/1)
# ["cast error: age must have a valid format"]

Notice that when the age input value cannot be casted according to our specification, we get a nice “cast error” message and we immediately know what went wrong.

Domain validation rules

Contracts allow you to split data validation into schema validation and arbitrary domain validation rules that you can implement. Thanks to this approach we can focus on the essential logic in your rules as we don’t have to worry about the types of values that the rules depend on.

Let me explain this using a simple example:

defmodule Users.Signup do
  use Drops.Contract

  schema do
    %{
      required(:name) => string(:filled?),
      required(:password) => string(:filled?),
      required(:password_confirmation) => string(:filled?)
    }
  end

  rule(:password, %{password: password, password_confirmation: password_confirmation}) do
    if password != password_confirmation do
      {:error, {[:password_confirmation], "must match password"}}
    else
      :ok
    end
  end
end

Users.Signup.conform(%{
  name: "John",
  password: "secret",
  password_confirmation: "secret"
})
# {:ok, %{name: "John", password: "secret", password_confirmation: "secret"}}

{:error, errors} = Users.Signup.conform(%{
  name: "John",
  password: "",
  password_confirmation: "secret"
})

Enum.map(errors, &to_string/1)
# ["password must be filled"]

{:error, errors} = Users.Signup.conform(%{
  name: "John",
  password: "foo",
  password_confirmation: "bar"
})

Enum.map(errors, &to_string/1)
# ["password_confirmation must match password"]

Here we check whether password and password_confirmation match but first we define in our schema that they must be both non-empty strings. Notice that our domain validation rule is not applied at all if the schema validation does not pass.

As you can probably imagine, this type of logic could be easily implemented as a higher-level macro, something like validate_confirmation_of :password. This is something that I’ll most likely add to Drops in the near future.

Safe map atomization

Another very useful feature is support for atomizing input maps based on your schema definition. This means that the output map will include only the keys that you defined turned into atoms, in the case of string-based maps.

Here’s an example:

defmodule Users.Signup do
  use Drops.Contract

  schema(atomize: true) do
    %{
      required(:name) => string(:filled?),
      optional(:age) => integer()
    }
  end
end

Users.Signup.conform(%{
  "name" => "Jane Doe",
  "age" => 42
})
# {:ok, %{name: "Jane Doe", age: 42}}

About the first release

Today I’m releasing v0.1.0 of Drops and even though it’s the first release, it already comes with a lot of features! It’s already used in our backend system at valued.app, processing and validating millions of JSON payloads regularly.

It is an early stage of development though and I encourage you to test it out and provide feedback! Here are some useful links where you can learn more:

Check it out and let me know what you think!

35 Likes

Very interested in this. Coming from Clojure, I was looking for something along the lines of clojure.spec, that happened to save my bacon a number of times in production exactly because it catches weird and unexpected input from external systems, documents what went wrong, and then goes boom.

1 Like

I wrote a similar library that may be interesting to you, GitHub - Adzz/data_schema: Declarative schemas for data transformations. it’s attempting the same idea - casting of data at the boundary so it errors nice and early on unexpected input.

@solnic thanks for sharing I’m always interested in how others approach this problem.

3 Likes

Greetings, I was a huge fan of your work and all the dry-rb stuff before I left ruby behind. I’ve used (abused) Ecto schemas for doing much of the same things this lib can do, but that gets kinda clunky when you starting nesting things :sweat_smile:

I will take this for a spin ASAP :slight_smile:

4 Likes

Me too! :slight_smile:

I’ve been wanting this for a while. Unfortunately I never came across @Adzz’s lib.

It’s the legend himself! I Love dry-rb, I’ve introduced parts of it (especially dry-schema) to pretty much all the apps at work and I’ve long wished for something equally nice in Elixir!

2 Likes

Well thank you! That was so nice to read :two_hearts:

1 Like

Hi kwando!! Been a while :slight_smile: And yes, nested data structures is a classic example when solutions like Drops schemas turn out to work really well vs ORM-like approaches.

Fantastic! Let me know if it works or doesn’t and why :slight_smile: Report any issues you find! It’s my current OSS focus as we use it extensively at work :muscle:

I added some initial thoughts on the GitHub discussion :slight_smile:

I fell into another trap yesterday that I didn’t find a way around, basically I wanted a parse a JSON number to a float(). The “float” 1.0 will be encoded as 1 in the JSON payload but that is not acceptable by float() since it is an integer. Tried with custom casters and what not but didn’t get something that worked.

I ended up encoding the number as a string in the JSON and then cast it back to float like this, which is not that great.

%{
  required(:threshold) => cast(:string) |> float()
}

On rules, maybe it would be nice to support guards as well:

rule(:my_rule, %{threshold: threshold}) when threshold > 100 do
  {:error, "threshold overload"}
end

(yeah, know float(lteq?: 100) would be better in this case, but just to illustrate)

Maybe specify in the docs that you can return :ok from a rule, it wasn’t immediately obvious for me :slight_smile: had to look at the source code.

rule(:my_rule, data) do
  if data.threshold > 100 do
    {:error, "threshold overload"}
  else
    :ok
  end
end
3 Likes

I don’t have much experience with dry-rb, but a long time ago, when working with nested objects validations (and validations of arrays of objects) in Ruby, I wished I could use it in a project. It seemed to handle such nested validations much better than what was included in Rails by default.

I see that in Drops, errors are not returned in a nested way, but instead you went with a flat list of errors, and path specified on each one of them. Can you say a bit more about this decision?

I see the project provides a macro-based DSL. Would it be feasible to use it through a more verbose “functional API”, to make it easier for people unfamiliar with Drops to read and debug the code? Or would it be too verbose and unstable / hard to maintain?

And in relation to Phoenix, are there plans to provide conveniences for integrating with Phoenix Forms? Or would this belong to a separate package?

If possible, please report what you tried to do as an issue. I added support for custom casters so that lack of a built-in casting logic would not block anybody. So I’m wondering now why it didn’t work for you.

A list of errors can be transformed into a nested map with errors. This is how it works in dry-validation. Eventually, different error representations will be a built-in feature.

Yes the DSL could be translated into a more verbose syntax where you’d do essentially the same thing that you do when using Ecto changesets. I’m not sure if I’d be interested in maintaining such functionality but it may turn out that this would Just Work OOTB. We’ll see how it goes :slight_smile:

Phoenix and Ecto integrations will be done as part of the package, at least in the beginning. If they grow a lot, I may extract them into ecto-drops and phoenix-drops.

5 Likes

Looks amazing, i love the definition in the module

1 Like

Great job!

So many validation libraries out there. It would be great if you wrote a comparison table, where you show how this library is different from the others, feature by feature

5 Likes

Agreed. And especially a more detailed breakdown of why it might be worth trying something like this if you’re already using Ecto in particular, since that’s going to be the toughest sell given that so many apps will be using Ecto already.

2 Likes

Thanks folks :purple_heart: Eventually, I’ll just document and compare Drops validation with other solutions, especially Ecto since it’s the most popular one.

2 Likes

thanks for your efforts and for sharing. I will be reviewing this code to see how I can use it in my projects

1 Like