TypeCheck - Fast and flexible runtime type-checking for your Elixir projects

Version 0.3.2 has been released.
This adds support for unquote fragments inside types and specs, so now you can use TypeCheck in metaprogramming. David’s earlier example will now compile.

Another example:

defmodule MetaExample do
  use TypeCheck
  people = ~w[joe robert mike]a
  for name <- people do
    @type! unquote(name)() :: %{name: unquote(name), coolness_level: :high}
  end
end
iex> MetaExample.joe
#TypeCheck.Type< %{coolness_level: :high, name: :joe} >

iex> MetaExample.mike
#TypeCheck.Type< %{coolness_level: :high, name: :mike} >

5 Likes

It was long overdue:
Version 0.4.0 has been released! :cake:

This adds two main features:

  • Protocol-based types
  • Type overrides.

Protocol-based types

This adds supports for impl(ProtocolName). This is a way to use "any type that implements protocol ProtocolName" in your types and specs.

TypeCheck supports this both in type-checks (checking whether a protocol is implemented for the given term), as well as for property-testing generation (generating "any value of any type that implements ProtocolName"):

An example of using protocols in specs:

defmodule OverrideExample do
  use TypeCheck

  @spec! average(impl(Enumerable)) :: {:ok, float()} | {:error, :empty}
  def average(enumerable) do
    if Enum.empty?(enumerable) do
      {:error, :empty}
    else
      res = Enum.sum(enumerable) / Enum.count(enumerable)
      {:ok, res}
    end
  end
end
OverrideExample.average([10, 20])
{:ok, 15.0}
iex(18)> OverrideExample.average(MapSet.new([1,2,3,4]))
{:ok, 2.5}
OverrideExample.average([])              
{:error, :empty}

OverrideExample.average(10)                   
** (TypeCheck.TypeError) At iex:11:
The call to `average/1` failed,
because parameter no. 1 does not adhere to the spec `impl(Enumerable)`.
Rather, its value is: `10`.
Details:
  The call `average(10)`
  does not adhere to spec `average(impl(Enumerable)) :: {:ok, float()} | {:error, :empty}`. Reason:
    parameter no. 1:
      `10` does not implement the protocol `Elixir.Enumerable`
    lib/type_check/spec.ex:156: OverrideExample.average/1

And an example of some generating some data:

iex> require TypeCheck.Type
iex> import TypeCheck.Builtin
iex> TypeCheck.Type.build(impl(Enumerable)) |> TypeCheck.Protocols.ToStreamData.to_gen |> Enum.take(5) 
[
  %{{false, ""} => -1.0},
  #MapSet<[]>,
  -2..2,
  0..3,
  %{:EG => {}, {false} => 1.0, [] => %{-1 => ""}}
]

Type Overrides

From time to time we need to interface with modules written in other libraries (or the Elixir standard library) which do not expose their types through TypeCheck yet.
We want to be able to use those types in our checks, but they exist in modules that we cannot change ourselves.

The solution is to allow a list of ‘type overrides’ to be given as part of the options passed to use TypeCheck, which allow you to use the original type in your types and documentation, but have it be checked (and potentially property-generated) as the given TypeCheck-type.

An example:

      defmodule Original do
        @type t() :: any()
      end

      defmodule Replacement do
        use TypeCheck
        @type! t() :: integer()
      end

      defmodule Example do
        use TypeCheck, overrides: [{&Original.t/0, &Replacement.t/0}]

        @spec! times_two(Original.t()) :: integer()
        def times_two(input) do
          input * 2
        end
      end

Or indeed:

defmodule TypeOverrides do
  use TypeCheck
  import TypeCheck.Builtin
  @opaque! custom_enum() :: impl(Enumerable)
end

defmodule Example do
  use TypeCheck, overrides: [{&Enum.t/0, &TypeOverrides.custom_enum/0}]

  @spec! average(Enum.t()) :: {:ok, float()} | {:error, :empty}
  def average(enumerable) do
    # ... (see first example of post)
  end
end

As this feature is still very new, there are bound to still be some bugs or edge cases in there.
Also, it would be nice to have support by default already provided for all remote types of Elixir’s standard library.
This is something which will be added in the very near future; probably in the next release.


What is next?

A detailed long-term roadmap is available in the Readme.
In the short-term, focus is on the following:

  • Improve code-coverage of the testing suite
  • Move the CI from Travis to GitHub’s workflows, and test against newer (and possibly some older) Elixir versions
  • Add a set of ‘default overrides’ for the common remote types that are part of the Elixir standard library (such as Enum.t(), Range.t(), DateTime.t() etc.).
  • Be able to limit the depth of the generated checks, to further increase performance for production environments.
11 Likes

Version 0.5.0 has been released! :blush:

This version adds a number of stability and ‘quality of life’ improvements.

Additions & Improvements

  • Adding the option debug: true, wich can be passed to use TypeCheck or TypeCheck.conform/3 (and variants), which will (at compile-time) print the checks that TypeCheck is generating. Example.
  • Allow disabling the generation of a typespec, by writing @autogen_typespec false. This will ensure that no typespec is exported for the next @type!/@opaque/@spec! encountered in a module. Example.
  • Actually by default autogenerate a @spec typespec for all @spec!'s.
  • Code coverage of the test-suite increased to > 85%.

Fixes

  • Bugfixes w.r.t. generating typespecs that Elixir/Dialyzer is happy with.
  • Fixes compiler-warnings on unused named types when using a type guard.
  • Fixes any warnings that were triggered during the test suite before.

Meta

  • Moving from Travis CI to GitHub Workflows
  • Setting up Coveralls for code coverage.
12 Likes

Wow, this is a nice release! Really great to have escape holes with the new @autogen_typespec false option and the use TypeCheck, debug: true debug output is really interesting for everyone wanting to understand the inner workings of TypeCheck. Thank you, I feel now the library is ready to be used for production code.

1 Like

Version 0.6.0 has been released! :rocket:

Its main changes are the addition of spectests and the implementation of a very large portion of the types in all modules of Elixir’s standard library.

Additions & Improvements:

  • Adding TypeCheck.ExUnit, with the function spectest to test function-specifications.
    • Possibility to use options :except, :only, :initial_seed.
    • Possibility to pass custom options to StreamData.
  • Adding TypeCheck.DefaultOverrides with many sub-modules containing checked typespecs for the types in Elixir’s standard library (75% done).
    • Ensure that these types are correct also on older Elixir versions (1.9, 1.10, 1.11)
  • By default load these ‘DefaultOverrides’, but have the option to turn this behaviour off in TypeCheck.Option.
  • Nice generators for Enum.t, Collectable.t, String.t.
  • Support for the builtin types:
    • pid()
    • nonempty_list(), nonempty_list(type).
  • Allow use TypeCheck in IEx or other non-module contexts, to require TypeCheck and import TypeCheck.Builtin in the current scope (without importing/using the macros that only work at the module level.)
  • The introspection function __type_check__/1 is now added to any module that contains a use TypeCheck.

Fixes

  • Fixes the Inspect implementation of custom structs, by falling back to Any, which is more useful than attempting to use a customized implementation that would try to read the values in the struct and failing because the struct-type containing types in the fields.
  • Fixes conditional compilation warnings when optional dependency :stream_data was not included in your project.

What is a spectest?

A ‘function-specification test’ is a property-based test in which
we check whether the function adheres to its invariants
(also known as the function’s contract or preconditions and postconditions).

We generate a large amount of possible function inputs,
and for each of these, check whether the function:

  • Does not raise an exception.
  • Returns a result that type-checks against the function-spec’s return-type.

While @spec!s themselves ensure that callers do not mis-use your function,
a spectest ensures¹ that the function itself is working correctly.

Spectests are given its own test-category in ExUnit, for easier recognition
(Just like ‘doctests’ and ‘properties’ are different from normal tests, so are ‘spectests’.)

¹: Because of the nature of property-based testing, we can never know for 100% sure
that a function is correct. However, with every new randomly-generated
test-case, the level of confidence grows a little. So while we
can never by fully sure, we are able to get asymptotically close to it.

Example:

Given the module

defmodule SpectestExample do
  use TypeCheck

  @spec! average(list(number())) :: number()
  def average(vals)  do
    Enum.sum(vals) / Enum.count(vals)
  end
end

And the test file

defmodule SpectestTest do
  use ExUnit.Case
  import TypeCheck.ExUnit

  spectest SpectestExample
end

We receive the following output when running the tests:

mix test test/spectest_test.exs 
Compiling 1 file (.ex)


  1) spectest average(list(number())) :: number() (SpectestTest)
     test/spectest_test.exs:5
     Spectest failed (after 0 successful runs)
     
     Input: SpectestExample.average([])
     
     ** (ArithmeticError) bad argument in arithmetic expression
     
     code: #TypeCheck.Spec<  average(list(number())) :: number() >
     stacktrace:
       (type_check 0.5.0) lib/debug_example.ex:6: SpectestExample."average (overridable 1)"/1
       lib/type_check/ex_unit.ex:5: anonymous fn/1 in SpectestTest."spectest average(list(number())) :: number()"/1
       (stream_data 0.5.0) lib/stream_data.ex:2102: StreamData.check_all/7
       lib/type_check/ex_unit.ex:5: (test)



Finished in 0.08 seconds (0.00s async, 0.08s sync)
1 spectest, 1 failure

Randomized with seed 792447

So in this example, we forgot to handle empty lists correctly.
We might decide to either require the user to pass a nonempty_list() (in which case a TypeError will be raised for empty lists), or instead decide to alter the internals and the return type of the function (like returning {:ok, number} | {:error, :empty}).

in either case, this will then make the spectest pass :blush: .

Default Overrides

The large amount of ‘default overrides’ now means that you can just use Range.t, String.t, Enum.t, MapSet.t etc. in your types and specs to your hearts content.
Not all types of Elixir’s standard library are supported yet, but most of them are.


I am very excited and eager to hear what you think of the new features!

~Marten

11 Likes

@baldwindavid pointed out that it had gotten a bit out of date, so I spent some time to rewrite the Comparing TypeCheck and Norm page.

If you’re curious about how TypeCheck’s approach to data validation and generation compares to @keathley’s Norm, do check it out!

7 Likes

This is really neat! Typechecking is one of the missing pieces in the Elixir ecosystem, excited to see this added!

2 Likes

That page is an instant bookmark, thanks for writing it!

1 Like

Version 0.7.0 has been released! :partying_face:

Additions & Improvements

  • Addition of the option enable_runtime_checks. When false, all runtime checks in the given module are completely disabled. This is useful to for instance disable checks in a particular environment. (c.f. #52) Thank you, @baldwindavid!
  • Adding DateTime.t to the default overrides of Elixir’s standard library, as it was still missing.

Besides this new version being released, I have spent some time to write an in-depth introductionary article:

Type-checking and spec-testing with TypeCheck

(Forum topic about the article)

If you were still unsure whether TypeCheck was for you, or how to use it, I urge you to give it a read! :blush:

7 Likes

Version 0.8.0 has been released! :ship:

Additions & Improvements

  • Pretty-printing of types and TypeError output in multiple colors:

(This is the ‘Rating’ example from the Type-checking and spec-testing with TypeCheck article).
Types and values are pretty-printed in colour, similarly to how this is normally done in IEx.

  • Nicer indentation of errors. Amongst other things, this means that error-highlighting in the documentation now works correctly (although in 100% ‘red’).
  • use TypeCheck now also calls require TypeCheck.Type so there no longer is a need to call this manually if you want to e.g. use TypeCheck.Type.build/1 (which is rather common if you want to test out particular types quickly in IEx).
  • named types are now printed in abbreviated fashion if they are repeated multiple times in an error message. This makes a nested error message much easier to read, especially for larger specs.
  • Remote/user-defined types are now also ‘named types’ which profit from this change.
    For instance in above example picture, we first talk about Rating.t() and String.t() and only when looking at the problem in detail do we expand this to %Rating{} and binary().
  • [type] no longer creates a fixed_list(type) but instead a list(type) (just as Elixir’s own typespecs.)
  • Support for [...] and [type, ...]as alias for nonempty_list() and nonempty_list(type) respectively.

Fixes

  • Fixes prettyprinting of TypeCheck.Builtin.Range.
  • Remove support for list literals with multiple elements.
  • Improved documentation.
6 Likes

Are there any plans on how to do this without requiring a use in every module? I had been thinking through this problem and got stuck. Tracer and use are the best options I could consider.

There is not. This kind of functionality can only be provided using macros. This means one of three possibilities:

  1. use, which TypeCheck uses now and is arguably the most straightforward/well-known.
  2. Redefining defmodule. However, to be able to use the overridden defmodule, someone requires a require YourModule; import Kernel, except: [defmodule: 2] somewhere earlier in the file as well. The only way to hide this is again by using use.
  3. A separate compilation pass/tracer. While this might be possible, it would mean a whole lot of extra work to get it working reliably.

So for now, use TypeCheck it is.
The added benefit of having this line visibly there in your modules, is of course that it is less magical/more explicit. My hope is that people will be able to grasp that there might be a connection between @type! and @spec! with TypeCheck when they are reading a module’s source code, even if they’ve never heard of it before.

3 Likes

@sb8244 Probably doesn’t solve your problem, but my use of this in the wild is to bring it into modules that are already shared with the various parts of my app. For example, contexts will already use a module like this:

defmodule MyApp.Context do
  defmacro __using__(_) do
    quote do
      use TypeCheck
      # ... other context concepts that should always be present
    end
  end
end
5 Likes

The technique @baldwindavid describes is a nice one. You might already have used it (consciously or subconsciously) elsewhere, such as your app’s Ecto Repo module, or your app’s Phoenix Endpoint module.

TypeCheck requires its options to be specified at compile-time, and some of the options might be different per module. As such, passing the options as parameters to use is a better approach than using e.g. Application.compile_env.
So in the case you need some more complex options which might be shared across multiple modules of your application, I would suggest doing something like:

defmodule MyApp.TypeCheck do
  defmacro __using__(opts) do
    quote do
      use TypeCheck, (unquote(opts) ++ [
      # Some example options: 
      overrides: MyApp.TypeCheck.app_specific_overrides(), 
      enable_runtime_checks: if Mix.env() != :prod, 
       # etc.
      ])
    end
  end

  def app_specific_overrides() do
    [
      {{Foo, :bar, 1}, {Alternative, :baz, 1}},
      # etc.
    ]
  end
end

Then in your other modules, a plain use MyApp.TypeCheck is enough.

3 Likes

Version 0.9.0 has been released! :blush:

Additions & Improvements

Support for bitstring literal types.

The types bitstring(), binary() and String.t() were already supported.
However, support has now been added for the types:

  • <<>> (matching an empty bitstring),
  • <<_ :: size>> (matching a bitstring of exactly size bits long),
  • <<_ :: _ * unit>> (matching any bitstring of x * unit where x :: pos_integer()),
  • <<_ :: size, _ :: _ * unit>> (matching any bitstring of size + x * unit where x :: pos_integer()).

Property-testing generators have also been constructed for them, so you can immediatly start using them with spectests.

7 Likes

Version 0.10.0 has been released! Ⲗ

Additions & Improvements

Support for function-types (for typechecks as well as property-testing generators):

  • (-> result_type)
  • (...-> result_type)
  • (param_type, param2_type -> result_type)

Type-checking function-types

Type-checking a function value against a function-type works a bit differently from most other types.
The reason for this is that we can only ascertain whether the function-value works correctly when the function-value is called.

Specifically:

  • When a call to TypeCheck.conforms/3 (and variants) or a function wrapped with a @spec! is called, we can immediately check whether a particular parameter:
    • is a function
    • accepts the expected arity
  • Then, the parameter-which-is-a-function is wrapped in a ‘wrapper function’ which, when called:
    • typechecks whether the passed parameters are of the expected types (This checks whether your function uses the parameter-function correctly.)
    • calls the original function with the parameters.
    • typechecks whether the result is of the expected type. (This checks whether the parameter-function works correctly.)
    • returns the result.

In other words, the ‘wrapper function’ which is added for a type (param_type, param_type2 -> result_type) works similarly
to a named function with the spec @spec! myfunction(param_type, param_type2) :: result_type.

As an example:

      iex> # The following passes the first check...
      iex> fun = TypeCheck.conforms!(&div/2, (integer(), integer() -> boolean()))
      iex> # ... but once the function returns, the wrapper will raise
      iex> fun.(20, 5)
      ** (TypeCheck.TypeError) The call to `#Function<...>/2` failed,
          because the returned result does not adhere to the spec `boolean()`.
          Rather, its value is: `4`.
          Details:
            The result of calling `#Function<...>.(20, 5)`
            does not adhere to spec `(integer(), integer() -> boolean())`. Reason:
              Returned result:
                `4` is not a boolean.

This was quite the adventure to implement. I am happy that it turned out to be possible, and it is working great!

Data generation for function types

For property-testing generators, the data passed to a generated function is converted into a seed (using [param1, param2, param3] |> :erlang.term_to_binary |> Murmur.hash_x86_32) and this seed is then used as seed for the data returned from the function.
This means that for any particular test run, any generated function will be pure (i.e. when given the same input multiple times, the same output will be returned).

Fixes

  • Wrapping private functions no longer make the function public. (c.f. #64)
  • Wrapping macros now works correctly. (also related to #64)
  • Using __MODULE__ inside a struct inside a type now expands correctly. (c.f. #66)
4 Likes

Version 0.10.1 has been released!

Fixes

  • Swap Murmur out for :erlang.phash2 in the generation of function-values.

In 0.10.0, the capability to generate arbitrary values from function-types was introduced (for use in e.g. spectesting and property-based testing).
However, this required adding :murmur as extra (optional) dependency.
Since it turns out that the capability of generating a (non-cryptographic) integer hash value from an arbitrary term is built in in Erlang, TypeCheck now no longer needs the dependency.

3 Likes

For who wants some more in-depth knowledge on how the library works and could be used, be sure to check out Episode 72 of the Thinking Elixir podcast, in which Mark Ericksen, David Bernheisel and Cade Ward interviewed me about the library :blush: !

14 Likes

I listened to it. It was really good interview Martin.

3 Likes

When I tried adding this to an existing Phoenix project that I had been using Dialyzer on before, I got a lot of undefined or private errors on typespecs that had been working fine with Dialyzer. I got them for things like Plug.Conn.t and Phoenix.LiveView.Socket.t. Is TypeCheck not able to detect types that aren’t explicitly defined with its @spec! syntax?