ProtocolEx - extended Protocol library using Matchers

Hex: https://hex.pm/packages/protocol_ex
Docs: https://hexdocs.pm/protocol_ex/readme.html
Source: https://github.com/OvermindDL1/protocol_ex

Made yet another library, this time it is an enhanced protocol style thing that is not limited to base types and structs as the built-in one is, and I’ve not got around to making a compiler pass like the built-in one has yet so right now you just need to consolidate it manually anywhere (which is just a single call), so first, simple example usage in iex:

iex> import ProtocolEx
ProtocolEx
iex>   defprotocolEx Blah do
...>     def empty(a)
...>     def succ(a)
...>     def add(a, b)
...>
...>     def a_fallback(a), do: inspect(a)
...>   end
{:module, :"Elixir.Blah.$ProtocolEx_description$",
 <<70, 79, 82, 49, 0, 0, 6, 72, 66, 69, 65, 77, 69, 120, 68, 99, 0, 0, 0, 127,
   131, 104, 2, 100, 0, 14, 101, 108, 105, 120, 105, 114, 95, 100, 111, 99, 115,
   95, 118, 49, 108, 0, 0, 0, 4, 104, 2, ...>>, {:spec, 0}}
iex>   defimplEx Integer, i when is_integer(i), for: Blah do
...>     def empty(_), do: 0
...>     def succ(i), do: i+1
...>     def add(i, b), do: i+b
...>
...>     def a_fallback(i), do: "Integer: #{i}"
...>   end
:ok
iex> # An error when you do not fullfill the protocol properly:
nil
iex>   defimplEx TaggedTuple.Vwoop, {Vwoop, i} when is_integer(i), for: Blah do
...>     def empty(_), do: {Vwoop, 0}
...>   end
** (ProtocolEx.MissingRequiredProtocolDefinition) On Protocol `Elixir.Blah` missing a required protocol callback on `TaggedTuple.Vwoop` of:  add/2
    (protocol_ex) lib/protocol_ex.ex:194: anonymous fn/3 in ProtocolEx.verify_valid_spec_on_module/3
         (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
         (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
iex>
nil
iex>   defimplEx TaggedTuple.Vwoop, {Vwoop, i} when is_integer(i), for: Blah do
...>     def empty(_), do: {Vwoop, 0}
...>     def succ({Vwoop, i}), do: {Vwoop, i+1}
...>     def add({Vwoop, i}, b), do: {Vwoop, i+b}
...>   end
:ok
iex> defmodule MyStruct do
...>   defstruct a: 42
...> end
{:module, MyStruct,
 <<70, 79, 82, 49, 0, 0, 9, 12, 66, 69, 65, 77, 69, 120, 68, 99, 0, 0, 0, 186,
   131, 104, 2, 100, 0, 14, 101, 108, 105, 120, 105, 114, 95, 100, 111, 99, 115,
   95, 118, 49, 108, 0, 0, 0, 4, 104, 2, ...>>, %MyStruct{a: 42}}
iex>   defimplEx MineOlStruct, %MyStruct{}, for: Blah do
...>     def empty(_), do: %MyStruct{a: 0}
...>     def succ(s), do: %{s | a: s.a+1}
...>     def add(s, b), do: %{s | a: s.a+b}
...>   end
:ok
iex> # Now let's consolidate it manually (you can even call this at runtime to dynamically add/remove protocol implementations too
nil
iex>   ProtocolEx.resolveProtocolEx(Blah, [
...>     Integer,
...>     TaggedTuple.Vwoop,
...>     MineOlStruct,
...>   ])
:ok
iex> # and let's call it with some tests:
nil
iex> 0                 = Blah.empty(42)
0
iex> {Vwoop, 0}        = Blah.empty({Vwoop, 42})
{Vwoop, 0}
iex> %MyStruct{a: 0}   = Blah.empty(%MyStruct{a: 42})
%MyStruct{a: 0}
iex>
nil
iex> 43                 = Blah.succ(42)
43
iex> {Vwoop, 43}        = Blah.succ({Vwoop, 42})
{Vwoop, 43}
iex> %MyStruct{a: 43}   = Blah.succ(%MyStruct{a: 42})
%MyStruct{a: 43}
iex>
nil
iex> 47                 = Blah.add(42, 5)
47
iex> {Vwoop, 47}        = Blah.add({Vwoop, 42}, 5)
{Vwoop, 47}
iex> %MyStruct{a: 47}   = Blah.add(%MyStruct{a: 42}, 5)
%MyStruct{a: 47}
iex>
nil
iex> "Integer: 42"      = Blah.a_fallback(42)
"Integer: 42"
iex> "{Vwoop, 42}"      = Blah.a_fallback({Vwoop, 42})
"{Vwoop, 42}"
iex> "%MyStruct{a: 42}" = Blah.a_fallback(%MyStruct{a: 42})
"%MyStruct{a: 42}"

Should work properly when aliased out and in sub-modules and all forth as well (as the tests test for).

But basically a defprotocolEx is simple, no guards allowed (might change on non-first arguments actually, I’m tempted…), either empty function heads, which must be implemented, or a full function with body, which becomes the default implementation if not implemented (a fallback, also great to add functions to call the protocol functions too).

In the defimplEx it takes a unique ‘name’ for this implementation (so it can be referred to later), a match specification, a for: ProtocolName to specify the protocol, and the module body. (Implementation detail: For a given implementation named Bloop for a protocol named Blah creates a module of Bloop.Blah that you can call directly).

The resolveProtocolEx takes the protocol to consolidate, and a list of the implementations to make into the final module, they are tested based on their matchspec in the order specified here, so for the above iex session it generates this for the protocol module:

def Blah do
  # Snip some extra stuff like the __protocolEx__ description function
  def(empty(i = a) when is_integer(i)) do
    Testering.Blah.Integer.empty(a)
  end
  def(empty({Vwoop, i} = a) when is_integer(i)) do
    Testering.Blah.TaggedTuple.Vwoop.empty(a)
  end
  def(empty(%MyStruct{} = a)) do
    Testering.Blah.MineOlStruct.empty(a)
  end
  def(empty(value)) do
    raise(%ProtocolEx.UnimplementedProtocolEx{proto: Testering.Blah, name: :empty, arity: 1, value: value})
  end
  def(succ(i = a) when is_integer(i)) do
    Testering.Blah.Integer.succ(a)
  end
  def(succ({Vwoop, i} = a) when is_integer(i)) do
    Testering.Blah.TaggedTuple.Vwoop.succ(a)
  end
  def(succ(%MyStruct{} = a)) do
    Testering.Blah.MineOlStruct.succ(a)
  end
  def(succ(value)) do
    raise(%ProtocolEx.UnimplementedProtocolEx{proto: Testering.Blah, name: :succ, arity: 1, value: value})
  end
  def(add(i = a, b) when is_integer(i)) do
    Testering.Blah.Integer.add(a, b)
  end
  def(add({Vwoop, i} = a, b) when is_integer(i)) do
    Testering.Blah.TaggedTuple.Vwoop.add(a, b)
  end
  def(add(%MyStruct{} = a, b)) do
    Testering.Blah.MineOlStruct.add(a, b)
  end
  def(add(value, _)) do
    raise(%ProtocolEx.UnimplementedProtocolEx{proto: Testering.Blah, name: :add, arity: 2, value: value})
  end
  def(a_fallback(i = a) when is_integer(i)) do
    Testering.Blah.Integer.a_fallback(a)
  end
  def(a_fallback(a)) do
    inspect(a)
  end
end

And I’m thinking about being able to inline from the implementations into the protocol as well, but a lot of things to consider there like macro’s and module attributes and so forth, so I might make it opt-in since it will really only benefit recursively calling such a protocol anyway.

As mentioned it does not have a consolidation compiler built yet like elixir’s, but that is easy enough to do and since all implementations are submodules of the protocol then it will be easy to find them, I might add a @procotolEx_priority module attribute to allow the matching order they should follow for the very few use-cases that should actually matter (I.E. it can be ignored 99% of the time).

As usual, this is not really intended for use yet (although it should be perfectly fine and efficient to use), I’m still debating on the design. I’m also playing with an idea of inverting the setup, so you make the implementations as just normal modules, nothing special there, then define the protocol and resolver all at the same time as it checks if the modules fulfill that. A benefit of that style would be a single module could fulfill multiple protocols, but a side effect is less control overall, although it does mean no consolidator is necessary since it is all a single step, and it also means no reconsolidation at run-time too, so I’m debating, I think I’ll keep the current style since it is more powerful.

I’m also considering adding in property checks (ala the type_class library), that would let you enforce certain contracts that the implementers absolutely must fulfill or it would fail compilation. Is that really in the purview of this library though or should it be in another like type_class, hmm… Thoughts?

Regardless, it could still use more work, testing, and features before I consider it 1.0.0.

11 Likes

Eh, I came up with multiple use-cases for when guards and matchers would be useful on the protocol root definitions (example below), so I added it and released 0.2.0.

  defprotocolEx Blah do
    def empty(a)
    def succ(a)
    def add(a, b)
    def map(a, f) when is_function(f, 1)

    def a_fallback(a), do: inspect(a)
  end

  defimplEx Integer, i when is_integer(i), for: Blah do
    def empty(_), do: 0
    def succ(i), do: i+1
    def add(i, b), do: i+b
    def map(i, f), do: f.(i)

    def a_fallback(i), do: "Integer: #{i}"
  end

  defimplEx TaggedTuple.Vwoop, {Vwoop, i} when is_integer(i), for: Blah do
    def empty(_), do: {Vwoop, 0}
    def succ({Vwoop, i}), do: {Vwoop, i+1}
    def add({Vwoop, i}, b), do: {Vwoop, i+b}
    def map({Vwoop, i}, f), do: {Vwoop, f.(i)}
  end

  defmodule MyStruct do
    defstruct a: 42
  end

  defimplEx MineOlStruct, %MyStruct{}, for: Blah do
    def empty(_), do: %MyStruct{a: 0}
    def succ(s), do: %{s | a: s.a+1}
    def add(s, b), do: %{s | a: s.a+b}
    def map(s, f), do: %{s | a: f.(s.a)}
  end

  43                 = Blah.map(42, &(&1+1))
  {Vwoop, 43}        = Blah.map({Vwoop, 42}, &(&1+1))
  %MyStruct{a: 43}   = Blah.map(%MyStruct{a: 42}, &(&1+1))

Wouldn’t it be more consistent to have macros named defimpl_ex / defimplex and defprotocol_ex / defprotocolex to be more consistent with elixir’s naming style?

Maybe, I’m bad at naming, hence why I’m asking for feedback. ^.^

I had a few minutes available earlier today so I wrote a compiler for it.

So now just depend on it:

    {:protocol_ex, "~> 0.3.1"},

Add it as a compiler after the elixir compiler:


  def project do
    [
      # snip others
      compilers: Mix.compilers ++ [:protocol_ex],
      deps: deps(),
    ]

Make your protocol(s) somewhere:

import ProtocolEx

defprotocolEx Blah do
  def empty(a)
  def succ(a)
  def add(a, b)
  def map(a, f) when is_function(f, 1)

  def a_fallback(a), do: inspect(a)
end

defprotocolEx Bloop do
  def get(thing)
  def get_with_fallback(thing), do: {:fallback, thing}
end

And implement them somewhere for whatever shapes you want:

import ProtocolEx

defimplEx Integer, i when is_integer(i), for: Blah do
  @priority 1
  def empty(_), do: 0
  def succ(i), do: i+1
  def add(i, b), do: i+b
  def map(i, f), do: f.(i)

  def a_fallback(i), do: "Integer: #{i}"
end

defmodule MyStruct do
  defstruct a: 42
end

defimplEx TaggedTuple.Vwoop, {Vwoop, i} when is_integer(i), for: Blah do
  def empty(_), do: {Vwoop, 0}
  def succ({Vwoop, i}), do: {Vwoop, i+1}
  def add({Vwoop, i}, b), do: {Vwoop, i+b}
  def map({Vwoop, i}, f), do: {Vwoop, f.(i)}
end

defimplEx MineOlStruct, %MyStruct{}, for: Blah do
  def empty(_), do: %MyStruct{a: 0}
  def succ(s), do: %{s | a: s.a+1}
  def add(s, b), do: %{s | a: s.a+b}
  def map(s, f), do: %{s | a: f.(s.a)}
end

defimplEx Integer, i when is_integer(i), for: Bloop do
  def get(i), do: {:integer, i}
  def get_with_fallback(i), do: {:integer, i}
end

And set up your tests as normal or so:

defmodule ConsolidateTest do
  use ExUnit.Case

  test "Blah" do
    assert 0                  = Blah.empty(42)
    assert {Vwoop, 0}         = Blah.empty({Vwoop, 42})
    assert %MyStruct{a: 0}    = Blah.empty(%MyStruct{a: 42})

    assert 43                 = Blah.succ(42)
    assert {Vwoop, 43}        = Blah.succ({Vwoop, 42})
    assert %MyStruct{a: 43}   = Blah.succ(%MyStruct{a: 42})

    assert 47                 = Blah.add(42, 5)
    assert {Vwoop, 47}        = Blah.add({Vwoop, 42}, 5)
    assert %MyStruct{a: 47}   = Blah.add(%MyStruct{a: 42}, 5)

    assert "Integer: 42"      = Blah.a_fallback(42)
    assert "{Vwoop, 42}"      = Blah.a_fallback({Vwoop, 42})
    assert "%MyStruct{a: 42}" = Blah.a_fallback(%MyStruct{a: 42})

    assert 43                 = Blah.map(42, &(&1+1))
    assert {Vwoop, 43}        = Blah.map({Vwoop, 42}, &(&1+1))
    assert %MyStruct{a: 43}   = Blah.map(%MyStruct{a: 42}, &(&1+1))

    assert {:integer, 42}     = Bloop.get(42)
    assert {:integer, 42}     = Bloop.get_with_fallback(42)
    assert {:fallback, 6.28}  = Bloop.get_with_fallback(6.28)

    assert_raise ProtocolEx.UnimplementedProtocolEx, fn ->
      Bloop.get(6.28)
    end
  end
end

Pretty simple, about as verbose as the built-in protocols (except you need to import things as they are not default imported in kernel like normal protocols, but eh). :slight_smile:

New version, new features.

This version add the deftest call when defining a protocol. It allows you to run tests at compile time on each implementation that uses this protocol to verify that they follow the proper rules of the protocol, an example:

  defprotocolEx Functor do
    def map(v, f)

    deftest identity do
      StreamData.check_all(prop_generator(), [initial_seed: :os.timestamp()], fn v ->
        if v === map(v, &(&1)) do
          {:ok, v}
        else
          {:error, v}
        end
      end)
    end

    deftest composition do
      f = fn x -> x end
      g = fn x -> x end
      StreamData.check_all(prop_generator(), [initial_seed: :os.timestamp()], fn v ->
        if map(v, fn x -> f.(g.(x)) end) === map(map(v, g), f) do
          {:ok, v}
        else
          {:error, v}
        end
      end)
    end
  end

Just very basic rules (and not even really entirely correct for a Functor, but this is an example), using the StreamData library to do the testing, which shows off another feature that the test is run in the scope of the implementation, thus they have to define a prop_generator/0 function to return a StreamData generator to do the testing (this could easily be just another protocol too actually).

For example, for integers and lists:

  defimplEx Integer, i when is_integer(i), for: Functor do
    def prop_generator(), do: StreamData.integer()
    def map(i, f), do: f.(i)
  end

  defimplEx List, l when is_list(l), for: Functor do
    def prop_generator(), do: StreamData.list_of(StreamData.integer())
    def map([], _f), do: []
    def map([h | t], f), do: [f.(h) | map(t, f)]
  end

Pretty simple, but it does do the testing (tested by introducing a variety of bugs as well) and it runs over the tests when it consolidates the implementations. :slight_smile:

1 Like

New version new features. ^.^

  • Allowing you to define a defimpl_ex callback as a macro so you can easily inline it into the body of the protocol specification (there is still the ‘hidden’ :inline attribute internally if you want to override everything about a function, including the matcher, but it is very low level so it will stay hidden for now, using defmacro will work for 99% of cases now so whoo!).

  • Added ability to name the matcher variable so you can use it in locations other than the front (it defaults to front to match Elixir’s piping semantics, plus to match Elixir’s current protocol semantics). This allows you to give the matching variable a name, like defprotocol_ex Monad, as: monad do ... end then you can do something like def wrap(value, monad) inside and it will match on the monad variable at the end of the arguments there. If you name the variable then you must use the name at every definition to prevent accidental mis-spellings and so forth. If a name is not specified then it just uses the first variable, whatever it is, in every definition, like it does in normal Elixir protocols. :slight_smile:

  • Added ability to specify a 0-arity function that creates a 1-arity function of just the matcher that delegates to the appropriate 0-arity function on the implementation. Just a nice helper that helps to clean up API’s that use it. ^.^

  • Added bouncer methods for elixir’y names to the current names, the current names will be deprecated in time but not yet, feel free to use either but might be best to start using the elixir’y versions (just take uppercase and make it an underscore then the lowercase version of it, like defimplEx -> defimpl_ex and so forth). Requested by @tmbb. ^.^

An example of the new usages:

import ProtocolEx

defprotocol_ex Monad, as: monad do
  def wrap(value, monad)
  def flat_map(monad, fun)
end

defimpl_ex EmptyList, [], for: Monad do
  defmacro wrap(value, monad_type) do
    quote do
      _ = unquote(monad_type)
      [unquote(value)]
    end
  end
  defmacro flat_map(empty_list, fun) do
    quote do
      _ = unquote(empty_list)
      _ = unquote(fun)
      []
    end
  end
end

defimpl_ex List, [_ | _], for: Monad do
  defmacro wrap(value, monad_type) do
    quote do
      _ = unquote(monad_type)
      [unquote(value)]
    end
  end
  def flat_map([], _fun), do: []
  def flat_map([head | tail], fun), do: fun.(head) ++ flat_map(tail, fun)
end

Normally there is not much/any need to defmacro like I am here, but this makes a great test and does eek out a tiny bit more performance regardless. ^.^

1 Like

And 0.3.15 is published, it just adds support for calling defprotocol_ex and defimpl_ex from within other modules, which will also inherit their names, thus:

defmodule EmbTesting do
  defprotocol_ex Emb do
    def add(a)
  end

  defimpl_ex Atom, a when is_atom(a), for: Emb do
    def add(a), do: a
  end
  defimpl_ex Binary, b when is_binary(b), for: EmbTesting.Emb do
    def add(b), do: b
  end
end

## This can't be in the same file as the module definition as the module
## is not compiled by the time this is ready.
#defimpl_ex Integer, i when is_integer(i), for: EmbTesting.Emb do
#  def add(a), do: a+1
#end

defmodule MoreEmbTesting do
  defimpl_ex Float, f when is_float(f), for: EmbTesting.Emb do
    def add(a), do: a+1.0
  end
end

Will define an EmbTesting.Emb protocol, and EmbTesting.Emb.EmbTesting.Atom, EmbTesting.Emb.EmbTesting.Binary, and EmbTesting.Emb.MoreEmbTestingFloat implementations. The implementations inherit their module names for easy namespacing or so, and of course defimpl_ex's at the global scope don’t get auto-namespaced like that anyway, just this example cannot do that because the module is not fully defined by the time the implementation is trying to be called (put in different files if you want that).

Previously trying to define a protocol in a module wouldn’t compile, but implementations would, should probably increment the version 3 to a 4, but it’s still in dev anyway and that is not a feature people seem to have ever used before now… ^.^;

2 Likes

Up to 0.3.19 out so far, a couple minor features added and so forth, the biggest of which is optional/default arguments are supported (and will generate individually overrideable functions as well, just like they were manually each put in). :slight_smile:

defprotocol_ex Defaults do
  def succ(a, b \\ 1)
end

defimpl_ex Integer, i when is_integer(i), for: Defaults do
  def succ(a, b), do: a + b
end

test "Defaults" do
  assert 2 = Defaults.succ(1)
  assert 3 = Defaults.succ(1, 2)
end
3 Likes