[style] Parametrising generic functionality: behaviours vs protocols

[mjadczak]

Hi,

I’m currently working on a final project/thesis for my CompSci course at the University of Cambridge. I have used Elixir for about a year now and I have decided to port the GoogleDataflow/Apache Beam SDK/local runner to Elixir.

The reference implementations are written in Java and Python, and while I can write well-structured functional code from scratch, porting something using so many OOP features into idiomatic Elixir is a challenge (and one which I intend to treat in my thesis), especially since a lot of the domain-specific concepts are realised in this OOP universe only for now.

I am starting by porting many of the classes used into modules, and replacing inheritance with behaviours (which is already resulting in cleaner code IMO), planning to possibly refactor some parts into a more idiomatic structure at a later point. The sticking point for me is the case where instead of just static functionality overrides, the functionality is parametrised in Python/Java by using instances of objects. For example, elements may be assigned to windows of time, and while an IntervalWindow defines some functionality which conforms to that expected by a Window, it needs different parameters from, say, a GlobalWindow (which needs none).

The default answer for this may be “use protocols”, and indeed Flow uses structs for this purpose (but not protocols, as the windowing functions there do not have to be user-extensible). Indeed for many entities in my framework structs with protocols are the clear solution.

However for certain “behaviours” or strategies, apart from the parametrisation requirement, behaviours seem to be the right solution—a strategy for doing something seems to me like a behaviour, not a data structure. In many cases, any extensions will only need to implement a small subset of the available functionality, a perfect use case of behaviours along with default implementations and defoverridable. Protocols seem to me to imply much looser semantics and restrictions, along the lines of “do this thing, but I don’t care how you do it”, versus the behaviours’ idea of “give me this small custom function which I will plug into my larger system”.

Thinking about this then, my first idea was to simply represent these “parametrised behaviours” as tuples of {Module, data} where data is arbitrary and simply passed along to all of the behaviour functions. But immediately afterwards, I realised this is just Erlang’s tuple modules, one of the things explicitly solved by protocols!

What I am currently doing is indeed using protocols “under the hood”, however I am also using a behaviour to enforce a stricter contract on the using modules. In my __using__ macro, I make the implementing (using) module implement the required behaviour, and define an automatic implementation for the protocol which calls into these behaviour functions (possibly with default implementations).

This way the user is encouraged to think of the module as a behaviour, a pluggable strategy if you will, which can also get a struct of itself as a parameter to its callbacks. Of course there is nothing stopping more advanced users from just providing their own implementation of the protocol, but the goal here is to make it as easy as possible for users to plug their custom functionality into the system.

My question is: is this a code smell? Too much “magic”? Should I just explicitly use protocols and make users explicitly call into the default implementations if they want them? Again, usually there will be 5–6 functions in such a module and a user may override 1 or 2 of them.

I’d appreciate any advice or insight people may have on this.

[josevalim]

All of that makes sense. Protocols are literally behaviours with custom dispatching rules, so your impression that they impose some rules in favor of some features is correct.

You are also correct that Flow could use protocols and the reason we didn’t use them is we are not ready to expose the Window materialization API.

@mjadczak

What I am currently doing is indeed using protocols “under the hood”, however I am also using a behaviour to enforce a stricter contract on the using modules. In my using macro, I make the implementing (using) module implement the required behaviour, and define an automatic implementation for the protocol which calls into these behaviour functions (possibly with default implementations).

That’s the only part I don’t agree with. If you are already using a protocol then you already have a behaviour. Protocols do not allow default implementations because it generating coupling between protocols and implementations. We already have the coupling at the specification level, which is the point of protocols, but moving it to the implementation level means that changing the default implementation of a protocol will affect all implementations that depend on it.

The __using__+defoverridable trick can also be really harmful for augmenting the protocol. For example, imagine the protocol has 2 functions today. In the future, you add a third function but you also provide a default implementation for it. Now implementations have no idea there is a third function! Maybe they would like to implement it but the default implementation means no warning is emitted. Situation can be even worse on libraries pre-1.0: what if you decide to rename a function? With the default implementation, they won’t even know something was renamed and developers will be unaware the contract even changed, wondering for hours why their implementation isn’t being called.

While we cannot solve the first problem (the one the developer will not get a warning if new functions are added to the protocols), we can solve the second one. Instead of:

# Inside quote
def add(a, b) do
  a + b
end

defoverridable add: 2

# Inside implementation
def add(a, b) do
  # ...
end

We should have:

@overridable true
def add(a, b) do
  a + b
end

# Inside implementation
@override true
def add(a, b) do
  # ...
end

This way if you override something that was not marked as overridable or something that does not exist, you will get a warning. It also improves code readability, because anyone reading the code will see the whole purpose of add/2 is to override a callback. Previously readers of the code could be left wondering why there is an add/2 function in the first place.

I want to send a proposal for the new overridable system after Elixir v1.4 is out. So you need to be aware of the limitations of the current system today.

[mjadczak]

Thanks for your reply. Just to clarify then, the @overridable/@override system is not in place yet, but will be submitted in a future proposal?

That’s the only part I don’t agree with. If you are already using a protocol then you already have a behaviour. Protocols do not allow default implementations because it generating coupling between protocols and implementations. We already have the coupling at the specification level, which is the point of protocols, but moving it to the implementation level means that changing the default implementation of a protocol will affect all implementations that depend on it.

The __using__+defoverridable trick can also be really harmful for augmenting the protocol. For example, imagine the protocol has 2 functions today. In the future, you add a third function but you also provide a default implementation for it. Now implementations have no idea there is a third function! Maybe they would like to implement it but the default implementation means no warning is emitted. Situation can be even worse on libraries pre-1.0: what if you decide to rename a function? With the default implementation, they won’t even know something was renamed and developers will be unaware the contract even changed, wondering for hours why their implementation isn’t being called.

I completely understand the rationale there, but like I said, this is a port from an OOP system where there is coupling at the implementation level, and a lot of the functionality depends on it. The idea is that it’s up to the developer of the library to make future changes to default implementations such that they preserve backwards compatibility or at least log some warnings to let developers know if something is wrong.

Further, this is the point I was making before as regards the “feeling” of protocols vs behaviours—protocols are more loosely coupled and just have to conform to the interface, while behaviours can have default implementations and feel more coupled. What I “need” are “parametrised behaviours”, callback functions tightly coupled to an implementation but which can also receive a few custom parameters. Hence my initial solution of using protocol dispatch since it’s already been worked on, tested and tried, but hiding that away somewhat from the user to discourage them from thinking of the module as a protocol implementation, since as you said, that implies much looser coupling than what is actually expected.

I see two possible other solutions:

1. Ask the developers to explicitly opt into default implementations of certain functions, perhaps as parameters to the __using__ macro.
2. Split up the large interfaces into many smaller protocols with one or two functions each, being able to opt into default implementations using @derive.

Number 1 seems somewhat verbose, while number 2 may be something I consider in the “refactoring” phase once the system is actually working.

I realise a lot of the friction here arises from trying to port a large system architected with OOP in mind to Elixir which uses different idioms, but I believe that it’s easier to first move it over into an Elixir codebase and only then make large structural refactorings. I could not find any other examples of large OOP codebases being ported over to something like Erlang or Elixir, usually people opt for a full rewrite from ground-up.

[josevalim]

@mjadczak

Thanks for your reply. Just to clarify then, the @overridable/@override system is not in place yet, but will be submitted in a future proposal?

Yes, it will submitted soon™.

Other than that, it seems you are well aware of the trade-offs involved, which was the goal of my original reply. I would only add that you can also provide default implementations for protocols, if you would like to, exactly as you do for behaviours (but they are also discouraged for exactly the same reasons).

I always find porting such systems fun, exactly because of the challenges in converting from OO to FP.

[OvermindDL1]

@josevalim

I always find porting such systems fun, exactly because of the challenges in converting from OO to FP.

Heh, porting my C++ ECS to Erlang years ago is still one of my most difficult conversions, I was never able to get the performance to anything that I’d consider acceptable (though the purity that I had to sacrifice in C++ for performance made it oh so clean ^.^).