I see the intent behind either as being quite different. However historically OO has often conflated the notion of polymorphism and implementation inheritance - so it would be very easy to fall into the same trap.

• Protocols are ultimately about polymorphism. One part is the interface that will give access to the same polymorphic behaviour. Then each data type wishing to expose that behaviour implements it separately.
• Behaviours are about separating the generic implementation from the specific. A Behaviour captures the common implementation for reuse and uses callbacks to invoke the specific implementation (i.e. the part that varies - a Behaviour acts more like a framework).

Erlang OTP Design Principles: 1.2 Behaviours:

In OO the callback module would likely be implemented as a Strategy.

From a development perspective you are implementing an interface and you are implementing a callback module but that is focussing on the “what you are doing” and the “how you are doing it”.

The “why you are doing it” is quite different.

A “behaviour” defines an API that a module has to provide.

A “protocol” is implemented on top of behaviours and chooses from a set of modules to use when dispatching function calls with data, based on their “type”.

To complement a bit on what @peerreynders wrote. In OTP behaviours are generally coupled to processes in that it is around a process where the generic and specific are defined. The generic typically describes the basic process interface and the specific which callbacks are required and when and how they are called/used. So for example a GenServer is a process with a defined interface and a set of 6 callbacks.