Why are behaviour methods referred to as "callbacks"?

fireproofsocks · February 13, 2019, 3:27pm

Coming from an OO background, my first reaction is to think of behaviours as interfaces. So it makes sense that a module must define one or more functions outlined in a behaviour if it wishes to implement that behaviour.

So why in Elixir are the behaviour’s functions referred to as “callbacks”? In other languages, a “callback” has quite different connotations which have little to do with inheritance.

In fact, the page at https://elixir-lang.org/getting-started/typespecs-and-behaviours.html has left me scratching my head, especially the section on “Dynamic dispatch”. Since “callbacks” are apparently not at all what you might think, it makes me really wonder what exactly is meant by “dynamic dispatch”. There are no clarification or examples in that section (!!!), only a reassurance that “you don’t need to define a behaviour in order to dynamically dispatch on a module, but those features often go hand in hand.”

Can someone help shed light on this? Thank you!

peerreynders · February 13, 2019, 3:29pm

In simple terms the “behaviour” already exists as the behaviour module capturing the generic parts. You are implementing the callback module which contains the specific parts. To execute those specific parts the behaviour module calls the “callback” functions on the callback module.

The behaviour module is passed a callback module much in the same way a function is passed a callback function - the difference is that the behaviour module expects certain functions to be implemented by the callback module rather than there just being that one function.

defmodule Parser do
  @callback parse(String.t) :: {:ok, term} | {:error, String.t}
  @callback extensions() :: [String.t]

  def parse!(callback_module, contents) do
    case callback_module.parse(contents) do
      {:ok, data} -> data
      {:error, error} -> raise ArgumentError, "parsing error: #{error}"
    end
  end
end

I swapped implementation with callback_module - so it makes sense that callback_module.parse/1 is a callback function.

Another example from the Access behaviour

 def fetch(%module{} = container, key) do
    module.fetch(container, key)
  rescue
    exception in UndefinedFunctionError ->
      raise_undefined_behaviour(exception, module, {^module, :fetch, [^container, ^key], _})
  end

The fetch/2 function implemented by Access grabs the callback module from the struct so it can turn around and call the module.fetch/2 callback function.

With behaviours you are composing the callback module with the behaviour module - there is no inheritance.

fireproofsocks · February 13, 2019, 6:50pm

Thank you for the explanation, but I still don’t follow the execution flow here. When you call a function on an implementation, I don’t see how that has anything to do with the behaviour module other than to reference the function signature. I guess I don’t follow what happens when you use @impl above the function…

Could you field an example of how to actually call the parse!() function? I assume you would do something like Parser.parse!(MyParser, contents), but that syntax just seems really backwards.

asummers · February 13, 2019, 6:57pm

@impl MyBehaviour would give a compiler error if you missed a callback that you were required to implement per the behaviour or if the function that has the @impl is not actually part of the behaviour.

E.g. if I have

defmodule Foo do
  @callback foo() :: :ok
end
defmodule Bar do
  @behaviour Foo

  @impl Foo
  def bar() do
    :ok
  end
end

then you would get a compiler error. Similarly,

defmodule Foo do
  @callback foo() :: :ok 
  @callback bar() :: :ok
end
defmodule Bar do
  @behaviour Foo

  @impl Foo
  def bar() do
    :ok
  end
end

would also give a compiler error. I actually don’t remember if it’s a warning or an error, but we run mix compile --warnings-as-errors so they’re the same to us. @impl is purely a safeguard to help against changing behaviours and during refactoring, but it’s optional.

NobbZ · February 13, 2019, 7:01pm

You have not yet used GenServers? They do it very similar to this…

peerreynders · February 13, 2019, 7:15pm

You have not yet used GenServers? They do it very similar to this…

Just to underline this point: example

The Demo module is the callback module. None of the usual behaviour ceremony is actually necessary - other than implementing the mandatory callback functions.

{:ok, pid} = GenServer.start_link(Demo,[])

The Demo callback module is composed with the GenServer behaviour module leading to a GenServer based process with the specialized Demo functionality.

In OO:

parse!/1 would be a method on the abstract class. It may have some generic logic but doesn’t do the actual work. It would call a hook method do_parse/1 that is left for the subclass to implement the content specific parsing.
The subclass overides do_parse/1 implementing the content specific implementation. You use the subclass to parse the content it is specialized for.

So:

Parser.parse!/2 is the generic function in the behaviour module that will delegate the details of parsing the specific content to MyParser.
MyParser is a collection of hook function implementations needed to parse the specific content (organized as a callback module).

LostKobrakai · February 13, 2019, 9:14pm

It’s backwards because MyParser is known and you could just call MyParser.parse/1. But the code using a callback module most often don’t know said callback module (MyParser) in advance, but rather it’s configured or passed in at runtime / startup. This is where behaviours shine. Like OTP has the :gen_server behaviour shipped with erlang, but you are implementing the callback module to make it actually do something in your project. Another example is swoosh/bamboo. They just say an adapter needs to implement a set of function to be usable by them and anybody can implement is using their favorite email service.

This also makes those modules be comparable to callback functions. You pass your own implementation of something into pre-existing code, which can then call into your supplied implementation whenever needed. It’s just that a callback function is just a single function, but a callback module for a behaviour can bundle multiple functions.

peerreynders · February 14, 2019, 2:48am

The free sample of Designing for Scalability with Erlang/OTP: Implement Robust, Fault-Tolerant Systems on Google play has the complete Chapter 3. Behaviors. The code is in Erlang - for the equivalent Elixir code see below. Points of note:

Here the Frequency callback module injects itself into the Server behaviour module through its start/0 function.
The Server behaviour module loop/2 function keeps the callback module that the process was initialized with as part of the loop state. The arguments to the loop are 1.) the callback module, 2.) the state which is unique to this particular process and transformed by the callback module.

The callbacks implemented by the Frequency callback module are:

init/1 (used in Server.init/2)
handle/2 (used in Server.loop/2)
terminate/1 (used in Server.loop/2)

Frequency before behaviour separation

#
# file: freq.exs - based on:
#   "Designing for Scalability with Erlang/OTP Implement Robust, Fault-Tolerant Systems"
#   by Francesco Cesarini and Steve Vinoski (O’Reilly).
#   Copyright 2016 Francesco Cesarini and Stephen Vinoski, 978-1-449-32073-7.
#
#   https://github.com/francescoc/scalabilitywitherlangotp/blob/master/ch3/frequency.erl
#
defmodule Frequency do
  def start do
    __MODULE__
    |> Kernel.spawn(:init, [])
    |> Process.register(__MODULE__)
  end

  def init do
    # state: {[free frequencies], [allocated frequencies]}
    frequencies = {get_frequencies(), []}
    loop(frequencies)
  end

  ## hard-coded
  defp get_frequencies(),
    do: Enum.to_list(10..15)

  def stop,
    do: call(:stop)

  def allocate,
    do: call(:allocate)

  def deallocate(freq),
    do: call({:deallocate, freq})

  defp call(message) do
    send(__MODULE__, {:request, self(), message})

    receive do
      {:reply, reply} ->
        reply
    end
  end

  defp reply(pid, reply),
    do: send(pid, {:reply, reply})

  defp loop(frequencies) do
    receive do
      {:request, pid, :allocate} ->
        {new_frequencies, reply} = allocate(frequencies, pid)
        reply(pid, reply)
        loop(new_frequencies)

      {:request, pid, {:deallocate, freq}} ->
        new_frequencies = deallocate(frequencies, freq)
        reply(pid, :ok)
        loop(new_frequencies)

      {:request, pid, :stop} ->
        reply(pid, :ok)
    end
  end

  ## The internal helper functions used to allocate and
  ## deallocate frequencies.
  defp allocate({[], _allocated} = frequencies, _pid),
    do: {frequencies, {:error, :no_frequency}}

  defp allocate({[freq | free], allocated}, pid),
    do: {{free, [{freq, pid} | allocated]}, {:ok, freq}}

  defp deallocate({free, allocated}, freq) do
    new_allocated = List.keydelete(allocated, freq, 0)
    {[freq | free], new_allocated}
  end
end

freqs = []
IO.inspect(Frequency.start())
IO.puts("> started")
# allocate 10
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 11
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 12
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 13
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 14
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 15
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate error
IO.inspect(Frequency.allocate())
freqs = :lists.reverse(freqs)
# deallocate 10
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# allocate 10
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# deallocate 10
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 11
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 12
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 13
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 14
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 15
[freq | _] = freqs
IO.inspect(Frequency.deallocate(freq))
IO.puts("> stopping")
IO.inspect(Frequency.stop())

After split into the generic Server behaviour module and the specific Frequency callback module

#
# file: server.exs - based on:
#   "Designing for Scalability with Erlang/OTP Implement Robust, Fault-Tolerant Systems"
#   by Francesco Cesarini and Steve Vinoski (O’Reilly).
#   Copyright 2016 Francesco Cesarini and Stephen Vinoski, 978-1-449-32073-7.
#
#   https://github.com/francescoc/scalabilitywitherlangotp/blob/master/ch3/behavior/server.erl
#
#   https://github.com/francescoc/scalabilitywitherlangotp/blob/master/ch3/behavior/frequency.erl
#
defmodule Server do
  # behaviour module (generic)

  def start(module, args) do
    __MODULE__
    |> Kernel.spawn(:init, [module, args])
    |> Process.register(module)
  end

  def stop(module) do
    send(module, {:stop, self()})

    receive do
      {:reply, reply} ->
        reply
    end
  end

  def init(module, args) do
    state = module.init(args)
    loop(module, state)
  end

  def call(name, msg) do
    send(name, {:request, self(), msg})

    receive do
      {:reply, reply} ->
        reply
    end
  end

  defp reply(to, reply),
    do: send(to, {:reply, reply})

  ## note how the callback "module" is part
  ## of the loop state
  ##
  defp loop(module, state) do
    receive do
      {:request, from, msg} ->
        {new_state, reply} = module.handle(msg, state)
        reply(from, reply)
        loop(module, new_state)

      {:stop, from} ->
        reply = module.terminate(state)
        reply(from, reply)
    end
  end
end

defmodule Frequency do
  # callback module (specific)

  # Here the behaviour module (Server) is provided with
  # the callback module (Frequency)
  def start,
    do: Server.start(__MODULE__, [])

  # state: {[free frequencies], [allocated frequencies]}
  def init(_args),
    do: {get_frequencies(), []}

  ## hard-coded
  defp get_frequencies(),
    do: Enum.to_list(10..15)

  def stop,
    do: Server.stop(__MODULE__)

  def allocate,
    do: Server.call(__MODULE__, {:allocate, self()})

  def deallocate(freq),
    do: Server.call(__MODULE__, {:deallocate, freq})

  def handle({:allocate, pid}, frequencies),
    do: allocate(frequencies, pid)

  def handle({:deallocate, freq}, frequencies),
    do: {deallocate(frequencies, freq), :ok}

  def terminate(_frequencies),
    do: :ok

  ## The internal helper functions used to allocate and
  ## deallocate frequencies.
  defp allocate({[], _allocated} = frequencies, _pid),
    do: {frequencies, {:error, :no_frequency}}

  defp allocate({[freq | free], allocated}, pid),
    do: {{free, [{freq, pid} | allocated]}, {:ok, freq}}

  defp deallocate({free, allocated}, freq) do
    new_allocated = List.keydelete(allocated, freq, 0)
    {[freq | free], new_allocated}
  end
end

freqs = []
IO.inspect(Frequency.start())
IO.puts("> started")
# allocate 10
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 11
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 12
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 13
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 14
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate 15
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# allocate error
IO.inspect(Frequency.allocate())
freqs = :lists.reverse(freqs)
# deallocate 10
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# allocate 10
{:ok, freq} = IO.inspect(Frequency.allocate())
freqs = [freq | freqs]
# deallocate 10
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 11
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 12
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 13
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 14
[freq | freqs] = freqs
IO.inspect(Frequency.deallocate(freq))
# deallocate 15
[freq | _] = freqs
IO.inspect(Frequency.deallocate(freq))
IO.puts("> stopping")
IO.inspect(Frequency.stop())

$ elixir server.exs
true
> started
{:ok, 10}
{:ok, 11}
{:ok, 12}
{:ok, 13}
{:ok, 14}
{:ok, 15}
{:error, :no_frequency}
:ok
{:ok, 10}
:ok
:ok
:ok
:ok
:ok
:ok
> stopping
:ok
$

fireproofsocks · February 14, 2019, 6:33pm

Thank you for so many educational responses! So the name “callbacks” DOES make sense, but ONLY for the parse! function where we are passing in a module and then calling a method on it. I still don’t see any callbacks happening for our regular behaviour implementations. How about the following?:

Parser.parse(AnotherParser, contents)

That would make sense: at least that looks like it is performing a dispatching/callback option, but of course that does not work because function Parser.parse/2 is undefined or private.

So when we actually define a @callback and an @impl of it, then as far as I can see, we aren’t actually DOING a callback. We are just enforcing that implementations follow our behaviour blueprint in the same way that OO languages implement an interface and must implement the required methods.

When we define the parse!/2 function and provide a module and actually DO module dispatch via a callback, then we aren’t actually directly using anything to do with the @callback or @impl functions. In principle, we could do that entire module-dispatch and callback stuff entirely without any of the @behaviour, @callback, or @impl stuff.

Forgive me for being dense, but the naming convention here still does not make sense to me. I would not call those @callbacks, I would call them “signatures”, “members”, “actions”, or almost anything other than a loaded term like “callbacks” because nothing about them requires a callback: you can write them without delving into behaviours at all! “Callbacks” really seems like the wrong word for what is going on here and it has sent my brain on a wild goose chase. It seems that they were named after HOW they get typically used and NOT after what they actually ARE. For me at least, this has caused a lot of mental friction.

Summary

The best I can sum up for myself after the patient guidance of the contributors here is this:

The name @callbacks can be a confusing misnomer! A @callback simply defines a function signature that every implementation of the behaviour must define. This is conceptually the same as an Object Oriented class implementing all of the functions defined by an interface.
The practice of using “dynamic dispatch” works well with behaviours: your “dispatching” function can accept a module as input, then call a method on that module. Behaviours offer a way to guarantee that the module provided as input has implemented the function that you wish to call on it. E.g. in the following example, you had better be 100% sure that your provided target module has implemented the do_something/1 function:

    def dispatch_to_module(target_module, args) do
        target_module.do_something(args)
    end

asummers · February 14, 2019, 6:45pm

Not to detract from your point, but the name callback in the Erlang case is 30+ years old at this point. Some of the original authors that pop around here every now and then (rvirding from above, e.g.) may be able to comment on the etymology of the name but due to the telephony nature of Erlang, I think it may actually be referencing a literal (phone) call back. GenServer has a concept called call so I would imagine it stemmed from the generic behavior around GenServers. That’s pure speculation, however.

NobbZ · February 14, 2019, 7:24pm

In erlang many terms are meant a bit different than what I’ve learnt those terms mean from CS courses. I learnt to deal with it. You should do so as well. Many “modern” terms only got their meaning a couple of years ago, while they are used in erlang a lot longer…

Callback might be one of these, which got its negative conotation only recently through JavaScript…

peerreynders · February 14, 2019, 9:49pm

Thought experiment - referring to the Frequency callback module from my last post

def start,
    do: Server.start(__MODULE__, [])

That sample code (server.exs) can be easily rewritten to work in the following fashion:

def start,
    do: Server.start(
          &__MODULE__.init/1,
          &__MODULE__.handle/2,
          &__MODULE__.terminate/1,
          []
        )

Are init/1, handle/2, terminate/1 callbacks now?

Wouldn’t developers say: Why can’t I just pass the module that implements those functions?