Function generation / dynamic dispatch

My current favorite library to read for learning about neat code generation is NimbleParsec. It might give you some ideas.

Edit: The Elixir Forum thread for the library is also good- searching it for “quoted expressions” might give you a few more ideas.

I don’t think you need a macro for this. Something like this should do it. Of course you would probably
encapsulate the header->function mapping differently in real use rather than a module attribute.

defmodule Dynamic do
  @function_map [
    {1, Packets.Login},
    {2, Packets.Chat}
  ]

  def process_packet(<< header::integer-little-8, data::bytes >>) do
    process_packet(header, data)
  end

  for {header, module} <- @function_map do
    function_name = Module.concat(module, "parse")
    def process_packet(unquote(header), data), do: unquote(function_name)(data)
  end
end

Part of my goal is to avoid creating a map of header => Module manually. I want the packet header to be part of the packet definition and all the parse functions to be generated.

I just can’t seem to figure out how to create that list at compile time. Heck if I could even just extend the Packet module that would be great.

defextend Packet do
    def parse(1, data), do: # parse packet
end

defmodule Packets.Login do
  def run() do: # logic to handle the packet
end

I just can’t seem to figure out a way to do this. Even if I keep a module attribute around using persist: true I will end up with an error because the module gets compiled and I can no longer continue to Module.put_attribute

I really just don’t want to end up with a separate file to touch every time a new packet type is created and I also don’t want to have a module that is just a massive list of mappings if this kind of thing can be generated.

Um, so how do you know at compile type what the packet types are, and what the processing should be for each packet? I’m surely misunderstanding you but it sounds like you’re saying something like this, which pattern matches on the header. These can easily be generated at compile time - but you’d still need to know what the processing-per-packet would be?

defmodule Dynamic do
  def process_packet(<< 1::integer-little-8, data::bytes >>) do
    :do_something_with_1
  end

  def process_packet(<< 2::integer-little-8, data::bytes >>) do
    :do_something_with_2
  end

end

You’re right that I want to pattern match on the header. I want the header to be defined in the module that handles the actual packet though. Otherwise you end up with a module that is nothing but a large list mapping the header numbers to modules. I don’t want to separate that information.

If I was able to get every module that has use Packet, header: 1 or implements a Packet behaviour, or every module with a @packet_header attribute, then I could build the list without having to list them all out manually.

Here’s some code that will introspect the known modules and identify if they implement a behaviour. I’ve shown two ways to get the known modules, one is Elixir release dependent, the other uses the Mix api. Both are really only available in a build environment but I guess thats OK for your use. The second issue is that they introspect artefacts that are already compiled which may introduce some dependency issues.

defmodule Implementation do
  @doc """
  Returns the functions known in the 
  Elixir compiler manifest.
  
  Depends on the implementation of the
  manifest which is not guaranteed
  to stay the same from release to
  release
  """
  def elixir_modules do
    Mix.Project.manifest_path()
    |> Path.join("compile.elixir")
    |> File.read!
    |> :erlang.binary_to_term
    |> Enum.map(fn
        {:module, module, _, _, _, _, _} -> module
        {:source, _, _, _, _, _, _, _, _, _} -> nil
        n when is_integer(n) -> :nil
       end)
    |> Enum.reject(&is_nil/1)   
  end
  
  @doc """
  Returns modules known to Mix.Tasks.Xref
  
  This is a consistent API. 
  """
  def caller_modules do
    Mix.Tasks.Xref.calls 
    |> Enum.flat_map(&([&1.caller_module, elem(&1.callee, 0)])) 
    |> Enum.uniq
  end
  
  @doc """
  Returns modules that implement a specific behaviour
  """
  def behaviour(name) do
    caller_modules()
    |> Enum.filter(&implements_behaviour?(&1, name))
  end
  
  defp implements_behaviour?(module, name) do
    behaviours = 
      module.__info__(:attributes)
      |> Keyword.get(:behaviour, [])
      
    name in behaviours
  end
end

Using the code above you could then generate the functions you’re after with something like:

defmodule PacketHandler do
  @moduledoc """
  Assuming the @behaviour :packet
  includes the callback `:packet_type` to
  return the type of packet it processes
  """
  for module <- Implementation.behaviour(:packet) do
    packet_type = module.packet_type()
    call = Module.concat(module, :parse)
    def parse(unquote(packet_type), data), do: unquote(call)(data)
  end
end

I really should release my library of mine that does that at compile-time… >.>

Hmm, actually you could use my ProtocolEx (it’s like the Protocol Library built into elixir, except it is based on matchers and guards instead of types, and has the ability to be faster too by inlining and more) library if you don’t mind putting your parse functions in implementation modules (you can of course put multiples of the parse functions in a single implementation if you want, so you can group like-functionality.

For some reason the functions are not getting generated unless I compile the app, modify the router

I have defined 2 modules

packet.ex

defmodule Packet do
  @callback packet_type() :: integer
  @callback parse(any) :: {:ok, struct}
  @callback run(struct) :: :ok | {:ok, any} | {:error, String.t()}
end

router.ex

defmodule Packet.Router do
  defmacro __using__(opts) do
    app = Keyword.get(opts, :app, %{})

    for module <- Packet.Router.behaviour(:Packet, app, __CALLER__.module) do
      call = Module.concat(module, :parse)
      packet_type = module.packet_type()

      quote do
        def parse(unquote(packet_type), data), do: unquote('#{inspect(call)}(data)')
      end
    end
  end

  @doc """
  Returns modules that implement a specific behaviour
  """
  def behaviour(name, app, caller) do
    name = :"Elixir.#{to_charlist(to_string(name))}"

    caller_modules()
    |> Enum.filter(&implements_behaviour?(&1, name, app, caller))
  end

  defp caller_modules do
    Mix.Tasks.Xref.calls()
    |> Enum.flat_map(&[&1.caller_module, elem(&1.callee, 0)])
    |> Enum.uniq()
  end

  defp implements_behaviour?(module, name, app, caller) do
    app_str = "#{to_charlist(String.capitalize(to_string(app)))}"

    try do
      in_app = Kernel.in(app_str, Module.split(module)) && module !== caller

      case in_app do
        true ->
          behaviours =
            module.__info__(:attributes)
            |> Keyword.get(:behaviour, [])

          name in behaviours

        _ ->
          false
      end
    rescue
      _e in UndefinedFunctionError ->
        implements_behaviour?(module, name, app, caller)

      _ ->
        false
    end
  end
end

Now if I use this by defining

server/packet/login.ex

defmodule Server.Packet.Login do
  @behaviour Packet
  defstruct username: "", password: ""
  def packet_type(), do: 1
  def parse(packet), do: # decode packet and return struct
  def run(user = %Server.Packet.Login{}), do: # run some code to handle the packet
end

server/packet/router.ex

defmodule Server.Packet.Router do
  use Packet.Router, app: :server
end

The router does not end up with any parse functions in it.
If I compile the app and then modify server/packet/router.ex (add or remove a blank line) then recompile the parse function does get generated.

Have I done something wrong here or did I miss something important?

That just means the compile cannot figure out the dependency chain properly, adding some require’s would help that but that would miss ‘new’ ones, or making the module as always dirty would fix it but would have to recompile it every time, or a compiler plugin (as ProtocolEx does) would always work instead too.

Basically the issue is that the Elixir compiler doesn’t know when it needs to recompile a module, which require’s fixes for known things, but for unknown things it doesn’t know when to add the dependency chain.