Function generation / dynamic dispatch

We try, don’t hesitate to ask!

I’ll likely either hard code or use ProtocolEx as you’ve suggested. ProtocolEx does look really nice, the only reason I’ve been shying away from it is this is a learning experience for me.

Prior to this project I didn’t even know the difference between use, import, and require. As I’m reading and learning I’m trying to use everything I’m learning in some way even if it’s not the most appropriate for the situation. I need to get it to sink in somehow!

I appreciate everyone’s patience and I’ve been really surprised by how helpful and knowledgeable the Elixir community is. Thank you for the examples, links, and information you’ve provided. It’s been a huge help.

That says they are either on the wrong file or not put in early in the compilation process.

Honestly I’m thinking this is just a bit odd of a design overall regardless, I would probably just hardcode those all in as it’s not like they are going to be changing often anyway and adding new ones is easy. ^.^;

If you really want to go the more complex route (which be sure that you do), I’d really say take a look at my afore-mentioned protocol_ex, your solution might look similar to (typed in-post so likely has syntax errors):

# Define the main module:
import ProtocolEx
defprotocolEx Blah do
  def parse(header, data) # Since no fallback body then this must be handled by the implementations

  def parse(<<header::integer-little-8, data::bytes), do: parse(header, data)
end

Then just write the implementations like:

import ProtocolEx
defimplEx Login, 1, for: Packet do
  def parse(_, data) do
    # whatever you need here...
  end
  # You could add more `parse/2` callbacks here that handle other numbers as well if you want,
  # or make other implementations like below
end

Then maybe another one somewhere else:

import ProtocolEx
defimplEx Chat, 2, for: Packet do
  def parse(_, data) do
    # whatever you need here...
  end
end

Etc…

And of course you can add a run function and all sorts as well too. Whatever. (If you need new features in protocol_ex, just ask). It has a lot of other abilities and optimizations and such you can do as well, but as-is it already does what you posted that you want.

Don’t forget to add the compile for protocol_ex to the mix.exs file as per the documentation too (or you can always handle it manually too if you want).

(EDIT: Updated examples to be actual code rather than just placeholder code)

I’ve been messing around with require quite a bit and still can’t seem to get it to work. What seems to be happening is Mix.Tasks.Xref.calls() comes up empty on first compile. If I edit what seems to be just about any file related to the router and then recompile, it comes up with a list of callers like I expect.

Doing a require on the router module, packet module, or even requiring the actual packet definition in the router does not seem to get it working.

I’ve started looking through the code on github for mix xref callers because it seems to always list out the dependency’s of Packet correctly (or at least the files that use it). I’m hoping that will shed some light on a possible solution.

I have to admit I’m still fairly new to elixir. If you think creating a compiler plugin is a good way to go I would love to know where I can find related documentation so I can learn more. If you don’t mind can you help point me in the right direction here?

That just means the compile cannot figure out the dependency chain properly, adding some require’s would help that but that would miss ‘new’ ones, or making the module as always dirty would fix it but would have to recompile it every time, or a compiler plugin (as ProtocolEx does) would always work instead too.

Basically the issue is that the Elixir compiler doesn’t know when it needs to recompile a module, which require’s fixes for known things, but for unknown things it doesn’t know when to add the dependency chain.

For some reason the functions are not getting generated unless I compile the app, modify the router

I have defined 2 modules

packet.ex

defmodule Packet do
  @callback packet_type() :: integer
  @callback parse(any) :: {:ok, struct}
  @callback run(struct) :: :ok | {:ok, any} | {:error, String.t()}
end

router.ex

defmodule Packet.Router do
  defmacro __using__(opts) do
    app = Keyword.get(opts, :app, %{})

    for module <- Packet.Router.behaviour(:Packet, app, __CALLER__.module) do
      call = Module.concat(module, :parse)
      packet_type = module.packet_type()

      quote do
        def parse(unquote(packet_type), data), do: unquote('#{inspect(call)}(data)')
      end
    end
  end

  @doc """
  Returns modules that implement a specific behaviour
  """
  def behaviour(name, app, caller) do
    name = :"Elixir.#{to_charlist(to_string(name))}"

    caller_modules()
    |> Enum.filter(&implements_behaviour?(&1, name, app, caller))
  end

  defp caller_modules do
    Mix.Tasks.Xref.calls()
    |> Enum.flat_map(&[&1.caller_module, elem(&1.callee, 0)])
    |> Enum.uniq()
  end

  defp implements_behaviour?(module, name, app, caller) do
    app_str = "#{to_charlist(String.capitalize(to_string(app)))}"

    try do
      in_app = Kernel.in(app_str, Module.split(module)) && module !== caller

      case in_app do
        true ->
          behaviours =
            module.__info__(:attributes)
            |> Keyword.get(:behaviour, [])

          name in behaviours

        _ ->
          false
      end
    rescue
      _e in UndefinedFunctionError ->
        implements_behaviour?(module, name, app, caller)

      _ ->
        false
    end
  end
end

Now if I use this by defining

server/packet/login.ex

defmodule Server.Packet.Login do
  @behaviour Packet
  defstruct username: "", password: ""
  def packet_type(), do: 1
  def parse(packet), do: # decode packet and return struct
  def run(user = %Server.Packet.Login{}), do: # run some code to handle the packet
end

server/packet/router.ex

defmodule Server.Packet.Router do
  use Packet.Router, app: :server
end

The router does not end up with any parse functions in it.
If I compile the app and then modify server/packet/router.ex (add or remove a blank line) then recompile the parse function does get generated.

Have I done something wrong here or did I miss something important?

I really should release my library of mine that does that at compile-time… >.>

Hmm, actually you could use my ProtocolEx (it’s like the Protocol Library built into elixir, except it is based on matchers and guards instead of types, and has the ability to be faster too by inlining and more) library if you don’t mind putting your parse functions in implementation modules (you can of course put multiples of the parse functions in a single implementation if you want, so you can group like-functionality.

Using the code above you could then generate the functions you’re after with something like:

defmodule PacketHandler do
  @moduledoc """
  Assuming the @behaviour :packet
  includes the callback `:packet_type` to
  return the type of packet it processes
  """
  for module <- Implementation.behaviour(:packet) do
    packet_type = module.packet_type()
    call = Module.concat(module, :parse)
    def parse(unquote(packet_type), data), do: unquote(call)(data)
  end
end

Here’s some code that will introspect the known modules and identify if they implement a behaviour. I’ve shown two ways to get the known modules, one is Elixir release dependent, the other uses the Mix api. Both are really only available in a build environment but I guess thats OK for your use. The second issue is that they introspect artefacts that are already compiled which may introduce some dependency issues.

defmodule Implementation do
  @doc """
  Returns the functions known in the 
  Elixir compiler manifest.
  
  Depends on the implementation of the
  manifest which is not guaranteed
  to stay the same from release to
  release
  """
  def elixir_modules do
    Mix.Project.manifest_path()
    |> Path.join("compile.elixir")
    |> File.read!
    |> :erlang.binary_to_term
    |> Enum.map(fn
        {:module, module, _, _, _, _, _} -> module
        {:source, _, _, _, _, _, _, _, _, _} -> nil
        n when is_integer(n) -> :nil
       end)
    |> Enum.reject(&is_nil/1)   
  end
  
  @doc """
  Returns modules known to Mix.Tasks.Xref
  
  This is a consistent API. 
  """
  def caller_modules do
    Mix.Tasks.Xref.calls 
    |> Enum.flat_map(&([&1.caller_module, elem(&1.callee, 0)])) 
    |> Enum.uniq
  end
  
  @doc """
  Returns modules that implement a specific behaviour
  """
  def behaviour(name) do
    caller_modules()
    |> Enum.filter(&implements_behaviour?(&1, name))
  end
  
  defp implements_behaviour?(module, name) do
    behaviours = 
      module.__info__(:attributes)
      |> Keyword.get(:behaviour, [])
      
    name in behaviours
  end
end

You’re right that I want to pattern match on the header. I want the header to be defined in the module that handles the actual packet though. Otherwise you end up with a module that is nothing but a large list mapping the header numbers to modules. I don’t want to separate that information.

If I was able to get every module that has use Packet, header: 1 or implements a Packet behaviour, or every module with a @packet_header attribute, then I could build the list without having to list them all out manually.