Function generation / dynamic dispatch

Questions / Help
,
1

I am writing an application that reads incoming packets and I’d like to decode them and route them to the appropriate handler.

The header of each packet is what defines its type.
So for example a user login packet may look like this:

<<
  header::integer-little-8,
  ulen::unsigned-little-integer-32,
  user::binary-size(ulen),
  plen::unsigned-little-integer-32,
  pass::binary-size(plen)
>>

I’d like to be able to do something like this in my receive function:

<<
  header::integer-little-8,
  data::bytes
>> = received

{:ok, packet} = Packet.parse(header, data)
:ok = packet.run()

Where Packet.parse/2 is defined at compile time.

defmodule Packet do
   defmacro __using__(header) do
     # track __CALLER__ and header?
     @behaviour Packet
   end

   @callback parse(BitString) :: {:ok, Map}
   @callback def run() :: :ok | {:error, reason}

   # Generate these at compile time
   def parse(1, data), do: Packets.Login.parse(data)
   def parse(2, data), do: Packets.Chat.parse(data)
end

defmodule Packets.Login do
  use Packet, header: 1

  defstruct :username, :password

  def parse(packet_data), do: # decode packet
  def run, do: # run code to handle packet
end

Is there a way to do this using macros or a better way that wouldn’t require macros?

2

My current favorite library to read for learning about neat code generation is NimbleParsec. It might give you some ideas. :slight_smile:

Edit: The Elixir Forum thread for the library is also good- searching it for “quoted expressions” might give you a few more ideas.

1 Like
3

I don’t think you need a macro for this. Something like this should do it. Of course you would probably
encapsulate the header->function mapping differently in real use rather than a module attribute.

defmodule Dynamic do
  @function_map [
    {1, Packets.Login},
    {2, Packets.Chat}
  ]

  def process_packet(<< header::integer-little-8, data::bytes >>) do
    process_packet(header, data)
  end

  for {header, module} <- @function_map do
    function_name = Module.concat(module, "parse")
    def process_packet(unquote(header), data), do: unquote(function_name)(data)
  end
end
2 Likes
4

Part of my goal is to avoid creating a map of header => Module manually. I want the packet header to be part of the packet definition and all the parse functions to be generated.

I just can’t seem to figure out how to create that list at compile time. Heck if I could even just extend the Packet module that would be great.

defextend Packet do
    def parse(1, data), do: # parse packet
end

defmodule Packets.Login do
  def run() do: # logic to handle the packet
end

I just can’t seem to figure out a way to do this. Even if I keep a module attribute around using persist: true I will end up with an error because the module gets compiled and I can no longer continue to Module.put_attribute

I really just don’t want to end up with a separate file to touch every time a new packet type is created and I also don’t want to have a module that is just a massive list of mappings if this kind of thing can be generated.

5

Um, so how do you know at compile type what the packet types are, and what the processing should be for each packet? I’m surely misunderstanding you but it sounds like you’re saying something like this, which pattern matches on the header. These can easily be generated at compile time - but you’d still need to know what the processing-per-packet would be?

defmodule Dynamic do
  def process_packet(<< 1::integer-little-8, data::bytes >>) do
    :do_something_with_1
  end

  def process_packet(<< 2::integer-little-8, data::bytes >>) do
    :do_something_with_2
  end

end
1 Like
6

You’re right that I want to pattern match on the header. I want the header to be defined in the module that handles the actual packet though. Otherwise you end up with a module that is nothing but a large list mapping the header numbers to modules. I don’t want to separate that information.

If I was able to get every module that has use Packet, header: 1 or implements a Packet behaviour, or every module with a @packet_header attribute, then I could build the list without having to list them all out manually.

7

Here’s some code that will introspect the known modules and identify if they implement a behaviour. I’ve shown two ways to get the known modules, one is Elixir release dependent, the other uses the Mix api. Both are really only available in a build environment but I guess thats OK for your use. The second issue is that they introspect artefacts that are already compiled which may introduce some dependency issues.

defmodule Implementation do
  @doc """
  Returns the functions known in the 
  Elixir compiler manifest.
  
  Depends on the implementation of the
  manifest which is not guaranteed
  to stay the same from release to
  release
  """
  def elixir_modules do
    Mix.Project.manifest_path()
    |> Path.join("compile.elixir")
    |> File.read!
    |> :erlang.binary_to_term
    |> Enum.map(fn
        {:module, module, _, _, _, _, _} -> module
        {:source, _, _, _, _, _, _, _, _, _} -> nil
        n when is_integer(n) -> :nil
       end)
    |> Enum.reject(&is_nil/1)   
  end
  
  @doc """
  Returns modules known to Mix.Tasks.Xref
  
  This is a consistent API. 
  """
  def caller_modules do
    Mix.Tasks.Xref.calls 
    |> Enum.flat_map(&([&1.caller_module, elem(&1.callee, 0)])) 
    |> Enum.uniq
  end
  
  @doc """
  Returns modules that implement a specific behaviour
  """
  def behaviour(name) do
    caller_modules()
    |> Enum.filter(&implements_behaviour?(&1, name))
  end
  
  defp implements_behaviour?(module, name) do
    behaviours = 
      module.__info__(:attributes)
      |> Keyword.get(:behaviour, [])
      
    name in behaviours
  end
end
8

Using the code above you could then generate the functions you’re after with something like:

defmodule PacketHandler do
  @moduledoc """
  Assuming the @behaviour :packet
  includes the callback `:packet_type` to
  return the type of packet it processes
  """
  for module <- Implementation.behaviour(:packet) do
    packet_type = module.packet_type()
    call = Module.concat(module, :parse)
    def parse(unquote(packet_type), data), do: unquote(call)(data)
  end
end
9

I really should release my library of mine that does that at compile-time… >.>

Hmm, actually you could use my ProtocolEx (it’s like the Protocol Library built into elixir, except it is based on matchers and guards instead of types, and has the ability to be faster too by inlining and more) library if you don’t mind putting your parse functions in implementation modules (you can of course put multiples of the parse functions in a single implementation if you want, so you can group like-functionality. :slight_smile:

3 Likes
10

For some reason the functions are not getting generated unless I compile the app, modify the router

I have defined 2 modules

packet.ex

defmodule Packet do
  @callback packet_type() :: integer
  @callback parse(any) :: {:ok, struct}
  @callback run(struct) :: :ok | {:ok, any} | {:error, String.t()}
end

router.ex

defmodule Packet.Router do
  defmacro __using__(opts) do
    app = Keyword.get(opts, :app, %{})

    for module <- Packet.Router.behaviour(:Packet, app, __CALLER__.module) do
      call = Module.concat(module, :parse)
      packet_type = module.packet_type()

      quote do
        def parse(unquote(packet_type), data), do: unquote('#{inspect(call)}(data)')
      end
    end
  end

  @doc """
  Returns modules that implement a specific behaviour
  """
  def behaviour(name, app, caller) do
    name = :"Elixir.#{to_charlist(to_string(name))}"

    caller_modules()
    |> Enum.filter(&implements_behaviour?(&1, name, app, caller))
  end

  defp caller_modules do
    Mix.Tasks.Xref.calls()
    |> Enum.flat_map(&[&1.caller_module, elem(&1.callee, 0)])
    |> Enum.uniq()
  end

  defp implements_behaviour?(module, name, app, caller) do
    app_str = "#{to_charlist(String.capitalize(to_string(app)))}"

    try do
      in_app = Kernel.in(app_str, Module.split(module)) && module !== caller

      case in_app do
        true ->
          behaviours =
            module.__info__(:attributes)
            |> Keyword.get(:behaviour, [])

          name in behaviours

        _ ->
          false
      end
    rescue
      _e in UndefinedFunctionError ->
        implements_behaviour?(module, name, app, caller)

      _ ->
        false
    end
  end
end

Now if I use this by defining

server/packet/login.ex

defmodule Server.Packet.Login do
  @behaviour Packet
  defstruct username: "", password: ""
  def packet_type(), do: 1
  def parse(packet), do: # decode packet and return struct
  def run(user = %Server.Packet.Login{}), do: # run some code to handle the packet
end

server/packet/router.ex

defmodule Server.Packet.Router do
  use Packet.Router, app: :server
end

The router does not end up with any parse functions in it.
If I compile the app and then modify server/packet/router.ex (add or remove a blank line) then recompile the parse function does get generated.

Have I done something wrong here or did I miss something important?

11

That just means the compile cannot figure out the dependency chain properly, adding some require’s would help that but that would miss ‘new’ ones, or making the module as always dirty would fix it but would have to recompile it every time, or a compiler plugin (as ProtocolEx does) would always work instead too.

Basically the issue is that the Elixir compiler doesn’t know when it needs to recompile a module, which require’s fixes for known things, but for unknown things it doesn’t know when to add the dependency chain.

12

I’ve been messing around with require quite a bit and still can’t seem to get it to work. What seems to be happening is Mix.Tasks.Xref.calls() comes up empty on first compile. If I edit what seems to be just about any file related to the router and then recompile, it comes up with a list of callers like I expect.

Doing a require on the router module, packet module, or even requiring the actual packet definition in the router does not seem to get it working.

I’ve started looking through the code on github for mix xref callers because it seems to always list out the dependency’s of Packet correctly (or at least the files that use it). I’m hoping that will shed some light on a possible solution.

I have to admit I’m still fairly new to elixir. If you think creating a compiler plugin is a good way to go I would love to know where I can find related documentation so I can learn more. If you don’t mind can you help point me in the right direction here?

13

That says they are either on the wrong file or not put in early in the compilation process.

Honestly I’m thinking this is just a bit odd of a design overall regardless, I would probably just hardcode those all in as it’s not like they are going to be changing often anyway and adding new ones is easy. ^.^;

If you really want to go the more complex route (which be sure that you do), I’d really say take a look at my afore-mentioned protocol_ex, your solution might look similar to (typed in-post so likely has syntax errors):

# Define the main module:
import ProtocolEx
defprotocolEx Blah do
  def parse(header, data) # Since no fallback body then this must be handled by the implementations

  def parse(<<header::integer-little-8, data::bytes), do: parse(header, data)
end

Then just write the implementations like:

import ProtocolEx
defimplEx Login, 1, for: Packet do
  def parse(_, data) do
    # whatever you need here...
  end
  # You could add more `parse/2` callbacks here that handle other numbers as well if you want,
  # or make other implementations like below
end

Then maybe another one somewhere else:

import ProtocolEx
defimplEx Chat, 2, for: Packet do
  def parse(_, data) do
    # whatever you need here...
  end
end

Etc…

And of course you can add a run function and all sorts as well too. Whatever. (If you need new features in protocol_ex, just ask). It has a lot of other abilities and optimizations and such you can do as well, but as-is it already does what you posted that you want.

Don’t forget to add the compile for protocol_ex to the mix.exs file as per the documentation too (or you can always handle it manually too if you want).

(EDIT: Updated examples to be actual code rather than just placeholder code)

4 Likes
14

I’ll likely either hard code or use ProtocolEx as you’ve suggested. ProtocolEx does look really nice, the only reason I’ve been shying away from it is this is a learning experience for me.

Prior to this project I didn’t even know the difference between use, import, and require. As I’m reading and learning I’m trying to use everything I’m learning in some way even if it’s not the most appropriate for the situation. I need to get it to sink in somehow! :slight_smile:

I appreciate everyone’s patience and I’ve been really surprised by how helpful and knowledgeable the Elixir community is. Thank you for the examples, links, and information you’ve provided. It’s been a huge help.

1 Like
15

We try, don’t hesitate to ask! :slight_smile: