Extensible binary decoding (looking up behaviours)

Hi all

I’m working on a library for KNX, a building automation protocol. In KNX, devices exchange binary encoded messages on a bus. The messages contain a payload, and the specific payload has to be encoded/decoded based on its type (called datapoint type, or DPT for short). DPTs are identified by their ID, which might look like “1.001”, “9.020”, etc.

The KNX spec itself has 300+ DPTs, and it allows for vendors to create their own proprietary DPTs. Because of this, one of the design goals of my library is to make it possible for users of the library to define their own DPTs.

When a message comes in from the bus, it’s adressed to some logical address (e.g. “2/2/10”). Depending on the bus configuration, that address might use DPT 9.020 for its payload. To decode it, I need to lookup the decode function for DPT 9.020. And since the user may have defined the type for this DPT, the lookup must be dynamic.

I have a solution for this which works, but I’d like to check with the community whether there are better ideas out there. My solution looks like this:

  1. All DPTs have their own module, which must implement the KNX.Datapoint.Type behaviour. It looks like this:

    defmodule KNX.Datapoint.Type do
      @callback id() :: String.t() # returns the DPT ID, e.g. "9.020"
      @callback decode(bitstring()) :: {:ok, any()}
      @callback encode(any()) :: {:ok, bitstring()}
      @callback to_string(any()) :: String.t()
    end
    
  2. A special compiler runs after all the other compilers. The compiler will

    1. Loop through all .beam files and find all modules implementing the KNX.Datapoint.Type behaviour (using this code from bitwalker)
    2. Define a module KNX.Datapoint.Types, and for each implementation found, generate a function clause for by_id(String.t()) :: atom(), e.g. def by_id("9.020"), do: KNX.Datapoint.Type.Value_Temp

This works, but since it has its own compiler, it means the users of the library will have to add this compiler to their mix.exs file:

defmodule MyApp.Mixfile do
  ...
  def project() do
    [
      ...
      compilers: Mix.compilers() ++ [:knx],
      ...
    ]
  end
  ...

end

I find that sort of clunky. I’ve never had to do it before (I’ve noticed Phoenix uses it, but they have their own generator which means users don’t have to think about it). I’ve tried adding the compiler in the library, but that makes compiling fail if users define their own types.

I guess another option is to compile the lookup table when the app boots. That would hide it for the user, but when things don’t work that would be a horrible user experience.

So what do you think? Is this a crazy idea? Is there something more straight-forward that could be done?

1 Like

I’ve had my fair share of “hey let’s register those behaviour implementing modules automatically”. I came to the conclustion it’s simply not worth it to use hacks like compile time registration. Just maintain a hardcoded list of implementations. If you really want to depend on elixir dispatching you could also use a protocol for dispatching once you typed your DPTs though.

2 Likes

Yeah, I’ve also considered protocols as an alternative to behaviours. They suffer from the same weakness though - I still need to map the string DPT to a module implementing the protocol. So I still need the lookup table…

I guess I could also specify a convention for mapping a string DPT into module names - so DPT “9.020” would have to be implemented in module KNX.Datatype.Type.DPT9_020. Not very pretty though…

Sure, but what speaks about manually maintaining that lookup table. It’s way easier than trying to jump through hoops just to get it done automatically by some compile time trickery.

How about passing the custom mapping to a macro which builds the parser, a bit like:

https://hexdocs.pm/plug/Plug.Parsers.html#module-examples

or via config like Plug.MIME?

https://hexdocs.pm/plug/1.1.3/Plug.MIME.html

1 Like

Yeah, I guess something like this could work. The user could pass in a list of lookup tables. So the library could ship with a lookup table for all those DPTs defined in the library (which could be managed by hand). If a user wants to provide their own DPTs, they’d define a lookup table for those types, and add it to the default list of lookup tables. And when decoding, the library will just try each lookup table in order.

Could work :slight_smile:

Not as useful when mixing multiple plugins in though.

My ProtocolEx library can give you a list of anything implementing a protocol_ex, and from that… ^.^

I also like the first approach that @dom mentioned (like Plug.Parsers). And this will also make it easy to allow the user to override the default lookups through the same mechanism.