If I understand correctly, this protocol resembles the OSI model, where multiple layers are stacked on top of each other with the following properties:

1. Each layer receives input packets and produces decoded packets for the consumer layer (the one directly on top of it).
2. Some layers might need to perform additional imperative logic, such as send a response to the peer layer on the other side, or defer an action for later.

Stacked behaviours

One way of implementing this would be to rely on stacked behaviours. The most foundational one, called Layer0 would receive raw bits (e.g. using gen_tcp) and invoke layer-specific callbacks, such as handle_packet which receives a decoded layer 0 packet. On top of this abstraction you can build Layer1, on top of that Layer2, and so on.

This approach allows you to test at lower levels. I’d still test as much as I can at the final level (Application), moving deeper to test the scenarios which are harder to setup at the higher level, such as lower-level timeout.

One downside of is that the implementation will always be coupled to the physical transport. If layer 0 uses tcp to communicate, you’ll always need a fake tcp peer to test the logic, which could become cumbersome to implement. Another issue is that implementing 6 behaviours could lead to a lot of boilerplate. For example, to accommodate deferred actions you’ll probably need to support the handle_info callback in the corresponding layer and all the layers below it.

Functional core, imperative shell

Another option is to go for functional core, imperative shell. You can see a detailed example in my post To spawn, or not to spawn?.

Basically in this approach we distinguish between the driver and the functional logic. The driver takes care of receiving bits/bytes from the remote client and shipping the responses back. It’s also responsible for time-based logic. So that’s the imperative shell. The functional core is a pure-functional state machine which receives inputs from the driver and produces outputs. At the highest level it could look something like:

defmodule KNX do
  @spec new(...) :: t

  @spec handle_impulse(t, impulse) :: [action]
end

where types have the following meaning:

• t - SM state represented as a map or a struct
• impulse - external actions such as {:bytes, binary}, {:message, payload}
• action - an imperative action that must be executed by the driver, such as {:send_bytes, binary}, {:delayed_impulse, time::pos_integer, payload}

With such contract, the driver would be a stateful process that establishes the connection to the remote client, accept bytes, sends impulses to the functional core (which is kept as the state of the process), and interprets the actions. An immediate benefit here is that you can test the core without needing to fake a remote TCP peer, and without needing to depend on time. For example, if the core sends {:delayed_impulse, ...} (a delayed self-send), the test can immediately pass the impulse back.

Inside the core module, I’d start by implementing the zero layer as two private functions:

@spec new_layer_0(...) :: layer_0_state

@spec handle_layer_0_impulse(layer_0_state, layer_0_impulse) :: [layer_0_actions]

This follows the same pattern as the high-level KNX API, and these functions would initially reside in the same module (I’d move them to a separate “sub-module” if the code grows). The impulses to layer 0 are e.g. raw bits/bytes, and the output actions are decoded packets.

Now on top of this we can build layer 1. When layer 1 receives an impulse, it needs to make a decision. If the impulse specific to layer-1 (e.g. a delayed self-send), the layer handles it and produces output actions. Otherwise, the lower-level layer is invoked to handle the impulse.

In the latter case, layer 0 will return a set of actions. Some of these actions will be decoded packets. Layer 1 has to interpret each decoded packet and produce its own additional actions. So for example, let’s say that layer 0 returns:

[
  layer_0_action_1,
  layer_0_action_2,
  layer_0_decoded_packet_1,
  layer_0_action_3,
  ...
]

Layer 1 could produce:

[
  layer_0_action_1,
  layer_0_action_2,
  layer_1_action_1,
  layer_1_decoded_packet_1,
  layer_0_action_3,
  ...
]

So by stacking all these layers you end up with the top-level handle_impulse that will return actions required by different layers in the proper order.

This approach also supports testing at the lower levels. You can expose each individual layer API, e.g. as public functions marked with @doc false, or by moving each layer into its own submodule. Again, I’d test as much as I can at the highest possible level.

FWIW I used this style to implement the server-side of the PostgreSQL protocol (only 1 layer, but still), and it has its pros and cons. As said, the biggest pro is that you decouple the protocol from the imperative logic, which simplifies testing. On the flip side, any combination of imperative + conditional logic will be trickier to implement. E.g. if you need to send something to the peer and than interpret the peer’s response, you need to produce an output action, store some piece of information in your state, and then reconcile that with a response impulse. This is much trickier than plain old imperative interpret_response(send(peer, ...)), where send is a blocking operation that awaits for the peer’s response.

But I think that you’ll face the same challenge in the first proposed approach (behaviour-based). In general, a straightforward imperative style is blocking, which means that you lose the ability to perform asynchronous actions (e.g. ship something else to the peer while you’re awaiting for its response).