Command Pattern via TCP

Questions / Help
,
1

Hi all, I come from a relatively brief Erlang background from many years ago, and I’m trying now to think (again) in that distributed way - while learning Elixir, which I prefer at a glance.

In short, I’d like to know what (if any) is the Elixir-way to deal with a pattern I encounter often my side projects: the command pattern.

Assume I have a little TCP server setup (via ranch, in fact) and I want to have a clean way of adding commands to manipulate “global state” (currently, I’m using Mnesia… but I already feel like this is not well supported in Elixir. A question for another day).

I went the polymorphism way, so each command is a module with behaviour Command which, in itself, defines callbacks. A series of processes run these commands.

However, clearly each string coming from the socket needs to be processed and validated as a Command. This is my attempt:

  defp parse!(_, ["quit" | _]) do {:ok, :quit} end
  defp parse!(_, ["shutdown" | _]) do {:ok, :shutdown} end
  defp parse!(_, ["echo" | opts]) do {:ok, {:echo, Enum.join(opts, " ")}} end
  defp parse!(_, cmd) when cmd == [] do {:ok, {:echo, ""}} end
  defp parse!(_state, [cmd | opts]) do
    module_name = Macro.camelize(cmd)
    try do
      module = String.to_existing_atom("Elixir.Commands.#{module_name}")
      {:ok, {module, opts}}
    rescue
      _ -> {:ok, {:echo, "#{cmd}: UNKNOWN_COMMAND"}}
    end
  end

It works, but I think it’s very brittle. So I’m trying meta-magic but I’m not sure I’m going in the right/sensible/idiomatic direction:

defmacro __using__(_opts) do
    quote do
      @behaviour Command

      @on_load :register_command

      def register_command() do
        Command.register_command(__MODULE__)
      end
    end
  end

This essentially allows me to register a command implementation, when it’s loaded. Again, it works, but I’m not sure it’s the right way.

If you can advise or point me in the right direction, I’d be very grateful :smiley: Elixir is a very interesting language and I’d love to continue working with it.

1 Like
2

I implementated a concept like this with tcp but I took another approach (without the polymerphism) .

By simply using the pattern matching , I have a module Message where there is a function encode , decode, and process.

The encode takes a message (struct) and serializes it by prepending a message which is then used to pattern match on the decode function.
And then the process take message struct as parameter to perform the task for the command.

Example:

defmodule Message do

  def encode(%MyMessage{}) do 
     <<0::8>>
  end

  def decode(<<0::8>>), do: %MyMessage{}

   def process(%MyMessage{}) do
       # Doing stuff
    end
end

And on the tcp handler:

data_to_send = data_received
|> Message.decode()
|> Message.process()
|> Message.encode() # If you want to send a response

If it can help you :wink:

3

For the level of dynamicity you seek, the on_load idea is correct.

For idiomacity:. I recommend not doing shenanigans with camelize and string.to_atom. instead, I recommend registering your modules by updating an application env value (this is backed by an ets table, so it is blazing fast). The keys should be stringified final term of Module.split and the value should be the module itself.

Nitpicky: parse should not be parse!

3 Likes
4

I’m not sure I understand this well - in my use case, I may have hundreds of commands of varying complexity, how would you organise this? Everything in the Message module? This cannot work for me :slight_smile:

5

Thanks! Essentially with the on_load, I’m not using string.to_atom anymore. I’m actually using an Agent as a key-value store (string → module as you also suggested). Eventually I’ll need a bit of a smarter key-value store (a treemap for example, if I want to support partial matching of command names, eg “rem” instead of “removefile”).

Nitpicky: parse should not be parse!

Can you explain why? In the “dynamic dispatch” tutorials they used it so I used it too, but it isn’t clear to me what it is or when it should be used instead…

6

In my use case I have few ten of messages. Each message has it own struct but the serialization, deserialization is perform on the message module as its define the message ID for a given command.

7

Use application.put_env and application.get_env instead of an agent, it doesn’t need to be supervised, you don’t have to worry about it going down, etc.

Functions that end in ! by convention signify that they are a raising equivalent of a function that emits ok/error tuples

1 Like
8

Looks like I was subtly wrong about ! convention:

https://hexdocs.pm/elixir/1.12/naming-conventions.html

But I would also say “don’t put a ! just because something can error”; I would say non-bang functions can raise on “programmer fault” (something analogous to :badarg); but should not raise on “user fault”.

1 Like
9

Consider the simplest thing that could work: listing the mapping from command to handler atom explicitly.

  @handlers %{
    "foo" => Commands.Foo,
    "bar" => Commands.Bar,
    # etc
  }
  defp parse!(_state, [cmd | opts]) do
    case Map.fetch(@handlers, cmd) do
      {:ok, mod} -> {:ok, {mod, opts}}
      :error -> {:ok, {:echo, "#{cmd}: UNKNOWN_COMMAND"}}
    end
  end

This approach also has logical extension points for useful things:

  • broadening the possible keys of the map to things like Regexes would allow for “partial match” commands
  • broadening the possible values of the map to {module, baked_in_opts} lets one “command module” serve multiple external commands

One downside is that the command → module mapping can get quite long; consider extracting parts of it to functions and combining them at compile-time to reduce clutter.

Worth looking into persistent_term for storing the map - an Agent still forces every access through a single thread.

2 Likes
10

I’m having trouble with the set_env, as it doesn’t have an update method so I have race conditions (modules are loaded in parallel with on_load executions, it seems; maybe I can find some way to have on_load block the whole loading until it’s done).

So I’m thinking about alternatives. Ets tables etc, would work. But your idea is interesting @al2o3cr - an “attribute map” of the module? I need to try this :smiley:

Essentially I still prefer to keep commands as separate files, and to register them automatically on load; it just seems cleaner (also works for hot code loading I think, as then the module is re-loaded and overwrites its entry in the command map). But your suggestion might fit nicely in my architecture.

11

Ah… nothing. It can’t be updated.

12

I’m trying with ETS tables. There’s something I don’t get… I create the table in the Command module, as soon as it’s loaded:

  @on_load :init
  def init() do
    res = :ets.new(:commands, [
      :named_table,
      :ordered_set,
      :public,
      {:read_concurrency, true},
      {:write_concurrency, true}
    ])
    whereis = :ets.whereis(:commands)
    content = :ets.tab2list(:commands)
    Logger.debug("table created: #{inspect(res)} #{inspect(whereis)} #{inspect(content)}")
  end

This works fine:

21:45:58.988 [debug] table created: :commands #Reference<0.2865878229.9043973.175739> []

The __using__ macro is like this now, with a Process.sleep to make sure the ETS table is setup at that point:

defmacro __using__(opts) do
    quote do
      @behaviour Command

      @on_load :register_command

      def register_command() do
        Process.sleep(5000)
        Command.register_command(__MODULE__, unquote(opts))
      end

      def parse_args(_cmd_args), do: {:ok, nil}
      defoverridable [parse_args: 1]
    end
  end

And then in the register_command, I try to insert:

  def register_command(module, cmd_opts) do
    Logger.debug("register_command: #{module} #{inspect(cmd_opts)}")
    cmd_name = Keyword.fetch!(cmd_opts, :name)
    :ets.insert(:commands, {cmd_name, module})
  end

But this, after the 5 seconds, crashes:

21:53:54.865 [warn]  The on_load function for module Elixir.Commands.Info returned:
{:badarg,
 [
   {:ets, :insert, [:commands, {"info", Commands.Info}],
    [error_info: %{cause: :id, module: :erl_stdlib_errors}]},
   {Command, :register_command, 2, [file: 'lib/command.ex', line: 60]},
   {:code_server, :"-handle_on_load/5-fun-0-", 1,
    [file: 'code_server.erl', ...]}
 ]}

21:53:54.858 [error] Process #PID<0.211.0> raised an exception
** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 3.15.2) :ets.insert(:commands, {"info", Commands.Info})
    lib/command.ex:60: Command.register_command/2
    (kernel 8.0.2) code_server.erl:1317: anonymous fn/1 in :code_server.handle_on_load/5

How does the table name, :commands, not refer to an existing ETS table? I don’t get. Is this because it’s happening at compile time and when the compilation finishes, the ETS table dies?

13

I had discounted this in the beginning, but… it’s brilliant. It works across compilation because the VM will not be stopped. It’s pretty much exactly my use case and it’s of a crazy simplicity.

Too bad it doesn’t support an update method but I don’t strictly need it, I’ll keep it flat: {:command, cmd_name} as key to avoid clashes, and the module as value. Brilliant.

14

Now, final question on the subject (thanks all for your help, by the way): why do I need to remove the _build directory to trigger the @on_load hook? Shouldn’t that trigger when a module is loaded, and not when compiled?