How to create some form of module registry that can be used to lookup a module by name?

Has anyone come up with a good way to create some form of module registry that can be used to lookup a module by name. My use case is to have a module that can deserialize a list of heterogeneous items. I would like the different items to be added to a registry automatically. Ultimately, it would be nice to ‘auto’ generate the Filters module below.

defmodule Filters do
  @moduledoc """
  Module that aids in dealing with filters. It acts as a registry that 
  knows about all the filter types at compile time.
  Supports serializing and deserializing a list of heterogeneous filter types.
  """

  # Serialize a specific filter, adding name to serialized value.
  # this can be better handled with `Protocols` and defimpl in __using__ macro. 
  # This is not the difficult part.
  def serialize(%Filter1{} = filter) do
    %{
        type: "filter1",
        value: Filter1.serialize(filter)
      } |> Jason.encode!()
  end

  def deserialize(s) when is_binary(s) do
     # s is a json encoded string that needs to be 
     # deserialized into one of the filter modules below

    deserialize(Jason.decode!(s))
  end

 # THIS IS THE FUNCTION THAT I WOULD LIKE TO GET AUTO GENERATED, or have
 # some kind of registry that can give me the module for some arbitrary name.
 #
 # Some means to create a specific struct given some name, in this case, "filter1"
 def deserialize(%{type: "filter1", value: value}) do
  Filter1.deserialize(value)
 end
end

defmodule Filter1 do
  # can something be done in the declaration of this module
  # to add it to the Filters module?
  def serialize(filter), do...
  def deserialize(data), do...
end

defmodule Filter2 do
...
end

You can go about this several ways. You can pattern match on the struct type.

def serialize(%type{} = data) do 
    IO.inspect(type)
    map = Map.from_struct(data)
    IO.inspect(map)
    s = struct(type, map)
    IO.inspect(s)
end

If it’s an Ecto schema, you can also get the schema type from obj.__meta__.schema

It’s the deserializing bit that I am having problems with. Given some name, what module handles the deserialization? I modified my post to try and clarify this better.

Sorry, I packed it all into one function.

def serialize(%type{} = data) do 
    IO.inspect(type)
    map = Map.from_struct(data)
    map = Map.put(map, :__type__, type)
    Jason.encode!(map)
end


def deserialize(data) do
    %{__type__: type} =  map = Jason.decode!(data, keys: :atoms!)
    struct(type, map)
end

only use the keys: :atoms! if you control the data coming in, and the keys are all known by the system at compile tim

only use the keys: :atoms! if you control the data coming in, and the keys are all known by the system at compile time

Maybe I’m wrong but I think this would benefit from Protocols — Elixir v1.16.0-rc.1

It does for serialize but not for the deserialize part. When deserializing, I don’t know what the type of struct I am deserializing into. @entone suggests adding a type to the map when serializing, then this can be used in the deserializing bit. I think this will work for me, I was just hoping to map an arbitrary name to a specific Elixir struct.

You can put the type wherever you want. You were putting it in a wrapper map, which would work fine as well. The main part is capturing the type at serialization. Doesn’t matter where you store it.

The struct function does this, you just need the fully qualified struct path. Which matching on the struct type at serialization will give you.

https://hexdocs.pm/elixir/Kernel.html#struct/2

Something like this?

defmodule Filter1 do
  @derive Jason.Encoder
  defstruct [:name]

  def serialize(%__MODULE__{} = struct) do
    Jason.encode!(struct)
  end

  def deserialize(json) do
    map = Jason.decode!(json, keys: :atoms!)
    struct(__MODULE__, map)
  end
end

defmodule Filter2 do
  @derive Jason.Encoder

  defstruct [:name]

  def serialize(%__MODULE__{} = struct) do
    Jason.encode!(struct)
  end

  def deserialize(json) do
    map = Jason.decode!(json, keys: :atoms!)
    struct(__MODULE__, map)
  end
end

defmodule Filters do
  def serialize(%{__struct__: type} = struct) do
    %{type: type, value: apply(type, :serialize, [struct])}
  end

  def deserialize(%{type: type, value: json}) do
    apply(type, :deserialize, [json])
  end
end

Other than a GenServer (Agent) or Registry I’m not sure how to do this. Ideally you want your filters to “register” themselves somewhere and for the Filters module to have a way to look them up.

You can write a function and hook it into your application startup that scans the list of modules and registers them if they are filters?

{:ok, modules_list} = :application.get_key(:myapp, :modules)

Edit: After some thinking it might be “better” to just make a lookup table in a module attribute.

defmodule Filters do
  @filters %{
    "filter1" => Filter1,
    ( ... )
  }

Having gone down this road many times (trying to build lists of modules dynamically based on properties) I’d advise against it. It can work well enough for production, but in development mode things can get very frustrating very quickly because of incremental recompilation.

My general suggestion for this is to instead have a module that lists all of these, i.e

defmodule MyFilters do
  def filters, do: [Filter1, Filter2, Filter3]
end

And then add this:

defmodule Filter do
  defmacro __using__(_) do
    quote do
       @behaviour Filter

       unless __MODULE__ in MyFilter.filters() do
          raise "Hey, don't forget to put this module into the list in MyFilter"
       end
    end
  end
end

Then in each filter, you say

use Filter

which sets up the behaviour and also validates that you are in the registry of filters. This way, when someone inevitably copies one and renames it, they get a helpful error. Its not as convenient as having a module magically detect other modules, but trust me that kind of thing has so many edge cases.

9 Likes

I like the shape of this pattern.

In the end I chose to give up on the ‘arbitrary’ naming of a serialized item and just added the name of the struct to the map being serialized. I was concerned about adding the full path of the module because if I ever refactor the code and move the module, I would no longer be able to deserialize the data.

defprotocol Serializer do
  def serialize(data)
end

defmodule MyApp.Filter do
  @callback deserialize(data :: map()) :: struct()
  @callback serialize(filter :: struct()) :: map()

  defmacro __using__(opts) do

    quote do
      @behaviour MyApp.Filter
      defimpl Serializer do
        def serialize(filter) do
          apply(@for, :serialize, [filter])
          |> Map.put(:__type__, Module.split(@for) |> List.last())
        end
      end
    end
  end

  def serialize(filter) do
    Serializer.serialize(filter)
  end

  def deserialize(%{"__type__" => type} = data) do
    mod = Module.safe_concat(__MODULE__, type)
    apply(mod, :deserialize, [data])
  end
end

defmodule MyApp.Filter.SomeFilter do
  use Filter
  defstruct [:id]

  def serialize(%__MODULE__{id: id}) do
    %{
      id: id
    }
  end

  def deserialize(%{"id" => id}) when is_binary(id) do
    %__MODULE__{
      id: id
    }
  end
end

This allows me to serialize/deserialize a any filter through the MyApp.Filter module.

filter = %MyApp.Filter.SomeFilter{id: :some_id}
data = MyApp.Filter.serialize(filter)
filter = MyApp.Filter.deserialize(data)

There are a few details omitted which are not important to the discussion at hand. This serialization/deserialization does not do the JSON encoding at this level since this is only part of the the overall structure that is getting serialized.