elvanja

Load all modules implementing a behaviour in escript

Hi, I’m trying to see if there’s a way to find all the modules that implement a behaviour, but from an escript.
This is one way I tried extracting those:

  defp modules_implementing_behaviour(behaviour) do
    for {module, _} <- :code.all_loaded(),
        behaviour in (module.module_info(:attributes)
                       |> Keyword.get_values(:behaviour)
                       |> List.flatten()) do
      module
    end
  end

Another attempt was this (also required adding :mix to :external_applications in mix.exs):

  # copied from https://github.com/findmypast/behaviour-introspection
  defp modules_implementing_behaviour(behaviour) do
    # Ensure the current projects code path is loaded
    Mix.Task.run("loadpaths", [])

    # Fetch all .beam files
    Path.wildcard(Path.join([Mix.Project.build_path(), "**/ebin/**/*.beam"]))
    # Parse the BEAM for behaviour implementations
    |> Stream.map(fn path ->
      {:ok, {mod, chunks}} = :beam_lib.chunks('#{path}', [:attributes])
      {mod, get_in(chunks, [:attributes, :behaviour])}
    end)
    # Filter out behaviours we don't care about and duplicates
    |> Stream.filter(fn {_mod, all_behaviours} ->
      is_list(all_behaviours) && behaviour in all_behaviours
    end)
    |> Stream.map(fn {module, _} -> module end)
    |> Enum.uniq()
  end

None of those actually work like that. The first one did kind of work if I did Code.ensure_loaded?(AModuleThatImplementsTargetBehaviour), but it defeats the purpose since I’d need to manually list all the related modules.

The problem I’m trying to solve is a project that knows how to perform a data sync between two databases. It has a number of sync scenarios to choose from. The whole idea is to create an escript that accepts a param that determines which data sync scenario we want, starts the application, finds that scenario and executes it. Not even sure that the escript is the correct way to do it. Thought of releases but can’t find a nice way to wrap it all up as an executable. Docker is something that would work but I was hoping to find a way without that. I can even make it a service that exposes some API and can be triggered via http, but that seems like an overkill for something so simple.

Anyway, thank you for your time

15 comments

#deployment #escript #introspection

1 2550 15

2020-07-27 07:25:49 UTC

Marked As Solved

dimitarvp

This got interesting to me because I’ve struggled with it in the past so I made a more complete example. I have a few meduim-sized Elixir projects whose source customers from 2018 and earlier have allowed me to keep. I tested the below module on them and it works quite fine in iex at least.

Curious to find out if it works for you. I realize your case is a bit different but, again, I feel this code is a good first step. Would ditching the escript idea for a Mix task work?

defmodule Behaviours do
  def modules_belonging_to_namespace(namespace)
  when is_binary(namespace) do
    :code.all_available()
    |> Enum.filter(fn {name, _file, _loaded} ->
      String.starts_with?(to_string(name), "Elixir." <> namespace)
    end)
    |> Enum.map(fn {name, _file, _loaded} ->
      name |> List.to_atom()
    end)
  end

  def ensure_all_modules_loaded(namespace)
  when is_binary(namespace) do
    modules_belonging_to_namespace(namespace)
    |> :code.ensure_modules_loaded()
  end

  def behaviours(mod)
  when is_atom(mod) do
    # Convert resulting list to [] if list of behaviours is nil
    mod.module_info
    |> get_in([:attributes, :behaviour])
    |> List.wrap()
  end

  def implementors(namespace, behaviour)
  when is_binary(namespace) and is_atom(behaviour) do
    ensure_all_modules_loaded(namespace)

    modules_belonging_to_namespace(namespace)
    |> Enum.filter(& behaviour in behaviours(&1))
  end
end

Post #4

Also Liked

elvanja

Nice, thank you! That’s protocol demo by lud · Pull Request #2 · elvanja/escript_testbed · GitHub (for reference). And you were right, it is abusing the protocols indeed There’s even a bit simpler way with protocols, added to comments on the PR. Cool, today I learned!

Post #15

elvanja

I’ve created a small project to demonstrate the problem, see GitHub - elvanja/escript_testbed · GitHub. One funny thing I noticed is that running it as a mix task also doesn’t work, unless I change one of the scenario modules. In that case, that module is compiled, and mix task is able to find it. But, if there are no changes, there’s also no result. So even in mix tasks I’m not getting the code in lib unless directly specified somehow.

Post #6

dimitarvp

I think you are sorted. I again used the module I proposed above and put a note that you need OTP 23 for the PR to work.

https://github.com/elvanja/escript_testbed/pull/1

Post #10

Last Post!

lud

That’s different because your trick is just to list all the modules that implement the protocol and dispatch a call. If you can do that, you can do it with behaviours too, no need to “resolve” them at runtime, the whole point of this topic.

Post #16