Load all modules implementing a behaviour in escript

Questions / Help
, ,
1

Hi, I’m trying to see if there’s a way to find all the modules that implement a behaviour, but from an escript.
This is one way I tried extracting those:

  defp modules_implementing_behaviour(behaviour) do
    for {module, _} <- :code.all_loaded(),
        behaviour in (module.module_info(:attributes)
                       |> Keyword.get_values(:behaviour)
                       |> List.flatten()) do
      module
    end
  end

Another attempt was this (also required adding :mix to :external_applications in mix.exs):

  # copied from https://github.com/findmypast/behaviour-introspection
  defp modules_implementing_behaviour(behaviour) do
    # Ensure the current projects code path is loaded
    Mix.Task.run("loadpaths", [])

    # Fetch all .beam files
    Path.wildcard(Path.join([Mix.Project.build_path(), "**/ebin/**/*.beam"]))
    # Parse the BEAM for behaviour implementations
    |> Stream.map(fn path ->
      {:ok, {mod, chunks}} = :beam_lib.chunks('#{path}', [:attributes])
      {mod, get_in(chunks, [:attributes, :behaviour])}
    end)
    # Filter out behaviours we don't care about and duplicates
    |> Stream.filter(fn {_mod, all_behaviours} ->
      is_list(all_behaviours) && behaviour in all_behaviours
    end)
    |> Stream.map(fn {module, _} -> module end)
    |> Enum.uniq()
  end

None of those actually work like that. The first one did kind of work if I did Code.ensure_loaded?(AModuleThatImplementsTargetBehaviour), but it defeats the purpose since I’d need to manually list all the related modules.

The problem I’m trying to solve is a project that knows how to perform a data sync between two databases. It has a number of sync scenarios to choose from. The whole idea is to create an escript that accepts a param that determines which data sync scenario we want, starts the application, finds that scenario and executes it. Not even sure that the escript is the correct way to do it. Thought of releases but can’t find a nice way to wrap it all up as an executable. Docker is something that would work but I was hoping to find a way without that. I can even make it a service that exposes some API and can be triggered via http, but that seems like an overkill for something so simple.

Anyway, thank you for your time :slight_smile:

1 Like
2

Update, a slight modification, without using Mix is marginally better:

  defp available_modules_with_behaviour(behaviour) do
    # compiled from `Mix.Task.run("loadpaths", [])`
    build_path = Path.expand("_build/prod")
    app_path = Path.join([build_path, "lib", Atom.to_string(:external_data_sync)])
    Path.join(app_path, "ebin")

    # Fetch all .beam files
    Path.wildcard(Path.join([Path.expand("_build/prod"), "**/ebin/**/*.beam"]))
    # the rest is the same
    ...
  end

This works, but in reality it’s only because lib is right where the code expects it. So it looks like the related lib module isn’t included in the escript.

I tried several ways to include all code from lib in that escript related module, no luck. Not even :ok = Application.ensure_started(:app) seems to be working. Even just aliasing those modules doesn’t work, they need to be explicitly used. This seems to work after all:

  @scenarios [
    SyncData1
    SyncData2
    SyncData3
  ]

  # and before requesting modules that implement `Scenario` behaviour
  Enum.each(@scenarios, &Code.ensure_loaded?(&1))

  # now finding those modules works
  all_scenarios = available_modules_with_behaviour(Scenario)

So I guess this is all about how to actually include the main app into escript. If it’s even possible…

3

Since dev mode starts the BEAM in interactive mode, my first stab at the problem would be to make sure all your app’s modules are loaded:

prefix = "YourApp.YourNamespace"

:code.all_available()
|> Enum.filter(fn {name, _, loaded} -> !loaded && String.starts_with?(to_string(name), "Elixir." <> prefix) end)
|> Enum.map(fn {name, _, _} -> name |> List.to_atom() end)
|> :code.ensure_modules_loaded()

…and then retry with your coding snippets from above.

(:code.all_available() docs for reference.)

This is not a bulletproof solution since f.ex. Phoenix projects with Postgres enums can produce modules defined in the global namespace if one is not careful (i.e. not belonging to your project’s namespace) but I’d think it’s good enough as a start.

NOTE 1: You’ll need OTP 23 for :code.all_available().

NOTE 2: Maybe this thread can help as well: Run code for each module implementing a Behaviour

NOTE 3: Maybe this answer in a thread can help as well? How to detect if a module implemented genserver callbacks

4

This got interesting to me because I’ve struggled with it in the past so I made a more complete example. I have a few meduim-sized Elixir projects whose source customers from 2018 and earlier have allowed me to keep. I tested the below module on them and it works quite fine in iex at least.

Curious to find out if it works for you. I realize your case is a bit different but, again, I feel this code is a good first step. Would ditching the escript idea for a Mix task work?

defmodule Behaviours do
  def modules_belonging_to_namespace(namespace)
  when is_binary(namespace) do
    :code.all_available()
    |> Enum.filter(fn {name, _file, _loaded} ->
      String.starts_with?(to_string(name), "Elixir." <> namespace)
    end)
    |> Enum.map(fn {name, _file, _loaded} ->
      name |> List.to_atom()
    end)
  end

  def ensure_all_modules_loaded(namespace)
  when is_binary(namespace) do
    modules_belonging_to_namespace(namespace)
    |> :code.ensure_modules_loaded()
  end

  def behaviours(mod)
  when is_atom(mod) do
    # Convert resulting list to [] if list of behaviours is nil
    mod.module_info
    |> get_in([:attributes, :behaviour])
    |> List.wrap()
  end

  def implementors(namespace, behaviour)
  when is_binary(namespace) and is_atom(behaviour) do
    ensure_all_modules_loaded(namespace)

    modules_belonging_to_namespace(namespace)
    |> Enum.filter(& behaviour in behaviours(&1))
  end
end
1 Like
5

Not sure if it is a good idea but you could abuse defprotocol and defimpl to create a “dispatch protocol” that would call the appropriate module for the desired scenario.

6

I’ve created a small project to demonstrate the problem, see https://github.com/elvanja/escript_testbed. One funny thing I noticed is that running it as a mix task also doesn’t work, unless I change one of the scenario modules. In that case, that module is compiled, and mix task is able to find it. But, if there are no changes, there’s also no result. So even in mix tasks I’m not getting the code in lib unless directly specified somehow.

1 Like
7

When one invokes the scenario modules directly in iex -S mix run it all runs nicely. However, even executing EscriptTestbed.CLI.main(["list"]) in the shell doesn’t yield any results. Need to try this out with OTP 23 yet.

8

Not sure if it is a good idea but you could abuse defprotocol and defimpl to create a “dispatch protocol” that would call the appropriate module for the desired scenario.

I feel that the problem actually lies in code in lib not being loaded, but maybe this might help. Need to try it out.

9

It should work unless you want to load code from lib/ dynamically without calling mix escript.build again.

10

I think you are sorted. :slight_smile: I again used the module I proposed above and put a note that you need OTP 23 for the PR to work.

1 Like
11

Tried this, no success:

defprotocol EscriptTestbed.ScenarioProtocol do
  @spec run(any()) :: any()
  def run(_)
end

defimpl EscriptTestbed.ScenarioProtocol, for: EscriptTestbed.Scenarios.DataSetA do
  def run(_), do: {:error, :use_scenario_directly}
end

defimpl EscriptTestbed.ScenarioProtocol, for: EscriptTestbed.Scenarios.DataSetB do
  def run(_), do: {:error, :use_scenario_directly}
end

And then use in CLI via one of:

Protocol.extract_impls(EscriptTestbed.ScenarioProtocol, :code.lib_dir(:elixir, :escript_testbed))
Protocol.extract_impls(EscriptTestbed.ScenarioProtocol, String.to_charlist(Application.app_dir(:escript_testbed)))
Protocol.extract_protocols(:code.lib_dir(:elixir, :escript_testbed))
Protocol.extract_protocols(String.to_charlist(Application.app_dir(:escript_testbed)))

All return empty list. It’s likely though that I’m not using it correctly!

12

Yes @dimitarvp, with OTP 23 and :code.all_available() (tried it with your Behaviours module idea) it seems to be working, both from mix task and as escript executable! Nice :smile:

P.S. Just saw your Load all modules implementing a behaviour in escript reply and related PR. That’s mostly what I tried locally, thanks! I’m probably gonna form it a bit differently, since I don’t need that behaviour module except in CLI, but still, the idea works. Great work man!

1 Like
13

Welp, if push comes to shove you can always try to re-implement :code.all_available() in OTP 22 and below but I wouldn’t recommend it. Those are implementation details of the BEAM and any hacks we could come up with (adding ebin/**/*.beam globs and such) are not and should not be guaranteed to work in the future.

Happy to help! :blush:

14

I sent you a pull request implementing what I meant. both list and sync commands work, (though I commented out a lot of code instead of configuring the repos, did not see the makefile before reading @dimitarvp posts)

1 Like
15

Nice, thank you! That’s https://github.com/elvanja/escript_testbed/pull/2 (for reference). And you were right, it is abusing the protocols indeed :smile: There’s even a bit simpler way with protocols, added to comments on the PR. Cool, today I learned!

2 Likes
16

That’s different because your trick is just to list all the modules that implement the protocol and dispatch a call. If you can do that, you can do it with behaviours too, no need to “resolve” them at runtime, the whole point of this topic.

1 Like