Load all modules implementing a behaviour in escript

Update, a slight modification, without using Mix is marginally better:

  defp available_modules_with_behaviour(behaviour) do
    # compiled from `Mix.Task.run("loadpaths", [])`
    build_path = Path.expand("_build/prod")
    app_path = Path.join([build_path, "lib", Atom.to_string(:external_data_sync)])
    Path.join(app_path, "ebin")

    # Fetch all .beam files
    Path.wildcard(Path.join([Path.expand("_build/prod"), "**/ebin/**/*.beam"]))
    # the rest is the same
    ...
  end

This works, but in reality it’s only because lib is right where the code expects it. So it looks like the related lib module isn’t included in the escript.

I tried several ways to include all code from lib in that escript related module, no luck. Not even :ok = Application.ensure_started(:app) seems to be working. Even just aliasing those modules doesn’t work, they need to be explicitly used. This seems to work after all:

  @scenarios [
    SyncData1
    SyncData2
    SyncData3
  ]

  # and before requesting modules that implement `Scenario` behaviour
  Enum.each(@scenarios, &Code.ensure_loaded?(&1))

  # now finding those modules works
  all_scenarios = available_modules_with_behaviour(Scenario)

So I guess this is all about how to actually include the main app into escript. If it’s even possible…

Since dev mode starts the BEAM in interactive mode, my first stab at the problem would be to make sure all your app’s modules are loaded:

prefix = "YourApp.YourNamespace"

:code.all_available()
|> Enum.filter(fn {name, _, loaded} -> !loaded && String.starts_with?(to_string(name), "Elixir." <> prefix) end)
|> Enum.map(fn {name, _, _} -> name |> List.to_atom() end)
|> :code.ensure_modules_loaded()

…and then retry with your coding snippets from above.

(:code.all_available() docs for reference.)

This is not a bulletproof solution since f.ex. Phoenix projects with Postgres enums can produce modules defined in the global namespace if one is not careful (i.e. not belonging to your project’s namespace) but I’d think it’s good enough as a start.

NOTE 1: You’ll need OTP 23 for :code.all_available().

NOTE 2: Maybe this thread can help as well: Run code for each module implementing a Behaviour

NOTE 3: Maybe this answer in a thread can help as well? How to detect if a module implemented genserver callbacks - #2 by NobbZ

This got interesting to me because I’ve struggled with it in the past so I made a more complete example. I have a few meduim-sized Elixir projects whose source customers from 2018 and earlier have allowed me to keep. I tested the below module on them and it works quite fine in iex at least.

Curious to find out if it works for you. I realize your case is a bit different but, again, I feel this code is a good first step. Would ditching the escript idea for a Mix task work?

defmodule Behaviours do
  def modules_belonging_to_namespace(namespace)
  when is_binary(namespace) do
    :code.all_available()
    |> Enum.filter(fn {name, _file, _loaded} ->
      String.starts_with?(to_string(name), "Elixir." <> namespace)
    end)
    |> Enum.map(fn {name, _file, _loaded} ->
      name |> List.to_atom()
    end)
  end

  def ensure_all_modules_loaded(namespace)
  when is_binary(namespace) do
    modules_belonging_to_namespace(namespace)
    |> :code.ensure_modules_loaded()
  end

  def behaviours(mod)
  when is_atom(mod) do
    # Convert resulting list to [] if list of behaviours is nil
    mod.module_info
    |> get_in([:attributes, :behaviour])
    |> List.wrap()
  end

  def implementors(namespace, behaviour)
  when is_binary(namespace) and is_atom(behaviour) do
    ensure_all_modules_loaded(namespace)

    modules_belonging_to_namespace(namespace)
    |> Enum.filter(& behaviour in behaviours(&1))
  end
end

Not sure if it is a good idea but you could abuse defprotocol and defimpl to create a “dispatch protocol” that would call the appropriate module for the desired scenario.

I’ve created a small project to demonstrate the problem, see GitHub - elvanja/escript_testbed · GitHub. One funny thing I noticed is that running it as a mix task also doesn’t work, unless I change one of the scenario modules. In that case, that module is compiled, and mix task is able to find it. But, if there are no changes, there’s also no result. So even in mix tasks I’m not getting the code in lib unless directly specified somehow.

When one invokes the scenario modules directly in iex -S mix run it all runs nicely. However, even executing EscriptTestbed.CLI.main(["list"]) in the shell doesn’t yield any results. Need to try this out with OTP 23 yet.

Not sure if it is a good idea but you could abuse defprotocol and defimpl to create a “dispatch protocol” that would call the appropriate module for the desired scenario.

I feel that the problem actually lies in code in lib not being loaded, but maybe this might help. Need to try it out.

It should work unless you want to load code from lib/ dynamically without calling mix escript.build again.

I think you are sorted. I again used the module I proposed above and put a note that you need OTP 23 for the PR to work.

https://github.com/elvanja/escript_testbed/pull/1

Tried this, no success:

defprotocol EscriptTestbed.ScenarioProtocol do
  @spec run(any()) :: any()
  def run(_)
end

defimpl EscriptTestbed.ScenarioProtocol, for: EscriptTestbed.Scenarios.DataSetA do
  def run(_), do: {:error, :use_scenario_directly}
end

defimpl EscriptTestbed.ScenarioProtocol, for: EscriptTestbed.Scenarios.DataSetB do
  def run(_), do: {:error, :use_scenario_directly}
end

And then use in CLI via one of:

Protocol.extract_impls(EscriptTestbed.ScenarioProtocol, :code.lib_dir(:elixir, :escript_testbed))
Protocol.extract_impls(EscriptTestbed.ScenarioProtocol, String.to_charlist(Application.app_dir(:escript_testbed)))
Protocol.extract_protocols(:code.lib_dir(:elixir, :escript_testbed))
Protocol.extract_protocols(String.to_charlist(Application.app_dir(:escript_testbed)))

All return empty list. It’s likely though that I’m not using it correctly!