How to download an external binary at compile-time / application start

[tmbb]

I have written an interface to a linear solver in Elixir. To solve the linear program, I just generate an input file, call the solver (which is an external binary) on the input and parse the output file back into Elixir.

I want to be able to download a binary from a specific source (the github release page for the project) at compile time or at build time. I’d prefer to use an approch that would allow me to have the binary in place before generating an Elixir release. That would forbid downloading it at application start time.

Any suggestions?

[evadne]

My preferred strategy:

1. Always find the binary in priv
2. At startup if no file in priv then go get it
3. Allow users to seed the binary themselves

Projects that use elixir_make would just use make so the side effect is if there is already a binary where make is supposed to put it then it wont make it again

tzdata does similar stuff

[cmo]

Something like this? It could do with some polish.

defmodule Include do
  @moduledoc """
  External applications to be distributed with our application.
  """

  @callback name() :: Atom.t()
  @callback owner() :: String.t()
  @callback repo() :: String.t()
  @callback extension() :: String.t()
  @callback find_asset(asset :: map) :: boolean
  @callback after_download(source_file :: String.t(), destination_folder :: String.t()) :: :ok
  @callback cli_version_cmd() :: String.t()
  @callback parse_cli_version(cli_version_output :: String.t()) :: version :: String.t()
  @callback version_to_tag(version :: String.t()) :: tag :: String.t()
  @callback tag_to_version(tag :: String.t()) :: version :: String.t()

  defmacro __using__(_) do
    quote do
      @behaviour Include

      @impl true
      def after_download(source_file, destination_folder) do
        destination_file = Path.join(destination_folder, "#{__MODULE__.name()}.exe")
        File.rename(source_file, destination_file)
      end

      defoverridable after_download: 2
    end
  end

  # TODO: don't rely on hardcoded module names
  @apps [
    Include.Caddy,
    Include.Relay,
    Include.Otelcol
  ]

  def all_app_names do
    for app <- @apps, do: to_string(app.name())
  end

  @spec current_version(module) :: String.t()
  def current_version(app, path \\ nil) do
    exe =
      cond do
        path == nil ->
          ["installer", "include", "#{app.name()}.exe"] |> Path.join() |> Path.expand()

        String.ends_with?(path, ".exe") ->
          Path.expand(path)

        true ->
          Path.join(path, "#{app.name()}.exe") |> Path.expand()
      end

    args = [app.cli_version_cmd()]

    if File.exists?(exe) do
      case System.cmd(exe, args) do
        {output, 0} ->
          version = output |> String.trim() |> app.parse_cli_version()
          {:ok, version}

        {_output, _exit_code} = error ->
          {:error, error}
      end
    else
      {:error, %File.Error{path: exe, reason: :enoent}}
    end
  end

  def latest_version(app) when app in @apps do
    with {:ok, release} <- get_release_from_github(app, "latest") do
      tag = Map.fetch!(release, "tag_name")
      {:ok, app.tag_to_version(tag)}
    end
  end

  def download_version(app, version) when app in @apps and is_binary(version) do
    with {:ok, release} <- get_release_from_github(app, version) do
      asset =
        release
        |> Map.fetch!("assets")
        |> Enum.find_value(fn asset -> if app.find_asset(asset), do: asset end)

      download_asset(app, asset)

      version =
        release
        |> Map.fetch!("tag_name")
        |> app.tag_to_version()

      {:ok, version}
    end
  end

  def get_release_from_github(app, version) do
    url = github_release_url(app, version)

    case Req.get!(url) do
      %{status: 200} = resp -> {:ok, resp.body}
      %{status: 404} = resp -> {:error, resp}
    end
  end

  # https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#get-the-latest-release
  defp github_release_url(app, "latest") when app in @apps do
    "https://api.github.com/repos/#{app.owner()}/#{app.repo()}/releases/latest"
  end

  # https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#get-a-release-by-tag-name
  defp github_release_url(app, version) when app in @apps and is_binary(version) do
    tag = app.version_to_tag(version)
    "https://api.github.com/repos/#{app.owner()}/#{app.repo()}/releases/tags/#{tag}"
  end

  defp download_asset(app, asset) when app in @apps and is_map(asset) do
    %{"browser_download_url" => browser_download_url, "content_type" => content_type} = asset

    path = download_path(app)
    destination_folder = Path.expand("installer/include")

    if File.exists?(path) and File.regular?(path) do
      app.after_download(path, destination_folder)
    else
      path |> Path.dirname() |> File.mkdir_p!()
      file_stream = File.stream!(path, 1024 * 8)

      req =
        Req.new(
          url: browser_download_url,
          headers: %{
            "accept" => content_type,
            "x-github-api-version" => "2022-11-28"
          },
          into: file_stream
        )

      with %{status: status} when status in [200, 302] <- Req.get!(req) do
        # TODO: make the path a config option
        app.after_download(path, destination_folder)
      end
    end
  end

  defp download_path(app) do
    [System.tmp_dir!(), "include", "#{app.name()}#{app.extension()}"]
    |> Path.join()
    |> Path.expand()
  end

  def find_app_by_name(app_name) do
    app_name = to_string(app_name)
    Enum.find_value(@apps, fn app -> if to_string(app.name()) == app_name, do: app end)
  end
end

defmodule Include.Caddy do
  use Include

  @impl true
  def name, do: :caddy

  @impl true
  def owner, do: "caddyserver"

  @impl true
  def repo, do: "caddy"

  @impl true
  def extension, do: ".zip"

  @impl true
  def find_asset(%{"name" => name}) do
    String.match?(name, ~r/^caddy_\d+\.\d+\.\d+_windows_amd64\.zip$/)
  end

  @impl true
  def after_download(source_file, destination_folder) do
    Mix.shell().info("extracting #{source_file} to #{destination_folder}")

    source_file
    |> String.to_charlist()
    |> :zip.extract(file_list: [~c"caddy.exe"], cwd: String.to_charlist(destination_folder))
  end

  @impl true
  def cli_version_cmd do
    "version"
  end

  @impl true
  def parse_cli_version(version) do
    version
    |> String.trim_leading("v")
    |> String.split(" ", parts: 2)
    |> Enum.at(0)
  end

  @impl true
  def version_to_tag(version) do
    "v" <> version
  end

  @impl true
  def tag_to_version(<<"v", version::binary>>) do
    version
  end
end

defmodule Mix.Tasks.Inc.Get do
  @moduledoc """
  Download included executables defined for the project.
  """
  @shortdoc "Download included executables"

  use Mix.Task

  alias Include.Shell

  @impl true
  def run(args) do
    {opts, args} = OptionParser.parse!(args, strict: [force: :boolean])

    {:ok, _} = Application.ensure_all_started(:req)

    all_app_names = Include.all_app_names()

    app_names =
      case args do
        [] ->
          all_app_names

        app_names ->
          Enum.flat_map(app_names, fn app_name ->
            if app_name in all_app_names do
              [app_name]
            else
              Shell.error("#{app_name} is not a valid include")
              []
            end
          end)
      end

    force? = Keyword.get(opts, :force) || false

    Mix.Project.get!()
    includes = Mix.Project.config() |> Keyword.fetch!(:includes)

    for {app_name, version} <- includes,
        app_name = to_string(app_name),
        app_name in app_names,
        app = Include.find_app_by_name(app_name) do
      case Include.current_version(app) do
        {:ok, current_version} ->
          different_version? = Version.compare(current_version, version) != :eq

          cond do
            different_version? ->
              Shell.info("#{app_name} #{current_version} -> #{version}, downloading")
              download_version(app, version)

            force? ->
              Shell.info("#{app_name} #{current_version} downloading")
              download_version(app, version)

            true ->
              Shell.info("#{app_name} #{version} already in destination")
          end

        {:error, %File.Error{reason: :enoent}} ->
          Shell.info("#{app_name} #{version} downloading")
          download_version(app, version)
      end
    end
  end

  defp download_version(app, version) do
    case Include.download_version(app, version) do
      {:ok, version} ->
        Shell.success("#{app.name()} #{version} downloaded")

      {:error, reason} ->
        Shell.error("#{app.name()} #{version} download error: #{inspect(reason)}")
    end
  end
end

defmodule Mix.Tasks.Inc.Outdated do
  @moduledoc """
  Check whether included executables are running the latest version.
  """
  @shortdoc "Check whether included executables are running the latest version"

  use Mix.Task

  alias Include.Shell

  @impl true
  def run(_) do
    {:ok, _} = Application.ensure_all_started(:req)

    all_app_names = Include.all_app_names()

    Mix.Project.get!()
    includes = Mix.Project.config() |> Keyword.fetch!(:includes)

    Shell.info("Checking included executables against latest version...")

    for {app_name, current_version} <- includes,
        app_name = to_string(app_name),
        app_name in all_app_names,
        app = Include.find_app_by_name(app_name) do
      case Include.latest_version(app) do
        {:ok, latest_version} ->
          case Version.compare(current_version, latest_version) do
            :eq -> Shell.success("#{app_name} #{current_version} == #{latest_version}")
            :lt -> Shell.warning("#{app_name} #{current_version} < #{latest_version}")
            :gt -> Shell.error("#{app_name} #{current_version} > #{latest_version}")
          end

        {:error, reason} ->
          Shell.error("failed to get latest version of #{app_name}: #{inspect(reason)}")
      end
    end
  end
end

[tmbb]

Well, if tzdata downloads the timezone data in the Application.start/2 callback (inside of a supervised task), then ir seems good enough for me.

Thanks!

[LostKobrakai]

tzdata has gotten a lot of critic for automatically doing requests and updating at runtime. Behaviour like that is imo best opt in.

More suitable might be looking at rustler_precompiled and cc_precompiler and how they make downloading binary artifacts happen.

[tmbb]

Thanks for your perspective! I actually don’t like tzdata’s approach for that reason, and if people also criticize that (nor just me) I’l looks into other approaches.

The source of rust_precompiled is very “meta”, with use macros inside use macros, but I guess I just have to dedicate a bit more time to ir.

Esbuild (whose source I’ve read) also downloads the binary at “runtime”, but Esbuild is used only at compile time, so those criticisms don’t apply to it.

[joelpaulkoch]

You can have a look at what I did for sqlite_vec.

I found octo_fetch for downloading GitHub releases.
Then I added a function to the list of compilers to trigger the download at compile time.

Actually I tried a few different ways to make the download happen at compile time, and I’m happy to hear what other people think is the best way.

Also, I think you can configure the octo_fetch dependency with runtime: false so it won’t get included, but then I guess the downloader module would crash at runtime if you’d ever call it.

You could also skip octo_fetch and just construct the link yourself and use req for the download.
You can check the checksum with req too.

[tmbb]

Thanks, I think that a custom compiler is the way to go! Whether one uses octo_fetch or custom code to actually fetch the file or not is a small detail, but I’ll look into what exactly octo_fetch brings here.