How to handle rust crate features in rustler libraries?

Questions / Help
1

I was curious if rustler can support dynamic crate features and I came up with something like this:

  • config.exs, or Mix.install:
config :ex_text_splitter,
  features: ["markdown", "tiktoken-rs"]
  • Native module
 @features Application.compile_env(:ex_text_splitter, :features, []) |> List.wrap()
  def text_splitter(_arg1, _arg2), do: err()

  if "tiktoken-rs" in @features do
    def tokenizer_text_splitter(_arg1, _arg2), do: err()
  end

  if "markdown" in @features do
    def markdown_splitter(_arg1, _arg2), do: err()
  end
  • But then the hacky part is in rust:
#[cfg(all(feature = "tiktoken-rs", not(feature = "markdown")))]
rustler::init!(
    "Elixir.ExTextSplitter.Native",
    [text_splitter, tokenizer_text_splitter]
);
#[cfg(all(not(feature = "tiktoken-rs"), feature = "markdown"))]
rustler::init!(
    "Elixir.ExTextSplitter.Native",
    [markdown_splitter, text_splitter]
);
#[cfg(all(not(feature = "tiktoken-rs"), not(feature = "markdown")))]
rustler::init!("Elixir.ExTextSplitter.Native", [text_splitter]);

I tried to use something like

match (cfg!(markdown), cfg!(feature = "tiktoken-rs")) {
(false, false) => [functions_to_export],
...
}

But the compiler complains because rustler::init! is a macro and it looks for ‘[’. Do you have any suggestions how this could be refactored?

2 Likes
2

If we merge this Discover NIFs at startup by filmor · Pull Request #613 · rusterlium/rustler · GitHub, you should be able to just #[cfg...] the respective NIFs.

1 Like