georgeguimaraes

georgeguimaraes

Hi all,

I want to show you Transmuter, a library that runs Hugging Face ONNX models on any Nx backend. (Nx is Elixir’s numerical library. ONNX is a open format to describe neural networks.)

Why

Bumblebee is great, but each architecture needs a port to Axon by hand, and Hugging Face gets new architectures and models weekly. Ortex runs ONNX models, but outside of Nx: you cannot use it in defn, and it has no Apple silicon GPU path.

Transmuter takes another approach. It implements the ONNX operators in Nx, one time each. Then it runs the graph of the exported model, operator by operator. So a new architecture needs no new Elixir code. It needs only operators that Transmuter already has, and Transmuter has all 203 operators of the ONNX standard domain, plus the ONNX Runtime contrib operators that Hugging Face exports use.

The result is plain Nx:

  • It runs on EMLX (Apple silicon GPU) and on EXLA (CUDA or CPU).
  • It works with Nx.Serving.
  • Nx.Defn.grad can differentiate it.
  • Shapes are set when a function compiles, not when the model loads. So decoders with a KV cache and graphs with If branches work.

Models that Bumblebee can’t run yet

Bumblebee does not have these architectures today, and Transmuter can run them (I validated them):

  • Vision: SAM, SAM 2 and the SAM 3 tracker, Depth Anything, Grounding DINO, OWL-ViT and OWLv2, Florence-2, DETR, YOLOS, RT-DETR and RF-DETR, Segformer, ORMBG, Swin2SR, SigLIP and SigLIP2, TrOCR and ViT-GPT2.
  • Chats about images: SmolVLM, Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Ministral 3 and LLaVA.
  • Text: GLiNER, Marian, Apertus, Granite 4.0-H (with Mamba-2 layers) and LFM2 (mixture of experts).
  • Audio: wav2vec2, AST, CLAP, VITS, SpeechT5 and Kokoro.

Example

opts = [backend: EMLX.Backend, defn_options: [compiler: EMLX]]

serving = Transmuter.Text.text_embedding({:hf, "Xenova/all-MiniLM-L6-v2"}, opts)
Nx.Serving.run(serving, "The cat sat on the mat.")
#=> %{embedding: #Nx.Tensor<f32[384] ...>}

serving = Transmuter.Text.generation({:hf, "onnx-community/Qwen3-0.6B-ONNX"}, opts)
Nx.Serving.run(serving, %{messages: [%{role: "user", content: "What is the capital of France?"}]})

What you get

The servings follow Bumblebee where Bumblebee has the task:

  • Text: embeddings, classification, reranking, zero-shot NER with GLiNER, NLI zero-shot, token classification, fill mask, question answering, generation (SmolLM2, SmolLM3, Qwen3, Gemma 3) and text to text (T5, Marian, NLLB, M2M100).
  • Vision: classification, CLIP and SigLIP embeddings, captions (TrOCR, Florence-2), chats about images (SmolVLM, Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Ministral 3, Gemma 3, LLaVA), object detection (DETR, RT-DETR, RF-DETR), zero-shot detection (OWLv2, Grounding DINO), depth (Depth Anything), segmentation, background removal, SAM, SAM 2 and the SAM 3 tracker, and super resolution.
  • Audio: Whisper with word timestamps, audio classification, CLAP zero-shot, and text to speech (VITS, SpeechT5).

Generation has greedy selection, sampling, beam search, logits processors, streaming and token log-probabilities.

Correctness

The tests compare the values with Python transformers pipelines and with ONNX Runtime on the same files. For the top Hugging Face export of each task, 75 of 77 repositories match. The README has the full coverage table, and it explains each model that does not match value by value.

Feedback

When a model has no ONNX file, you can export it one time with Optimum. If a model fails or gives other values than Python, please open an issue. That is the most useful feedback for me now.


Showing Posts 1 to 2

georgeguimaraes

georgeguimaraes OP

btw, I have access to a RTX 3060 with 12GB of VRAM and my Macbook Pro with 16GB of RAM, so I’m a bit limited on which models I can run. If you can try bigger models, please do :slight_smile:

tubedude

tubedude

Nice! I tried in my Jev/Laya implementation and here is a summary:

It was good! But there is a 30 second compilation in
for each padding that I have to figure out how to fix.

:sweat_smile: but I can’t help you with larger models since all I have is a RTX 3060 Ti (8 GB)

— All posts loaded —

Where Next? Top

Trending in Announcing Top

mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
danschultzer
I’ve moved the strategies out of PowAssent and created a multi-provider framework for Elixir. This way the strategies can be used in any ...
New
gtcode
Aloha gang, There’s a new class of AI model out from TypeSafe. Jev is their flagship model. You can learn about it here: ...
New
danschultzer
I needed something like premailer for rails, and after searching, I decided to make an easy-to-use library to handle inline styles and co...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
webofbits
With AI doing more of the implementation work, I’ve been wondering how much coding I should deliberately keep doing myself. My main conc...
#ai
New
AstonJ
With advancements in AI gathering pace and set to increasingly accelerate, perhaps it’s worth having an annual thread on the topic - feel...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews