Omni - a universal Elixir client for LLM APIs

News Announcing
, , ,
1

This thread is the home for updates and discussion across the Omni family of packages. What started as a single library for calling LLM APIs has grown into three packages that cover the full stack of building with LLMs in Elixir:

  • omni - Universal Elixir client for LLM APIs. Streaming text generation, tool use, an structured output.
  • omni_agent - Stateful LLM agents for Elixir - persistent, branching conversations, tool approval, and multi-session management.
  • omni_tools - Ready-to-use tools for Omni-powered agents - filesystem, shell, REPL, web fetch, and web search.

Original post follows.

Hey everyone - I’ve been building with Elixir on and off for over 8 years, but somehow have never posted on the actual Elixir forum. Time to fix that…

Also, I’d love to share with you Omni - a library for working with LLM APIs across multiple providers through a unified interface. Anthropic, OpenAI, Google Gemini, Ollama, OpenRouter, and OpenCode Zen are supported out of the box.

# Resolve model
{:ok, model} = Omni.get_model(:anthropic, "claude-sonnet-4-6")

# Simple text generation
{:ok, response} = Omni.generate_text(model, "Hello!")

# Stream with composable callbacks
{:ok, stream} = Omni.stream_text(model, "Tell me a story")

{:ok, response} =
  stream
  |> Omni.StreamingResponse.on(:text_delta, &IO.write(&1.delta))
  |> Omni.StreamingResponse.complete()

Tool use and structured outputs are supported. Pass tools in the context and Omni handles the execution loop automatically - calling the model, executing tool handlers, feeding results back, and repeating until the model is done. Structured output uses JSON Schema constraints with validation:

# Tool use - Omni manages the tool execution loop
{:ok, response} = Omni.generate_text(
  model,
  Omni.context(
    messages: [Omni.message(role: :user, content: "What's the weather in London?")],
    tools: [weather_tool]
  )
)

# Structured output
alias Omni.Schema
{:ok, response} = Omni.generate_text(
  model,
  "Extract the contact details: Reach me at jane@example.com or call 01234 567890",
  output: Schema.object(%{
    email: Schema.string(description: "Email address"),
    phone: Schema.string(description: "Phone number")
  }, required: [:email, :phone])
)

Omni also offers a lightweight take on agents. Omni.Agent is a GenServer that manages its own conversation context and tool execution, and communicates with callers via standard process messages. You control behaviour through lifecycle callbacks. It’s a building block, not a framework - what you build on top (planning, memory, multi-agent orchestration) is your concern.

I know req_llm covers similar ground, which - slightly annoyingly - I didn’t realise existed until I was 90% of the way done with Omni :man_facepalming:t2:. On the surface they have quite similar APIs, and both use Req, but how they handle implementing providers is a little different. Omni separates providers (the endpoint, configuration and auth) and dialects (wire format translation). The dialect does the heavy lifting, and as most providers share a dialect, adding a new provider is typically a small, mostly-declarative module. Everything is streaming-first - generate_text is built on top of stream_text, so there’s one code path through each dialect.

Anyway, please check it out. Let me know if you have any questions.

13 Likes
2

How does this compare to (or where does it sit in relation to) Langchain (Elixir)?

1 Like
3

They’re both text generation focused - so fundamentally do the same thing. Just a different style and take.

Langchain has a few things Omni does not: multimodal, RAG text splitting, EEx prompt templates - and probably some other stuff. Omni is light-weight, only has 2 dependencies.

The main difference for users is the surface API. The mental model for Omni: is build a request, get a stream, consume it. For Langchain it’s build a chain, add some messages, run the chain. Omni’s style is functional, data oriented; Langchain’s is stateful structs, callbacks, framework-y.

Internally the big difference is how Omni splits Providers and Dialects into two things, which should make it relatively painless to add more providers over time. Langchain has one big fat module per provider which I think looks hard to maintain. In theory Omni could sit underneath Langchain and be that provider translation layer.

1 Like
4

Thanks a lot @aaronrussell for the explanation and for the library. Could you please compare/contrast Omni with ReqLLM?

5

Regarding req_llm, they are honestly very similar. Both libraries attempt to solve the same problem and approach it in very similar ways. Both libraries are clearly influenced by the Vercel AI SDK, but in Omni I’ve taken a lot of inspiration from the Pi codebase, which I think steers it in a slightly leaner direction.

  • Both have a almost identical top-level API: generate_text(model, context, inference_opts) and stream_text(model, context, inference_opts)
  • Both attempt to establish a canonical data model that works across all LLM provider APIs. There are some minor differences in the shape of those data models, but essentially they’re doing the same thing.
  • Both source model data from the models.dev API.
  • Both can generate text, structured objects, can stream responses, track usage tokens and costs.
  • Both use Req under the hood.

Some gaps (some of these will narrow over time):

  • req_llm supports 45 providers with ~665 models - omni supports 6 providers with ~300 models
  • req_llm supports image generation, omni does not
  • Omni takes a slightly pragmatic view with more obscure inference options like top_p, logprobs etc and doesn’t currently attempt to support every option for every provider - req_llm appears to be have a bit more complete coverage
  • Omni provides a simple `Omni.Agent` genserver as a building block for agents - req_llm does not have anything like this (but is part of the wider Jido ecosystem)

Lower level differences:

  • Omni splits a provider into two behaviours - the Provider and the Dialect (the wire format). This should make adding new providers much quicker and easier to maintain, as most providers in the wild share dialects. This also makes it easy for users to create their own providers.
  • Omni is streaming first - so even generate_text/3 is a streaming request that is accumulated in one call. This means a dialect only needs to care about streaming requests - resulting in simpler implementations.
1 Like
6

Thanks for your reply! There may be a couple other distinctions. With ReqLLM, Ollama integration is not an out-of-the-box option, but Omni docs show Ollama support. Also: I believe ReqLLM is integrated with Jido and Ash.

I’ll give Omni a try with Ollama!

7

Yep there is an Ollama provider. A little config is needed:

# Ollama isn't a default provider, so load it in the config
config :omni, :providers, [:ollama]

# You need to configure your installed models
config :omni, Omni.Providers.Ollama, models: ["mistral:7b", "qwen3.5:4b"]

Oh, and tool calling, reasoning etc is model dependent. That little 4b qwen model is pretty good for testing.

1 Like
8

Omni updates this week…

Omni v1.2.0

  • Extracted Omni.Agent and associated modules into it’s own package. omni lives as a stateless LLM API layer for any LLM provider. omni_agent becomes it’s own thing (see below).
  • Model store updated, including latest GPT 5.4 mini and nano models.
  • Minor under the hood bits and bobs.

Omni Agent v0.1.0

  • Extracted from above, now it’s own package for creating stateful, multi-turn GenServer-powered agent processes.
  • API and lifecycle simplified, documentation cleared up.
  • Consider this a more experimental package - expect things to change and break.

Links

Omni - GitHub | Docs
Omni Agent - GitHub | Docs

1 Like
9

Thanks for releasing Omni. I like the approach, especially the low number of deps.

Question: With Omni, is it possible to receive the model’s tool selections without having Omni execute the tool itself? I wasn’t able to tell in the docs; they focus on the auto-execution loop, which makes sense for many cases, but I would like to have full control over the tool execution and resulting context additions.

10

Yep two ways to do that. First and simplest is pass max_steps: 1, eg:

Omni.stream_text(model, context, max_steps: 1)

Also tools themselves can be schema-only - they don’t need a function handler attached. So if any of your tools don’t have a function handler then it will just stop there with a stop_reason: :tool_use and then it’s up to you to handle the rest and feed the results back.

1 Like
11

I’ve just released new versions of omni and omni_agent… in fact, there’s a been a few updates since my last post so rolling it all into one.

Omni v1.3.2

  • New providers supported: Groq, Moonshot AI (Kimi), Z.ai and Alibaba now have tested provider implementations.
  • Updated model catalogue: over the last few weeks we’ve had Claude Opus 4.7, GPT-5.5 (and Pro), Kimi K2.6, Deepseek v4 (and Pro), Qwen 3.6 Plus - and more.
  • Improved live testing suite across all providers.
  • New Omni.Codec module for serialisation of Omni structs to and from JSON-safe maps.
  • New xhigh thinking level option.
  • And more - see changelog.

Omni Agent v0.3.0

  • New Omni.Session - a process that wraps an Omni.Agent process, providing persistence, and conversation branching and navigation (edits and regenerations).
  • New Omni.Session.Store behaviour and default Omni.Session.Store.FileSystem adapter.
  • New Omni.Session.Manager supervisor for managing multiple concurrent sessions.
  • New Omni.Session.Tree for holding branching conversations.
  • A Session mirrors all of Agent’s streaming events, plus a few more (:navigate and :tree events).
  • Agent has new :message, :step, and :state events, while the :stop and :continue events have been replaced by the :turn event.
  • Quite a lot more fixes, tweaks, etc - see changelog.

Links

Omni - GitHub | Docs
Omni Agent - GitHub | Docs

12

Hello,

I’m evaluating this library and I have two questions:

  • is it possible to integrate JSON schema validation with a custom validator (we want to use JSV to automatically validate and cast output to structs, using $ref schemas for various entities)
  • is it possible to enforce not using environment variables like OPENAI_API_KEY that may be defined.

Thank you

13

Hi, thanks for your interest.

  • JSON schema validation - as it stands right now, we validate against the JSON schema using an admittedly cobbled together solution. It does a decent enough job for basic shapes, but won’t cast to structs or resolve refs. That said, there’s room for improvement here. I previously looked at a few JSON-schema packages to see if we could just offload the problem to a dependency, but none that I looked at fitted they way I wanted. But, I didn’t look at JSV, and honestly it looks perfect. So if it’s of interest, I’d be more than happy to bake JSV in as the way we validate against schemas. I’ll dig a bit deeper in the week, but potentially that could be done this week.

  • Env variables - the auth method resolves like this:

    1. If the caller provides an API key, use that
    2. If the provider has an API key in the app config, use that
    3. If the provider environment variable is set, use that
    4. Fail

    So, in theory if your code provides keys in 1 or 2, then it will never drill down to 3. Is that good enough or do you need a way to explicitly remove 3 from the flow?

1 Like
14

Congrats OP!!! ReqLLM author here - any time you’d like to chat about this problem space I’m happy to grab time.

Only suggestion would be to consider collaborating on LLMDB: llm_db | Hex

Nailing the model database and model capabilities turned out to a huge challenge and that DB is a place where it makes more sense to work together if you’re open!

Great work.

PS

ReqLLM stands alone - but Ash and Jido build on top of it extensively.

1 Like
15

Hello,

Thank you for your answer!

If you want to integrate with JSV you may want to use JSV.Schema.normalize_collect to build a self contained schema with $defs and $refs that can be shared to an external service. This would work nicely with schema defined as modules (any module that exports a json_schema/0 function):

use JSV.Schema

defschema NextAction,
  task: string()

defschema MeetingRecap,
  summary: string(),
  next_action: NextAction

JSV.Schema.normalize_collect(MeetingRecap, as_root: true)

#=> %{
#   "$defs" => %{
#     "NextAction" => %{
#       "properties" => %{"task" => %{"type" => "string"}},
#       "required" => ["task"],
#       "title" => "NextAction",
#       "type" => "object",
#       "x-jsv-cast" => "Elixir.NextAction"
#     }
#   },
#   "properties" => %{
#     "next_action" => %{"$ref" => "#/$defs/NextAction"},
#     "summary" => %{"type" => "string"}
#   },
#   "required" => ["summary", "next_action"],
#   "title" => "MeetingRecap",
#   "type" => "object",
#   "x-jsv-cast" => "Elixir.MeetingRecap"
# }

Also please test with the main branch, an incoming release with breaking changes is on its way, it’s the last one before v1.0, I just did not have time to polish the docs.

If you don’t want to bake in, an option that accepts the output and returns {:ok, casted_value} | {:error, _} could be nice too.

So, in theory if your code provides keys in 1 or 2, then it will never drill down to 3. Is that good enough or do you need a way to explicitly remove 3 from the flow?

Yeah this is OK. The problem we have is we define an API key per-feature, but we have python apps that can only read the global env variables. So on dev machines it is a problem, but option 1 is ok for production code, thanks :slight_smile:

16

I worked on this a little yesterday so can tell you where I’ve landed.

Have decided not to integrate JSV directly. Instead sticking with our simple schema builder and validation code which is “good enough” 90% of the time. And then on top of that we provide a simple behaviour module for pluggable schema validation - so can integrate with JSV, Zoi or whatever. For JSV it would look a little like this:

defmodule JSVAdapter do
  @behaviour Omni.Schema.Adapter

  def to_schema(%JSV.Root{raw: raw}), do: raw

  def validate(%JSV.Root{} = root, input) do
    case JSV.validate(input, root) do
      {:ok, data} -> {:ok, data}
      # error pathway should return the error as a string
      {:error, error} -> {:error, format_error(error)}
    end
  end
end

Then in your call site you use it like this:

root = JSV.build!(MeetingRecap)
Omni.stream_text(model, context, output: {JSVAdapter, root})

And if needed, you can use the same pattern in tools:

defmodule MeetingTool do
  use Omni.Tool

  @root JSV.build!(MeetingRecap)
  def schema, do: {JSVAdapter, @root}

  # .... etc
end

If you want to play with this it’s already in the main branch - so will be in the next release.

1 Like
17

I think the schema needs to be collected to have proper refs when using modules but otherwise looks good.

Is there a way to use an external schema with the built-in schema system?

18

The :output option ordinarily receives a plain map that can serialise to a JSON schema. You can use the Omni.Schema builders if you want, or build it with some other tool. Then after the request the output is validated against the schema - the default implementation offloads this to Peri, using Peri.from_json_schema. I don’t know how well that will validate against complex schemas with $defs etc. Maybe it will be fine. But if not, then reach for an adapter

1 Like
19

I can’t find this on the main branch. Maybe you did not push it yet?

20

Looks sheepish :see_no_evil_monkey: :sweat_smile:

So sorry. It’s there now.