mikehostetler

mikehostetler

Hey everyone!

I’m excited to share ReqLLM - a new approach to LLM interactions in Elixir that I’ve been working on. After building agent systems with various LLM clients, I kept running into the same frustrations: they either lacked Elixir’s composability principles or didn’t integrate well with existing HTTP pipelines.

Why Another LLM Client?

While building out Jido features, I needed a lower-level API for making LLM requests. ReqLLM is built on Req, with each Provider built as a Req plugin that handles provider-specific wire formats. It’s designed to compose naturally with your existing Req-based applications.

Core Architecture

Plugin-Based Providers: Each LLM provider (Anthropic, OpenAI, Google, etc.) is a composable Req plugin.

Typed Data Structures: Every interaction uses proper structs (Context, Message, StreamChunk, Tool, ContentPart) that implement Jason.Encoder - no more wrestling with nested maps.

Two Client Layers: High-level helpers for quick wins (generate_text/3, stream_text/3, generate_object/4, etc) plus low-level Req plugin access when you need full control.

Built-in Observability: Usage and cost tracking on every response based on metadata sync’d from https://models.dev

Quick Example

# Simple approach
ReqLLM.put_key(:anthropic_api_key, "sk-ant-...")
{:ok, text} = ReqLLM.generate_text!("anthropic:claude-3-sonnet", "Hello")

# Tool calling with structured responses
weather_tool = ReqLLM.tool(
  name: "get_weather", 
  description: "Get weather for a location",
  parameter_schema: [location: [type: :string, required: true]],
  callback: fn args -> {:ok, "Sunny, 72°F"} end
)

{:ok, response} = ReqLLM.generate_text(
  "anthropic:claude-3-sonnet", 
  "What's the weather in Paris?",
  tools: [weather_tool]
)

Current Status

ReqLLM 1.0-rc is available on Hex with 45+ providers and 665+ models (auto-synced from models.dev). I’m using it in production for Jido agent systems and it’s been solid. Planning to add Ollama/LocalAI support and enhanced streaming soon.

Resources

https://github.com/agentjido/req_llm

I’d love to hear your thoughts and see what you build with it! The plugin architecture makes it pretty straightforward to add new providers if there’s one you need.

Showing Posts 1 to 10

josevalim

josevalim

Creator of Elixir

This is fantastic! Elixir does come with many of the building blocks for rolling your own agentic system but the “talking to different LLMs” is definitely one of the more time consuming bits. I also like that you have a purely data driven API for defining tools.

My only request would be to make the bit where providers automatically fetch keys optional (either opt-in or opt-out), as turning that on/off would be important for folks writing services where you BYOK (like Tidewave).

Also have you implemented the APIs for dealing with reasoning tokens? OpenAI, Anthropic, OpenRouter, etc all have different APIs for them, which is also annoying to deal with.

PS: You forgot to make Credo a dev-only dependency. :slight_smile:

mikehostetler

mikehostetler OP

PS: You forgot to make Credo a dev-only dependency.

Thanks for the catch! So tough to pull the trigger on a release :slight_smile:

My only request would be to make the bit where providers automatically fetch keys optional (either opt-in or opt-out), as turning that on/off would be important for folks writing services where you BYOK (like Tidewave).

Absolutely, I had similar ideas - implemented it twice - didn’t feel right yet. Agree this is necessary.

Also have you implemented the APIs for dealing with reasoning tokens?

Yes, this is currently unreleased - I hit another weird snag and pulled back - but it will be there by the formal 1.0.

jam

jam

Looks useful. Thanks for making it.

I thought functions ending in a bang would return the bare result, not a tuple. Curious why this wouldn’t follow that convention?

mikehostetler

mikehostetler OP

Great question - and I agree - I’ll make a PR

This is only due to my eyes glazing over after sprinting on this for a week!!!

martosaur

martosaur

I’m a little confused by the configuration, one place mentions Kagi, another JidoKeys. Is it possible to just use normal elixir configuration and/or pass keys explicitly?

mikehostetler

mikehostetler OP

one place mentions Kagi, another JidoKeys

Kagi was the old package name - the new is Jido Keys so that’s just a docs error

There’s a PR already up to help clarify this!

https://github.com/agentjido/req_llm/pull/3

martosaur

martosaur

What is json fetched from models.dev used for? I can see it’s being parsed for metadata and is made available for via metadata/0 function, but I don’t see being used much. Does it actually drive any provider behavior?

mikehostetler

mikehostetler OP

It does a few things:

  • validates the model_spec strings point to a defined model
  • pulls cost info, so total request cost can be calculated
  • providers can take advantage of modality and capability metadata, but that’s still being built out

ie. calling a future generate_image/3 method on a model that doesn’t support image generation would return an error before any API call would be made

thiagovarela

thiagovarela

Hi @mikehostetler, I love the inspiration on the AI SDK. I was wondering what is your roadmap?
On a more selfish note, right now I’d be more interested in the OpenAI provider with object generation and responses API :smile:
Let me now if this is on your radar, if not, I’m happy to help somehow.
Cheers.

mikehostetler

mikehostetler OP

Object Generation via OpenAI will be baked before this hits 1.0

The responses API will be there, but likely only available via the lower-level API - the main purpose/vision of the package is to make the flexible, lower-level API available for all the fun nuances that an LLM API provides, while keeping the “happy path” streamlined via generate_text/3 (and others)

Hopefully we can then have our cake and eat it too :smiley:

Work is pretty active on this right now, so shouldn’t be too long before this is ready - a few days or so

Where Next? Top

Trending in Announcing Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
261 13183 112
New
achempion
Hi, I would like to tell about my initiative to further maintain and develop Waffle project which is the fork of Arc library. The progre...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
charlenopires
Hi everyone :waving_hand: I’ve been working on PhiaUI, an open-source UI component library for Phoenix LiveView, and I wanted to share i...
New

Other Trending Topics Top

OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
bartblast
If you haven’t tried Hologram yet, what’s stopped you? Here’s why I’m asking. Building a framework turns out to be two jobs, the framewo...
New
mat-hek
Popcorn, the library that runs Elixir code in the browser, is now stable :popcorn: This is possible because it now builds on a very stab...
New
AstonJ
With advancements in AI gathering pace and set to increasingly accelerate, perhaps it’s worth having an annual thread on the topic - feel...
New
sntran
In my third year of college, around 2010, I found Erlang. A few years later, I found Elixir. I liked the ideas at once: many small proces...
New
gtcode
I’ve been tinkering with Lynx (by José) via hold_ledger. I used AI to write the code, a separate specification, the tests, and the Lean p...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews