mikehostetler

ReqLLM - Composable LLM client built on Req

Hey everyone!

I’m excited to share ReqLLM - a new approach to LLM interactions in Elixir that I’ve been working on. After building agent systems with various LLM clients, I kept running into the same frustrations: they either lacked Elixir’s composability principles or didn’t integrate well with existing HTTP pipelines.

Why Another LLM Client?

While building out Jido features, I needed a lower-level API for making LLM requests. ReqLLM is built on Req, with each Provider built as a Req plugin that handles provider-specific wire formats. It’s designed to compose naturally with your existing Req-based applications.

Core Architecture

Plugin-Based Providers: Each LLM provider (Anthropic, OpenAI, Google, etc.) is a composable Req plugin.

Typed Data Structures: Every interaction uses proper structs (Context, Message, StreamChunk, Tool, ContentPart) that implement Jason.Encoder - no more wrestling with nested maps.

Two Client Layers: High-level helpers for quick wins (generate_text/3, stream_text/3, generate_object/4, etc) plus low-level Req plugin access when you need full control.

Built-in Observability: Usage and cost tracking on every response based on metadata sync’d from https://models.dev

Quick Example

# Simple approach
ReqLLM.put_key(:anthropic_api_key, "sk-ant-...")
{:ok, text} = ReqLLM.generate_text!("anthropic:claude-3-sonnet", "Hello")

# Tool calling with structured responses
weather_tool = ReqLLM.tool(
  name: "get_weather", 
  description: "Get weather for a location",
  parameter_schema: [location: [type: :string, required: true]],
  callback: fn args -> {:ok, "Sunny, 72°F"} end
)

{:ok, response} = ReqLLM.generate_text(
  "anthropic:claude-3-sonnet", 
  "What's the weather in Paris?",
  tools: [weather_tool]
)

Current Status

ReqLLM 1.0-rc is available on Hex with 45+ providers and 665+ models (auto-synced from models.dev). I’m using it in production for Jido agent systems and it’s been solid. Planning to add Ollama/LocalAI support and enhanced streaming soon.

Resources

Hex Package: req_llm | Hex
Documentation: ReqLLM v1.17.0 — Documentation
Getting Started Guide: https://hexdocs.pm/req_llm/getting-started.html
GitHub:

https://github.com/agentjido/req_llm

I’d love to hear your thoughts and see what you build with it! The plugin architecture makes it pretty straightforward to add new providers if there’s one you need.

30 comments

#library #req #llm #jido

43 3445 30

2025-10-28 13:21:22 UTC

First 10 of 30 Posts

josevalim

Creator of Elixir

This is fantastic! Elixir does come with many of the building blocks for rolling your own agentic system but the “talking to different LLMs” is definitely one of the more time consuming bits. I also like that you have a purely data driven API for defining tools.

My only request would be to make the bit where providers automatically fetch keys optional (either opt-in or opt-out), as turning that on/off would be important for folks writing services where you BYOK (like Tidewave).

Also have you implemented the APIs for dealing with reasoning tokens? OpenAI, Anthropic, OpenRouter, etc all have different APIs for them, which is also annoying to deal with.

PS: You forgot to make Credo a dev-only dependency.

Post #1

mikehostetler

PS: You forgot to make Credo a dev-only dependency.

Thanks for the catch! So tough to pull the trigger on a release

My only request would be to make the bit where providers automatically fetch keys optional (either opt-in or opt-out), as turning that on/off would be important for folks writing services where you BYOK (like Tidewave).

Absolutely, I had similar ideas - implemented it twice - didn’t feel right yet. Agree this is necessary.

Also have you implemented the APIs for dealing with reasoning tokens?

Yes, this is currently unreleased - I hit another weird snag and pulled back - but it will be there by the formal 1.0.

Post #2

jam

Looks useful. Thanks for making it.

I thought functions ending in a bang would return the bare result, not a tuple. Curious why this wouldn’t follow that convention?

Post #3

mikehostetler

Great question - and I agree - I’ll make a PR

This is only due to my eyes glazing over after sprinting on this for a week!!!

Post #4

martosaur

I’m a little confused by the configuration, one place mentions Kagi, another JidoKeys. Is it possible to just use normal elixir configuration and/or pass keys explicitly?

Post #5

mikehostetler

one place mentions Kagi, another JidoKeys

Kagi was the old package name - the new is Jido Keys so that’s just a docs error

There’s a PR already up to help clarify this!

https://github.com/agentjido/req_llm/pull/3

Post #6

martosaur

What is json fetched from models.dev used for? I can see it’s being parsed for metadata and is made available for via metadata/0 function, but I don’t see being used much. Does it actually drive any provider behavior?

Post #7

mikehostetler

It does a few things:

validates the model_spec strings point to a defined model
pulls cost info, so total request cost can be calculated
providers can take advantage of modality and capability metadata, but that’s still being built out

ie. calling a future generate_image/3 method on a model that doesn’t support image generation would return an error before any API call would be made

Post #8

thiagovarela

Hi @mikehostetler, I love the inspiration on the AI SDK. I was wondering what is your roadmap?
On a more selfish note, right now I’d be more interested in the OpenAI provider with object generation and responses API
Let me now if this is on your radar, if not, I’m happy to help somehow.
Cheers.

Post #9

mikehostetler

Object Generation via OpenAI will be baked before this hits 1.0

The responses API will be there, but likely only available via the lower-level API - the main purpose/vision of the package is to make the flexible, lower-level API available for all the fun nuances that an LLM API provides, while keeping the “happy path” streamlined via generate_text/3 (and others)

Hopefully we can then have our cake and eat it too