ReqLLM - Composable LLM client built on Req

Your Libraries & OS Mentoring Libraries
, ,
1

Hey everyone!

I’m excited to share ReqLLM - a new approach to LLM interactions in Elixir that I’ve been working on. After building agent systems with various LLM clients, I kept running into the same frustrations: they either lacked Elixir’s composability principles or didn’t integrate well with existing HTTP pipelines.

Why Another LLM Client?

While building out Jido features, I needed a lower-level API for making LLM requests. ReqLLM is built on Req, with each Provider built as a Req plugin that handles provider-specific wire formats. It’s designed to compose naturally with your existing Req-based applications.

Core Architecture

Plugin-Based Providers: Each LLM provider (Anthropic, OpenAI, Google, etc.) is a composable Req plugin.

Typed Data Structures: Every interaction uses proper structs (Context, Message, StreamChunk, Tool, ContentPart) that implement Jason.Encoder - no more wrestling with nested maps.

Two Client Layers: High-level helpers for quick wins (generate_text/3, stream_text/3, generate_object/4, etc) plus low-level Req plugin access when you need full control.

Built-in Observability: Usage and cost tracking on every response based on metadata sync’d from https://models.dev

Quick Example

# Simple approach
ReqLLM.put_key(:anthropic_api_key, "sk-ant-...")
{:ok, text} = ReqLLM.generate_text!("anthropic:claude-3-sonnet", "Hello")

# Tool calling with structured responses
weather_tool = ReqLLM.tool(
  name: "get_weather", 
  description: "Get weather for a location",
  parameter_schema: [location: [type: :string, required: true]],
  callback: fn args -> {:ok, "Sunny, 72°F"} end
)

{:ok, response} = ReqLLM.generate_text(
  "anthropic:claude-3-sonnet", 
  "What's the weather in Paris?",
  tools: [weather_tool]
)

Current Status

ReqLLM 1.0-rc is available on Hex with 45+ providers and 665+ models (auto-synced from models.dev). I’m using it in production for Jido agent systems and it’s been solid. Planning to add Ollama/LocalAI support and enhanced streaming soon.

Resources

I’d love to hear your thoughts and see what you build with it! The plugin architecture makes it pretty straightforward to add new providers if there’s one you need.

42 Likes
How is the LLM scene with Elixir/Phoenix?
Responses - LLM library supporting OpenAI's and xAI's Responses API
2

This is fantastic! Elixir does come with many of the building blocks for rolling your own agentic system but the “talking to different LLMs” is definitely one of the more time consuming bits. I also like that you have a purely data driven API for defining tools.

My only request would be to make the bit where providers automatically fetch keys optional (either opt-in or opt-out), as turning that on/off would be important for folks writing services where you BYOK (like Tidewave).

Also have you implemented the APIs for dealing with reasoning tokens? OpenAI, Anthropic, OpenRouter, etc all have different APIs for them, which is also annoying to deal with.

PS: You forgot to make Credo a dev-only dependency. :slight_smile:

9 Likes
3

PS: You forgot to make Credo a dev-only dependency.

Thanks for the catch! So tough to pull the trigger on a release :slight_smile:

My only request would be to make the bit where providers automatically fetch keys optional (either opt-in or opt-out), as turning that on/off would be important for folks writing services where you BYOK (like Tidewave).

Absolutely, I had similar ideas - implemented it twice - didn’t feel right yet. Agree this is necessary.

Also have you implemented the APIs for dealing with reasoning tokens?

Yes, this is currently unreleased - I hit another weird snag and pulled back - but it will be there by the formal 1.0.

4 Likes
4

Looks useful. Thanks for making it.

I thought functions ending in a bang would return the bare result, not a tuple. Curious why this wouldn’t follow that convention?

1 Like
5

Great question - and I agree - I’ll make a PR

This is only due to my eyes glazing over after sprinting on this for a week!!!

2 Likes
6

I’m a little confused by the configuration, one place mentions Kagi, another JidoKeys. Is it possible to just use normal elixir configuration and/or pass keys explicitly?

7

one place mentions Kagi, another JidoKeys

Kagi was the old package name - the new is Jido Keys so that’s just a docs error

There’s a PR already up to help clarify this!

8

What is json fetched from models.dev used for? I can see it’s being parsed for metadata and is made available for via metadata/0 function, but I don’t see being used much. Does it actually drive any provider behavior?

9

It does a few things:

  • validates the model_spec strings point to a defined model
  • pulls cost info, so total request cost can be calculated
  • providers can take advantage of modality and capability metadata, but that’s still being built out

ie. calling a future generate_image/3 method on a model that doesn’t support image generation would return an error before any API call would be made

1 Like
10

Hi @mikehostetler, I love the inspiration on the AI SDK. I was wondering what is your roadmap?
On a more selfish note, right now I’d be more interested in the OpenAI provider with object generation and responses API :smile:
Let me now if this is on your radar, if not, I’m happy to help somehow.
Cheers.

11

Object Generation via OpenAI will be baked before this hits 1.0

The responses API will be there, but likely only available via the lower-level API - the main purpose/vision of the package is to make the flexible, lower-level API available for all the fun nuances that an LLM API provides, while keeping the “happy path” streamlined via generate_text/3 (and others)

Hopefully we can then have our cake and eat it too :smiley:

Work is pretty active on this right now, so shouldn’t be too long before this is ready - a few days or so

2 Likes
12

I wrote a library specifically to support that: OpenAI.Responses — OpenAI.Responses v0.8.4 (but the downside, of course, is that unlike Mike’s library, you can’t really switch providers). You might try it while you are waiting for ReqLLM to add this functionality.

One of the reasons I did that (as opposed to just using existing solutions like LangChain) is that if you only need to work with OpenAI, Responses API is superior: cheaper on average as tokens caching is more efficient, provides additional functionality such as web search, background requests, automatic conversation building via previous_response_id, support for multiple developer messages, etc.

But the downside is that it’s hard to make it fit into the “Chat Completions” world view that every other provider (following the original OpenAI’s API) is supporting.

1 Like
13

Congratulations on launching ReqLLM! I really like the wide support of many providers and the composable design.

One quick question: you mentioned cost tracking, but I can’t find how to enable it - e.g. the following script only prints token usage, but not cost, even after running mix req_llm.model_sync

alias ReqLLM.Response

ReqLLM.put_key(:google_api_key, System.fetch_env!("GOOGLE_API_KEY"))
ReqLLM.put_key(:openai_api_key, System.fetch_env!("OPENAI_API_KEY"))
ReqLLM.put_key(:xai_api_key, System.fetch_env!("XAI_API_KEY"))

models = ["openai:gpt-5-nano", "xai:grok-4-fast-non-reasoning", "google:gemini-2.5-flash"]

models
|> Enum.each(fn model ->
  {:ok, response} = ReqLLM.generate_text(model, "What model are you?")

  text = Response.text(response)
  usage = Response.usage(response)
  IO.puts("#{model}: #{text}\n(#{inspect(usage)}\n\n")
end)

prints only token usage:

openai:gpt-5-nano: I’m ChatGPT, a large language model created by OpenAI. I’m based on the GPT-4 architecture (the GPT-4 family of models). If you want, I can share more about my capabilities or limitations.
(%{input_tokens: 11, output_tokens: 824, total_tokens: 835}


xai:grok-4-fast-non-reasoning: I'm Grok, built by xAI.
(%{input_tokens: 132, output_tokens: 9, total_tokens: 142, live_search_sources: 0}


google:gemini-2.5-flash: I am a large language model, trained by Google.
(%{input_tokens: 6, output_tokens: 11, total_tokens: 17, cached_content_token_count: 0, total_token_count: 42}
1 Like
14

Thanks!!! I’m growing more satisfied with how the library has come together!

re: Cost tracking, it should happen automatically and be available via the usage code you showed above. I’ll file a bug on this.

EDIT: I see you’ve already created an issue, THANK YOU!!!

1 Like
15

I’ve been exploring jido and jido_ai which integrates with @brainlid ‘s LangChain. Since LangChain also abstracts HTTP requests to LLMs, I’m curious about how much overlap there is with ReqLLM.

I understand that LangChain covers many features outside the scope of ReqLLM, but if you were starting a new project with jido + jido_ai, which one would you choose?

Thanks again for your amazing contributions to the community!

16

Thanks!

jido_ai will be moving to use ReqLLM very shortly - and we will be dropping support for Langchain and Instructor.

API’s will be maintained - so you should be good to start and not worry too much about things changing.

LLM API’s are pretty dynamic, so I can’t promise 100% - but we’re doing our best to build a solid foundation for the Elixir community.

7 Likes
Llm_composer - Elixir library for building LLM-based applications
17

ReqLLM 1.0.0-rc.7 Released

I’ve published the 1.0.0-rc.7 release of ReqLLM with Elixir 1.19 support on Hex.

It’s been a journey to build this package. LLM Providers are all over the place - but with this release, I’ve been able to successfully prove that the high level API now supports 136 models via a custom fixture testing system I developed.

Several companies and projects have already begun integrating and battle testing this package. With this release, I’m now focused on getting to a stable 1.0 release.

What’s New

Test Coverage: 136 models validated across 10 providers with fixture-based testing against 10 scenarios:

  • Core: basic generation, streaming, token limits, usage metrics
  • Capabilities: multi-tool calling, tool round-trip, tool avoidance, object generation (streaming/non-streaming), reasoning tokens

Providers: Added Amazon Bedrock (streaming, multi-model, AWS SigV4 auth) and Z.AI (GLM-4.5/4.6, 204K context).

Architecture: Simplified tool call handling with ReqLLM.ToolCall helpers, provider normalize_model_id/1 callback, improved Context API with append/2 and prepend/2.

Examples: Full agent implementation and 16+ runnable scripts covering embeddings, multimodal analysis, tool calling, object generation, reasoning tokens, context reuse.

Validation: Run mix mc to see coverage status. Use REQ_LLM_FIXTURES_MODE=record to test against live APIs.

Links

9 Likes
18

Nice! I hope to get this soon! I like design of Jido.

1 Like
19

Hello, we are using Langchain in my team currently.

What would be the use cases where ReqLLM could be a better fit?

We are motsly calling chats with tool calling and structured outputs. Also opentelemetry is very important for us.

Thank you :slight_smile:

1 Like
20

Langchain is great! If it works - keep using it

ReqLLM gives a bit “lower” API level access to LLM’s - which I needed for my Jido project

1 Like