ExLLM - Multi-provider LLM library (inspired by Python’s LiteLLM)

ExLLM - Multi-provider LLM library (inspired by Python’s LiteLLM)

I started this a couple weeks ago using Claude Code. I am a relatively experienced developer, but a newbie with Elixir and unfamiliar with standard conventions. This has not been fully reviewed line-by-line, but it’s in a mostly usable state so I thought I’d share it.

I have used this alongside my other new library - ExMCP (yeah, I’m terrible at naming) to build out a chat client similar to Block’s Goose. It’s been tested fairly extensively with Anthropic, OpenAI, and Ollama, but less with the others.

Feature List:

  • Unified API: Single interface for multiple LLM providers
  • Streaming Support: Real-time streaming responses with error recovery
  • Cost Tracking: Automatic cost calculation for all API calls (if cost data is available)
  • Token Estimation: Heuristic-based token counting for cost prediction
  • Context Management: Automatic message truncation to fit model context windows
  • Session Management: Built-in conversation state tracking and persistence
  • Structured Outputs: Schema validation and retries via Instructor integration
  • Function Calling: Unified interface for tool use across providers
  • Model Discovery: Query and compare model capabilities across providers
  • Capability Normalization: Automatic normalization of provider-specific feature names
  • Error Recovery: Automatic retry with exponential backoff and stream resumption
  • Mock Testing: Built-in mock adapter for testing without API calls (use with Response Caching)
  • Response Caching: Cache real provider responses for offline testing and cost reduction
  • Type Safety: Comprehensive typespecs and structured data
  • Configurable: Flexible configuration system with multiple providers
  • Extensible: Easy to add new LLM providers via adapter pattern

Currently supported providers:

  • Anthropic
  • OpenAI
  • Gemini
  • OpenRouter
  • Groq
  • xAI/Grok

Local providers:

  • Ollama
  • LM Studio
  • Bumblebee

Please give feedback!

6 Likes