ExLLM - Multi-provider LLM library (inspired by Python’s LiteLLM)
I started this a couple weeks ago using Claude Code. I am a relatively experienced developer, but a newbie with Elixir and unfamiliar with standard conventions. This has not been fully reviewed line-by-line, but it’s in a mostly usable state so I thought I’d share it.
I have used this alongside my other new library - ExMCP (yeah, I’m terrible at naming) to build out a chat client similar to Block’s Goose. It’s been tested fairly extensively with Anthropic, OpenAI, and Ollama, but less with the others.
Feature List:
- Unified API: Single interface for multiple LLM providers
- Streaming Support: Real-time streaming responses with error recovery
- Cost Tracking: Automatic cost calculation for all API calls (if cost data is available)
- Token Estimation: Heuristic-based token counting for cost prediction
- Context Management: Automatic message truncation to fit model context windows
- Session Management: Built-in conversation state tracking and persistence
- Structured Outputs: Schema validation and retries via Instructor integration
- Function Calling: Unified interface for tool use across providers
- Model Discovery: Query and compare model capabilities across providers
- Capability Normalization: Automatic normalization of provider-specific feature names
- Error Recovery: Automatic retry with exponential backoff and stream resumption
- Mock Testing: Built-in mock adapter for testing without API calls (use with Response Caching)
- Response Caching: Cache real provider responses for offline testing and cost reduction
- Type Safety: Comprehensive typespecs and structured data
- Configurable: Flexible configuration system with multiple providers
- Extensible: Easy to add new LLM providers via adapter pattern
Currently supported providers:
- Anthropic
- OpenAI
- Gemini
- OpenRouter
- Groq
- xAI/Grok
Local providers:
- Ollama
- LM Studio
- Bumblebee
Please give feedback!
