Skip to content

Model Providers

The provider system provides a unified interface for interacting with different LLM backends. All providers implement the ModelProvider abstract class.

Provider Registry

graph TD
    R[ProviderRegistry] --> |resolves model → provider| LMS[LM Studio]
    R --> ANT[Anthropic]
    R --> OAI[OpenAI]
    R --> |loads| MY[models.yaml]

    LMS --> |OpenAI-compatible API| LOCAL[Local Models]
    ANT --> |Anthropic API| CLAUDE[Claude Models]
    OAI --> |OpenAI API| GPT[GPT Models]

ProviderRegistry is a lazy singleton that: - Loads model definitions from providers/models.yaml - Resolves model names to their provider (e.g., "claude-3-5-sonnet-latest" → Anthropic) - Creates provider instances on demand with config from env vars or data/config.json

registry = get_registry()
provider, model_id = registry.get_provider_for_model("claude-3-5-sonnet-latest")
result = await provider.complete(messages, model_id)

Implementations

Provider Class API Config
LM Studio LMStudioProvider OpenAI-compatible (local) LMSTUDIO_BASE_URL (default: http://localhost:1234/v1)
Anthropic AnthropicProvider Anthropic API ANTHROPIC_API_KEY
OpenAI OpenAIProvider OpenAI API OPENAI_API_KEY

All providers support the same interface:

Method Description
complete(messages, model, **kwargs) Full completion (async)
stream(messages, model, **kwargs) Streaming completion → AsyncIterator[StreamChunk]
get_capabilities(model) Returns ModelCapabilities for a model
list_models() Returns available model names
health_check() Tests connectivity

Common Parameters

Parameter Type Default Description
temperature float 0.7 Sampling temperature
max_tokens int null Maximum output tokens
tools list[dict] null Function calling tools
tool_choice string/dict null Tool selection ("auto", "none", or specific)
stop list[string] null Stop sequences

Model Configuration

Models are defined in providers/models.yaml:

models:
  claude-3-5-sonnet-latest:
    provider: anthropic
    context_window: 200000
    supports_tools: true
    supports_vision: true
    cost_per_1k_input: 0.003
    cost_per_1k_output: 0.015

  llama3.2:
    provider: lmstudio
    context_window: 8192
    supports_tools: true
    supports_streaming: true

defaults:
  chat: null           # Uses DEFAULT_MODEL env var
  reasoning: null      # Falls back to chat default
  extraction: null     # Falls back to chat default

The registry also discovers models dynamically from LM Studio's /v1/models endpoint.

Environment Variables

Variable Description
DEFAULT_MODEL Default model for chat (default: "llama3.2")
LMSTUDIO_BASE_URL LM Studio API URL
ANTHROPIC_API_KEY Anthropic API key
OPENAI_API_KEY OpenAI API key
DEBUG_LOG_LLM_REQUESTS Log full request payloads when set to 1/true

Provider settings can also be set at runtime via POST /api/config/update.

API Endpoints

Endpoint Method Description
/api/providers GET List configured providers
/api/providers/models GET List models with capabilities
/api/providers/health GET Health check all providers

See API Endpoints for full details.

  • API Models: Provider — Message, CompletionResult, ModelCapabilities schemas
  • Config file: api/agentx_ai/providers/models.yaml