Skip to main content

LLM Providers Overview

AxonFlow supports multiple LLM providers out of the box, allowing you to choose the best provider for your use case, compliance requirements, and cost constraints.

Supported Providers​

ProviderModelsBest ForEdition
OpenAIGPT-4o, GPT-4o-mini, GPT-4, o1, o3-miniGeneral purpose, latest capabilitiesCommunity
Azure OpenAIGPT-4o, GPT-4, GPT-4-turbo (Azure-hosted)Azure ecosystem, enterprise security, data residencyCommunity
AnthropicClaude Opus 4.6, Claude 4.5 Sonnet, Claude 4.5 Haiku, Claude 3.5 SonnetLong context, safety-focused, reasoningCommunity
Google GeminiGemini 2.0 Flash, Gemini 1.5 Pro, Gemini 1.5 FlashMultimodal, Google ecosystemCommunity
OllamaLlama 3.3, Mistral, Mixtral, DeepSeekSelf-hosted, air-gapped environmentsCommunity
AWS BedrockClaude, Llama, Titan, MistralHIPAA compliance, VPC isolationEnterprise
CustomAnyCustom providers via SDKCommunity

Try It: Route a Query Through AxonFlow​

Send a query through AxonFlow and have it routed to your configured LLM provider. AxonFlow handles provider selection, policy enforcement, and audit logging automatically.

curl​

curl -X POST http://localhost:8080/api/v1/query/execute \
  -H "Content-Type: application/json" \
  -H "X-Client-Id: my-tenant" \
  -H "X-Client-Secret: your-client-secret" \
  -d '{
    "query": "Explain the difference between symmetric and asymmetric encryption",
    "provider": "openai",
    "model": "gpt-4o"
  }'

Response:

{
  "response": "Symmetric encryption uses a single shared key for both encryption and decryption...",
  "provider": "openai",
  "model": "gpt-4o",
  "token_usage": {
    "prompt_tokens": 14,
    "completion_tokens": 210,
    "total_tokens": 224
  },
  "policy_info": {
    "allowed": true,
    "applied_policies": ["pii-global-email", "security-sqli-union"],
    "risk_score": 0.05
  }
}

SDK Examples​

import { AxonFlow } from '@axonflow/sdk';

const client = new AxonFlow({
  endpoint: 'http://localhost:8080',
  clientId: 'my-tenant',
  clientSecret: 'your-client-secret',
});

const result = await client.executeQuery({
  query: 'Explain the difference between symmetric and asymmetric encryption',
  provider: 'openai',
  model: 'gpt-4o',
});

console.log(result.response);
console.log('Tokens used:', result.tokenUsage.totalTokens);
console.log('Policy allowed:', result.policyInfo.allowed);

Provider Selection​

Choose your LLM provider based on:

Compliance Requirements​

RequirementRecommended Provider
HIPAAAWS Bedrock with VPC endpoints
FedRAMPOllama (self-hosted) or AWS GovCloud
Air-gappedOllama
Data residencyBedrock (regional) or Ollama

Cost Optimization​

ProviderCost per 1K tokens (approx)Best For
Ollama$0 (hardware only)High volume, predictable cost
Bedrock (Claude)$0.015HIPAA with cost savings
OpenAI (GPT-4o)$0.005General purpose
Anthropic (Sonnet)$0.003Cost-effective quality

Latency​

ProviderTypical LatencyBest For
Ollama50-200msReal-time applications
OpenAI200-500msInteractive apps
Bedrock300-800msBatch processing

Configuration​

Environment Variables​

The simplest way to configure providers:

# OpenAI
export OPENAI_API_KEY=sk-xxx

# Azure OpenAI
export AZURE_OPENAI_ENDPOINT=https://your-resource.cognitiveservices.azure.com
export AZURE_OPENAI_API_KEY=your-api-key
export AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o-mini

# Anthropic
export ANTHROPIC_API_KEY=sk-ant-xxx

# AWS Bedrock (uses AWS credential chain)
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx

# Google Gemini
export GOOGLE_API_KEY=xxx

# Ollama
export OLLAMA_ENDPOINT=http://localhost:11434

YAML Configuration​

For more control, use YAML configuration:

# axonflow.yaml
version: "1.0"

llm_providers:
  openai:
    enabled: true
    config:
      model: gpt-4o
      max_tokens: 4096
    credentials:
      api_key: ${OPENAI_API_KEY}
    priority: 10
    weight: 0.5

  anthropic:
    enabled: true
    config:
      model: claude-sonnet-4-20250514
      max_tokens: 8192
    credentials:
      api_key: ${ANTHROPIC_API_KEY}
    priority: 8
    weight: 0.3

  bedrock:
    enabled: true
    config:
      model: anthropic.claude-sonnet-4-20250514-v1:0
      region: us-east-1
      max_tokens: 4096
    priority: 5
    weight: 0.2

  ollama:
    enabled: true
    config:
      endpoint: ${OLLAMA_ENDPOINT:-http://localhost:11434}
      model: llama3.2:latest
    priority: 3
    weight: 0.0  # Fallback only

Multi-Provider Routing​

AxonFlow supports intelligent routing across multiple providers:

Routing Strategies​

StrategyDescriptionUse Case
PriorityUse highest priority availableFailover scenarios
WeightedDistribute by weightLoad balancing
Cost-OptimizedRoute to cheapest firstCost reduction
Round-RobinEven distributionEven load distribution

Automatic Failover​

When a provider fails:

  1. Request is retried with exponential backoff
  2. After threshold failures, provider is marked unhealthy
  3. Traffic automatically routes to healthy providers
  4. Health checks restore provider when recovered

Circuit Breaker​

Prevents cascading failures:

  • Opens after configurable failure threshold (default: 5)
  • Blocks requests to unhealthy provider
  • Automatically closes after reset timeout

Provider-Specific Guides​

Community vs Enterprise​

FeatureCommunityEnterprise
OpenAI, Anthropic, Gemini, Ollama✅✅
Azure OpenAI (environment variables)✅✅
Multi-provider routing and failover✅✅
Circuit breaker and health checks✅✅
Custom Provider SDK✅✅
Azure OpenAI (managed credentials)❌✅
AWS Bedrock (HIPAA, VPC isolation)❌✅
Runtime provider configuration❌✅
Credential encryption and rotation❌✅
Per-provider cost controls and budgets❌✅
Usage analytics and cost dashboards❌✅
SLA management and alerting❌✅
Enterprise LLM Management

Enterprise adds Azure OpenAI with managed credentials, AWS Bedrock for HIPAA-compliant deployments, credential encryption and rotation, per-provider cost controls, and usage analytics dashboards. Build custom providers with the Custom Provider SDK to integrate proprietary or specialized LLM endpoints. Compare Editions | Request Demo | AWS Marketplace

See Enterprise Provider Features for details.

Last updated on