LLM Provider Routing

AxonFlow supports server-side routing across multiple configured LLM providers. This is the page to read when you want to run more than one provider in production and control fallback, cost, latency, or hard provider pinning behavior.

What Routing Actually Controls

Routing applies to AxonFlow-managed provider calls such as:

• Proxy Mode
• MAP and related multi-step routed workflows

In Gateway Mode, your application still chooses and calls the provider directly. AxonFlow governs the request before and after that call, but it is not selecting the provider for you.

Provider Availability by Tier

Provider	Community	Enterprise
OpenAI	✅	✅
Anthropic	✅	✅
Azure OpenAI	✅	✅
Gemini	✅	✅
Ollama	✅	✅
AWS Bedrock	❌	✅
Custom Provider	❌	✅

Server Configuration

Routing is configured on the AxonFlow side:

Variable	Values	Default	Description
LLM_ROUTING_STRATEGY	weighted, round_robin, failover	weighted	Community and enterprise routing strategy
PROVIDER_WEIGHTS	provider:weight,...	Equal weights	Used by weighted routing
DEFAULT_LLM_PROVIDER	Provider name	None	Primary provider for failover
LLM_PROVIDERS	provider1,provider2,...	All configured	Comma-separated list of enabled provider names. Only providers in this list will be considered for routing. If not set, all configured providers are eligible.

Enterprise builds add:

Variable	Values	Default	Description
LLM_ROUTING_STRATEGY	cost_optimized	Off unless selected	Enterprise-only routing strategy
PROVIDER_COSTS	provider:cost,...	Runtime defaults	Used by cost-optimized routing

Enterprise Portal Routing Updates

In Enterprise deployments, the protected Customer Portal provider API can update routing metadata without redeploying the runtime:

PUT /api/v1/llm-providers/routing

The protected routing contract is provider-level weighting and ordering:

{
  "providers": [
    {
      "provider_name": "bedrock",
      "priority": 10,
      "weight": 0.7,
      "enabled": true
    },
    {
      "provider_name": "openai",
      "priority": 5,
      "weight": 0.3,
      "enabled": true
    }
  ]
}

Example portal-authenticated update:

curl -X PUT https://portal.example.com/api/v1/llm-providers/routing \
  --cookie "axonflow_session=$AXONFLOW_SESSION" \
  -H "Content-Type: application/json" \
  -d '{
    "providers": [
      {"provider_name": "bedrock", "priority": 10, "weight": 0.7, "enabled": true},
      {"provider_name": "openai", "priority": 5, "weight": 0.3, "enabled": true}
    ]
  }'

This protected API does not expose a separate strategy resource such as round_robin or cost_optimized. Treat priority, weight, and enabled as the operational controls for migration, failover, maintenance windows, and tenant-specific provider posture.

Routing Strategies

Weighted

LLM_ROUTING_STRATEGY=weighted
PROVIDER_WEIGHTS=openai:50,anthropic:30,gemini:20

Use this for gradual migrations, split traffic validation, and steady multi-provider production routing.

Round Robin

LLM_ROUTING_STRATEGY=round_robin

Use this when you want balanced distribution across healthy providers.

Failover

LLM_ROUTING_STRATEGY=failover
DEFAULT_LLM_PROVIDER=anthropic

Use this when one provider is primary and the others are operational backup paths.

Cost Optimized

LLM_ROUTING_STRATEGY=cost_optimized
PROVIDER_COSTS=ollama:0,anthropic:0.025,openai:0.03

Use this when you want the runtime to choose the cheapest healthy provider automatically. This strategy is enterprise-only.

Request-Level Hints

Clients can provide a provider and model preference in the request context:

const response = await client.proxyLLMCall({
  userToken: 'user-123',
  query: 'Draft a security review memo.',
  requestType: 'chat',
  context: {
    provider: 'anthropic',
    model: 'claude-sonnet-4-5-20250929',
  },
});

By default:

• context.provider is a preference
• AxonFlow may still route elsewhere if the selected provider is unhealthy or the server strategy says otherwise

To hard-pin a request:

const response = await client.proxyLLMCall({
  userToken: 'user-123',
  query: 'Draft a security review memo.',
  requestType: 'chat',
  context: {
    provider: 'anthropic',
    model: 'claude-sonnet-4-5-20250929',
    strict_provider: true,
  },
});

Organization Route Rows

An organization's route rows restrict and steer providers on top of the request-level hints: a policy row with a route action names allowed_providers (the only providers the request may use, failover included) and optionally a preferred_provider and a reason. A row applies to a request only when every one of its conditions holds.

Route rows are authored as dynamic policy rows, on /api/v1/policies, /api/v1/policies/dynamic and /api/v1/tenant-policies. That surface is deprecated: a write to it is refused with 409 LEGACY_POLICY_WRITE_FROZEN naming /api/v1/typed-policies, every response carries Link: </api/v1/typed-policies>; rel="successor-version" and X-AxonFlow-Removed-In: v12.0, and v12.0 removes the routes. The typed authoring contract has no equivalent yet: its route_restriction obligation carries an allowed-destination set only, and names neither a preferred provider nor a routing reason.

• allowed_providers intersect. When several rows apply, the request may use only the providers every applying row allows, whatever their order. Rows that permit no provider together refuse the request before any provider is called (no_compliant_provider), rather than lifting the restriction. A caller whose segment could not be established, or whose rows could not be resolved or read at all, is refused the same way and fail-closed, with segment_not_established, segment_resolution_failed or decision_enforcement_unavailable.
• The last applying row wins preferred_provider. Rows are walked by priority, highest first, then newest first, and each applying row that names a preferred provider replaces the one before it. So the winner is the lowest-priority applying row. The routing reason is taken the same way but independently: the last applying row that carries a reason replaces it, whether or not that row names a preferred provider, so the reason reported can come from a different row than the provider.
• A route row outranks the request's own hint. A context.provider outside the allowed providers is not used, and strict_provider cannot widen the list.
• They bind the LLM calls the orchestrator makes for your organization. That covers /api/v1/process and, from v11.1.0, the calls a multi-agent plan makes: the two that generate the plan and each llm-call step when it runs, resumed steps included. Each is decided over the prompt it sends, so a row conditioned on content applies to a step's prompt as it does to a query. Plan generation answers a refusal 403 naming the reason; a refused step fails with it. One route is not covered: POST /api/v1/llm-providers/{name}/test calls the named provider directly with the prompt you pass it, through a key registered for the tenant. It is a smoke test of that provider rather than a call on the organization's route, so no route row applies to it.

When Teams Usually Need More Than One Provider

• A cloud primary plus Ollama fallback for development or regulated environments
• Anthropic for long-context reasoning, OpenAI for general-purpose chat
• Azure OpenAI for production alignment with existing Azure controls, but a second provider for resilience
• Bedrock added later as the company moves from pilot to governed enterprise rollout

Practical Guidance

• Start with one provider and verify policy behavior first.
• Add a second provider when you have a concrete reason: resilience, cost, region, or model fit.
• Use soft preferences first and only enable strict pinning where the downstream workflow truly requires it.

Common Production Use Cases

• OpenAI primary with Anthropic failover for customer-facing copilots
• Ollama plus cloud fallback for private enterprise assistants
• Azure OpenAI for Microsoft-centric deployment with a second provider for resilience
• Bedrock for regulated AWS estates that need enterprise-only provider operations

Enterprise Migration Playbooks

OpenAI to Bedrock

1. Introduce Bedrock at a small traffic share, such as 10%, while OpenAI remains primary.
2. Increase Bedrock's share once model quality, latency, regional posture, and cost metadata are validated.
3. Keep OpenAI enabled as a fallback until incident-response and rollback paths are proven.

{
  "providers": [
    {"provider_name": "openai", "priority": 10, "weight": 0.9, "enabled": true},
    {"provider_name": "bedrock", "priority": 5, "weight": 0.1, "enabled": true}
  ]
}

Private Inference with Ollama

Use Ollama when internal workloads can run on private model infrastructure but a managed provider is still useful for higher-complexity fallback.

{
  "providers": [
    {"provider_name": "ollama", "priority": 10, "weight": 0.6, "enabled": true},
    {"provider_name": "anthropic", "priority": 5, "weight": 0.4, "enabled": true}
  ]
}

SaaS vs In-VPC Scope

Routing decisions apply within the tenant scope visible to the current session. In SaaS mode, that is the customer's tenant in the managed service. In In-VPC mode, the API is still session-tenant scoped, but a single-tenant deployment often behaves operationally like one shared enterprise routing posture.

Next Steps

• LLM Providers Overview
• OpenAI
• Anthropic
• Google Gemini
• Enterprise Provider Features

Provider Rollout Checklist

Before this provider path becomes production traffic, connect it to the runtime operating model:

• decide whether calls should use Proxy Mode, Gateway Mode, or Direct Orchestrator
• document credential ownership with the Provider And Credential Matrix
• use Provider Routing for failover, pinning, and cost-aware routing choices
• review Community vs Evaluation vs Enterprise when runtime provider management, Bedrock, or portal operations become requirements