Skip to main content

LLM Provider Management API

Use this API to register providers, inspect provider health, update routing weights, and control the provider layer behind POST /api/v1/process. These routes are served by the Orchestrator and are commonly reached through the Agent proxy.

Generated API reference

Full request bodies, response schemas, and status codes for every endpoint on this page are in the generated Orchestrator API reference, rendered from the published OpenAPI spec. Use the links in the table below to jump straight to an operation.

Endpoints​

MethodPathPurposeReference
GET/api/v1/llm-provider-typesList available provider factorieslistLLMProviderTypes
GET/api/v1/llm-providersList configured providerslistLLMProviders
POST/api/v1/llm-providersCreate a providercreateLLMProvider
GET/api/v1/llm-providers/{name}Get provider configgetLLMProvider
PUT/api/v1/llm-providers/{name}Update provider configupdateLLMProvider
DELETE/api/v1/llm-providers/{name}Delete provider configdeleteLLMProvider
GET/api/v1/llm-providers/routingRead routing weightsgetLLMRoutingConfig
PUT/api/v1/llm-providers/routingUpdate routing weightsupdateLLMRoutingWeights
GET/api/v1/llm-providers/statusHealth for all providersgetAllLLMProvidersStatus
GET/api/v1/llm-providers/{name}/healthHealth for one providergetLLMProviderHealth
POST/api/v1/llm-providers/{name}/testConnectivity testtestLLMProvider

Compatibility routes:

MethodPathPurposeReference
GET/api/v1/providers/statusLegacy runtime status map used by operational toolinggetProviderStatus
PUT/api/v1/providers/weightsLegacy runtime weight update using decimal weightsupdateProviderWeights

Base URL: http://localhost:8080 (Agent)

You can also call the same routes directly on the Orchestrator at http://localhost:8081.

Request-Level Routing Controls (Advanced)​

For inference requests sent to /api/v1/process, provider selection controls are passed in context:

  • context.provider (string): preferred provider (fallback allowed)
  • context.strict_provider (boolean, optional): hard-pin provider for that request (no fallback)

Example:

curl -X POST http://localhost:8080/api/v1/process \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Summarize this report",
    "request_type": "chat",
    "context": {
      "provider": "openai",
      "strict_provider": true
    },
    "user": {"email":"[email protected]","role":"analyst"},
    "client": {"id":"analytics-app","tenant_id":"tenant-1"}
  }'

Provider Types and Availability​

GET /api/v1/llm-provider-types (listLLMProviderTypes) tells you what factories the running build can configure. This is the key detail for edition accuracy: some provider types exist only when the build and license support them.

That means teams assessing production architecture can use this endpoint to verify what the current deployment actually supports, rather than relying on assumptions from older docs or past installs.

Create and Update Rules​

Verified create requirements:

  • name is required
  • type is required
  • unsupported provider types return VALIDATION_ERROR
  • duplicate names return CONFLICT
  • license-gated provider registration can return LICENSE_ERROR

Minimal example:

curl -X POST http://localhost:8080/api/v1/llm-providers \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'client-id:client-secret' | base64)" \
  -d '{
    "name": "azure-prod",
    "type": "azure-openai",
    "endpoint": "https://example.openai.azure.com",
    "model": "gpt-4o",
    "enabled": true,
    "priority": 1,
    "weight": 100,
    "settings": {
      "deployment_name": "gpt-4o-prod",
      "api_version": "2024-02-15-preview"
    }
  }'

Two credential details worth knowing before automating provider CRUD: api_key is accepted on create and update but never returned in responses (reads expose a has_api_key boolean instead), and api_key_secret_arn is available as an alternate secret source.

Routing Configuration​

The routing API is intentionally simple:

{
  "weights": {
    "openai": 70,
    "anthropic": 30
  }
}

Operationally, teams usually combine these routes in three phases:

  1. Start with one provider and confirm connectivity with /api/v1/llm-providers/{name}/test.
  2. Add a second provider and use /api/v1/llm-providers/status plus routing weights for failover or distribution.
  3. Push request-level overrides via /api/v1/process when specific workflows need hard provider pinning for compliance or quality reasons.

That makes it easy to implement provider failover and weighted routing without having to rewrite client applications. In practice, this becomes much more valuable as teams scale from one provider in community deployments to multi-provider and cost-aware routing in evaluation or enterprise programs.

Legacy Runtime Provider Controls​

Older operational tooling may still call the compact runtime provider endpoints:

Use the newer /api/v1/llm-providers/* endpoints for provider CRUD, provider health checks, and persistent routing configuration. Use these compatibility routes only when you need the legacy runtime shape or have existing automation built around it.

Behavioral notes for the legacy routes:

  • The status response is a JSON object keyed by provider name. If the runtime router is not initialized, the endpoint returns an empty object with HTTP 200.
  • The weight update accepts decimal weights (unlike the newer routing API): each weight must be between 0 and 1.0, and the total must not exceed 1.0. Weight violations return 400; an uninitialized runtime router returns 503 with LLM router not initialized.

Operational Readiness Checklist​

Before relying on this page in a production rollout, pair it with the core operations docs:

Last updated on