Skip to main content

Ollama Setup

Ollama is available in AxonFlow Community and Enterprise and is the default choice for local, self-hosted, and air-gapped model execution.

Runtime Defaults​

AxonFlow's Ollama configuration defaults to:

  • Endpoint: http://localhost:11434
  • Model: llama3.2:latest

Quick Start​

Start Ollama​

docker run -d -p 11434:11434 --name ollama ollama/ollama
docker exec ollama ollama pull llama3.2:latest

Configure AxonFlow​

export OLLAMA_ENDPOINT=http://localhost:11434
export OLLAMA_MODEL=llama3.2:latest
export OLLAMA_TIMEOUT_SECONDS=300    # Request timeout in seconds (default: 300)
info

The default timeout for Ollama is 300 seconds (5 minutes), intentionally higher than the 120-second default for cloud providers. Local models can be slower to load on first inference.

YAML Configuration​

version: "1.0"

llm_providers:
  ollama:
    enabled: true
    config:
      endpoint: ${OLLAMA_ENDPOINT:-http://localhost:11434}
      model: ${OLLAMA_MODEL:-llama3.2:latest}

Enterprise Portal-Managed Setup​

In Enterprise deployments, Ollama can also be represented as a protected provider record. Use this when a central platform team wants to route a share of production traffic to private model infrastructure and manage that routing through the Customer Portal provider APIs.

curl -X POST https://portal.example.com/api/v1/llm-providers \
  --cookie "axonflow_session=$AXONFLOW_SESSION" \
  -H "Content-Type: application/json" \
  -d '{
    "provider_name": "ollama",
    "display_name": "Ollama Private Cluster",
    "config": {
      "endpoint": "http://ollama.internal:11434",
      "model": "llama3.2:latest"
    },
    "priority": 6,
    "weight": 0.4,
    "enabled": true
  }'

The protected provider adapter defaults to http://localhost:11434 and llama3.2 if the endpoint or model is omitted. For production, set both explicitly so routing decisions, audit records, and incident reviews point to the intended private inference service.

Good Fits​

  • Local development without external provider keys
  • On-prem or air-gapped deployments
  • Teams that want predictable spend and infrastructure-level control
  • Enterprises that need data sovereignty, private AI infrastructure, or internal developer platforms built on local models

Proxy Mode​

import { AxonFlow } from '@axonflow/sdk';

const axonflow = new AxonFlow({
  endpoint: 'http://localhost:8080',
  clientId: process.env.AXONFLOW_CLIENT_ID,
  clientSecret: process.env.AXONFLOW_CLIENT_SECRET,
});

const response = await axonflow.proxyLLMCall({
  userToken: 'user-123',
  query: 'Explain the design tradeoffs in this distributed system.',
  requestType: 'chat',
  context: {
    provider: 'ollama',
    model: 'llama3.2:latest',
  },
});

console.log(response.data);

Operational Notes​

  • Ollama is a strong option for community deployments that cannot send prompts to a third-party cloud provider.
  • Proxy Mode is usually the simplest starting point.
  • Gateway Mode is also possible if your application already uses the Ollama HTTP API directly and you want AxonFlow only for pre-check and audit.

Gateway Mode Example​

import { AxonFlow } from '@axonflow/sdk';
import ollama from 'ollama';

const axonflow = new AxonFlow({
  endpoint: process.env.AXONFLOW_ENDPOINT,
  clientId: process.env.AXONFLOW_CLIENT_ID,
  clientSecret: process.env.AXONFLOW_CLIENT_SECRET,
});

// 1. Pre-check
const ctx = await axonflow.getPolicyApprovedContext({
  userToken: 'user-123',
  query: prompt,
});

if (!ctx.approved) throw new Error(ctx.blockReason);

// 2. Your direct Ollama call
const start = Date.now();
const response = await ollama.chat({
  model: 'llama3.2',
  messages: [{ role: 'user', content: prompt }],
});

// 3. Audit
await axonflow.auditLLMCall({
  contextId: ctx.contextId,
  responseSummary: response.message.content.substring(0, 100),
  provider: 'ollama',
  model: 'llama3.2',
  tokenUsage: { promptTokens: 0, completionTokens: 0, totalTokens: 0 },
  latencyMs: Date.now() - start,
});

Health Check​

You can verify the local Ollama instance directly:

curl http://localhost:11434/api/tags

The protected AxonFlow provider adapter infers health from successful completions; it does not perform a dedicated GET /api/tags check on its own. Use GET /api/tags as an external liveness check in your platform monitoring.

Enterprise Deployment Patterns​

For early rollout, a single internal Ollama service is usually enough: one endpoint, one or two approved models, and low operational complexity.

For multi-team enterprise use, treat Ollama as a private model-serving estate:

  • run a dedicated internal endpoint or cluster
  • plan GPU capacity, model warm-up, and concurrency limits
  • restrict network exposure and service identity
  • define fallback routing to a managed cloud provider for degraded private capacity

Common routing patterns are:

  • ollama primary, anthropic secondary
  • bedrock primary, ollama for internal-only workloads
  • ollama for bulk internal tasks, commercial providers for premium user flows

Troubleshooting​

Connection errors​

  • Confirm Ollama is listening on the same host or network as AxonFlow.
  • Verify the OLLAMA_ENDPOINT value includes the correct port.

Slow responses​

  • Start with a smaller local model.
  • Check whether inference is CPU-only or GPU-backed.
  • Measure end-to-end latency in your application path before assuming the governance layer is the bottleneck.

Next Steps​

Provider Rollout Checklist​

Before this provider path becomes production traffic, connect it to the runtime operating model:

Last updated on