Skip to main content

Gateway Mode - Lowest Latency AI Governance

Gateway Mode provides the lowest latency governance by letting you make direct LLM calls while AxonFlow handles policy enforcement and audit logging separately.

Policy Support

Gateway Mode evaluates system policies (PII detection, SQL injection blocking, dangerous query prevention) on every request. Custom tenant policies created in the Customer Portal UI or via the Orchestrator API require Proxy Mode.

System policies include: SQL injection detection, PII detection (SSN, credit cards, Aadhaar, PAN, email, phone), dangerous query blocking, and admin access controls.

Note: MCP connectors support both system and tenant policies regardless of mode. See MCP Policy Enforcement.

Prerequisites​

LanguageMinimum VersionSDK PackageInstall
TypeScriptNode.js 18+@axonflow/sdk v9.0.0npm install @axonflow/sdk
Python3.10+axonflow v9.0.0pip install axonflow
Go1.21+github.com/getaxonflow/axonflow-sdk-go/v9 v9.0.0go get github.com/getaxonflow/axonflow-sdk-go/v9
Java11+com.getaxonflow:axonflow-sdk v9.0.0Add to pom.xml or build.gradle

You also need:

  • A running AxonFlow Agent (local Docker or SaaS endpoint)
  • AXONFLOW_CLIENT_ID and AXONFLOW_CLIENT_SECRET
  • Your own LLM provider API key (e.g., OPENAI_API_KEY)

Why Gateway Mode?​

BenefitDescription
Lowest LatencyDirect LLM calls with only a lightweight pre-check before the provider request
Your LLM KeysUse your own API keys and accounts
Full ControlChoose any provider, model, or configuration
Complete Audit TrailEvery LLM call logged with context
Cost TrackingAutomatic token usage and cost estimation

How It Works​

  1. Your app calls getPolicyApprovedContext() or preCheck()
  2. AxonFlow evaluates policies and returns approval
  3. If approved, you make your LLM call directly
  4. Call auditLLMCall() to log the audit trail

The pre-check governs the request content and is connector-agnostic — it does not apply the per-connector controls (the static-policy connector allowlist or per-connector rate limits and budgets) that belong to MCP governance. Wrap the tool calls your model initiates with the MCP check-input / check-output endpoints where connector-scoped enforcement matters.

Quick Start​

TypeScript​

import { AxonFlow } from '@axonflow/sdk';  // v5.1.0+
import OpenAI from 'openai';

const axonflow = new AxonFlow({
  endpoint: process.env.AXONFLOW_ENDPOINT,
  clientId: process.env.AXONFLOW_CLIENT_ID,
  clientSecret: process.env.AXONFLOW_CLIENT_SECRET
});
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function queryWithGovernance(userToken: string, query: string) {
  // 1. Pre-check: Get policy-approved context
  const ctx = await axonflow.getPolicyApprovedContext({
    userToken,
    query,
    dataSources: ['postgres']
  });

  if (!ctx.approved) {
    throw new Error(`Query blocked: ${ctx.blockReason}`);
  }

  // 2. Make LLM call with approved data
  const startTime = Date.now();
  const response = await openai.chat.completions.create({
    model: 'gpt-4',
    messages: [{ role: 'user', content: query }]
  });
  const latencyMs = Date.now() - startTime;

  // 3. Audit the call
  await axonflow.auditLLMCall({
    contextId: ctx.contextId,
    responseSummary: response.choices[0].message.content?.substring(0, 100) || '',
    provider: 'openai',
    model: 'gpt-4',
    tokenUsage: {
      promptTokens: response.usage?.prompt_tokens || 0,
      completionTokens: response.usage?.completion_tokens || 0,
      totalTokens: response.usage?.total_tokens || 0
    },
    latencyMs
  });

  return response.choices[0].message.content;
}

Go​

import (
    "github.com/getaxonflow/axonflow-sdk-go/v9"  // v6.0.0+
    "time"
)

client := axonflow.NewClient(axonflow.AxonFlowConfig{
    Endpoint:     os.Getenv("AXONFLOW_ENDPOINT"),
    ClientID:     os.Getenv("AXONFLOW_CLIENT_ID"),
    ClientSecret: os.Getenv("AXONFLOW_CLIENT_SECRET"),
})

// 1. Pre-check
ctx, err := client.GetPolicyApprovedContext(
    userToken,
    query,
    []string{"postgres"},
    nil,
)
if err != nil {
    return err
}
if !ctx.Approved {
    return fmt.Errorf("blocked: %s", ctx.BlockReason)
}

// 2. Make LLM call (your code)
startTime := time.Now()
llmResponse, err := openaiClient.CreateChatCompletion(...)
latencyMs := time.Since(startTime).Milliseconds()

// 3. Audit
_, err = client.AuditLLMCall(
    ctx.ContextID,
    llmResponse.Choices[0].Message.Content[:100],
    "openai",
    "gpt-4",
    axonflow.TokenUsage{
        PromptTokens:     llmResponse.Usage.PromptTokens,
        CompletionTokens: llmResponse.Usage.CompletionTokens,
        TotalTokens:      llmResponse.Usage.TotalTokens,
    },
    latencyMs,
    nil,
)

Python​

from axonflow import AxonFlow, TokenUsage  # v6.0.0+
from openai import AsyncOpenAI
import time

openai = AsyncOpenAI()

async with AxonFlow(
    endpoint=os.environ.get("AXONFLOW_ENDPOINT", "http://localhost:8080"),
    client_id=os.environ["AXONFLOW_CLIENT_ID"],
    client_secret=os.environ["AXONFLOW_CLIENT_SECRET"]
) as client:
    # 1. Pre-check
    ctx = await client.get_policy_approved_context(
        user_token="user-jwt",
        query="Analyze customer data",
        data_sources=["postgres"]
    )

    if not ctx.approved:
        raise Exception(f"Blocked: {ctx.block_reason}")

    # 2. Direct LLM call
    start = time.time()
    response = await openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": query}]
    )
    latency_ms = int((time.time() - start) * 1000)

    # 3. Audit
    await client.audit_llm_call(
        context_id=ctx.context_id,
        response_summary=response.choices[0].message.content[:100],
        provider="openai",
        model="gpt-4",
        token_usage=TokenUsage(
            prompt_tokens=response.usage.prompt_tokens,
            completion_tokens=response.usage.completion_tokens,
            total_tokens=response.usage.total_tokens
        ),
        latency_ms=latency_ms
    )

Java​

import com.getaxonflow.sdk.AxonFlow;  // v5.1.0+
import com.getaxonflow.sdk.AxonFlowConfig;
import com.getaxonflow.sdk.types.PolicyApprovalResult;
import com.getaxonflow.sdk.types.TokenUsage;
import com.openai.OpenAIClient;

AxonFlow axonflow = AxonFlow.create(AxonFlowConfig.builder()
    .endpoint(System.getenv("AXONFLOW_ENDPOINT"))
    .clientId(System.getenv("AXONFLOW_CLIENT_ID"))
    .clientSecret(System.getenv("AXONFLOW_CLIENT_SECRET"))
    .build());

OpenAIClient openai = new OpenAIClient(System.getenv("OPENAI_API_KEY"));

// 1. Pre-check
PolicyApprovalResult ctx = axonflow.getPolicyApprovedContext(
    PolicyApprovalRequest.builder()
        .userToken("user-123")
        .query("Analyze customer data")
        .dataSources(List.of("postgres"))
        .build()
);

if (!ctx.isApproved()) {
    throw new RuntimeException("Blocked: " + ctx.getBlockReason());
}

// 2. Direct LLM call
long startTime = System.currentTimeMillis();
ChatCompletion response = openai.chat().completions().create(
    ChatCompletionRequest.builder()
        .model("gpt-4")
        .messages(List.of(
            ChatMessage.user("Analyze customer data")
        ))
        .build()
);
long latencyMs = System.currentTimeMillis() - startTime;

// 3. Audit
axonflow.auditLLMCall(
    AuditOptions.builder()
        .contextId(ctx.getContextId())
        .responseSummary(response.choices().get(0).message().content().substring(0, 100))
        .provider("openai")
        .model("gpt-4")
        .tokenUsage(TokenUsage.of(
            response.usage().promptTokens(),
            response.usage().completionTokens()
        ))
        .latencyMs(latencyMs)
        .build()
);

Pre-Check Response Fields​

The getPolicyApprovedContext() / preCheck() call returns an object with the following fields:

FieldTypeDescription
contextIdstringUnique identifier to correlate pre-check with audit. Pass this to auditLLMCall().
approvedbooleantrue if the request passed all policy checks.
approvedDataobjectFiltered/sanitized data safe to send to the LLM. May differ from original if policies modified content.
policiesstring[]List of policy names that were evaluated during the pre-check.
expiresAtDate / time.TimeWhen this approval expires. Make your LLM call before this time.
blockReasonstring (optional)Human-readable reason the request was blocked. Only present when approved is false.
rateLimitInfoobject (optional)Rate limit status: limit (max requests), remaining (requests left), resetAt (reset time).

Error Handling​

TypeScript​

try {
  const ctx = await axonflow.getPolicyApprovedContext({
    userToken: 'user-123',
    query: prompt,
  });

  if (!ctx.approved) {
    // Policy blocked the request -- do not make LLM call
    console.log('Blocked:', ctx.blockReason);
    return;
  }

  const start = Date.now();
  const response = await openai.chat.completions.create({
    model: 'gpt-4',
    messages: [{ role: 'user', content: prompt }],
  });

  await axonflow.auditLLMCall({
    contextId: ctx.contextId,
    responseSummary: response.choices[0].message.content?.substring(0, 100) || '',
    provider: 'openai',
    model: 'gpt-4',
    tokenUsage: {
      promptTokens: response.usage?.prompt_tokens || 0,
      completionTokens: response.usage?.completion_tokens || 0,
      totalTokens: response.usage?.total_tokens || 0,
    },
    latencyMs: Date.now() - start,
  });
} catch (error) {
  if (error.code === 'ECONNREFUSED') {
    console.error('Cannot reach AxonFlow Agent - check endpoint');
  } else if (error.code === 'TIMEOUT') {
    console.error('Pre-check timed out');
  } else if (error.status === 401) {
    console.error('Authentication failed - check credentials');
  } else {
    console.error('Unexpected error:', error.message);
  }
}

Go​

ctx, err := client.GetPolicyApprovedContext(userToken, query, dataSources, nil)
if err != nil {
    // Network error, timeout, or auth failure on pre-check
    log.Printf("Pre-check failed: %v", err)
    return
}

if !ctx.Approved {
    log.Printf("Blocked: %s", ctx.BlockReason)
    return
}

// Make LLM call...
start := time.Now()
llmResp, err := openaiClient.CreateChatCompletion(context.Background(), req)
if err != nil {
    log.Printf("LLM call failed: %v", err)
    return
}

// Audit -- log errors but do not fail the request
_, auditErr := client.AuditLLMCall(
    ctx.ContextID, summary, "openai", "gpt-4", tokenUsage,
    time.Since(start).Milliseconds(), nil,
)
if auditErr != nil {
    log.Printf("Warning: audit logging failed: %v", auditErr)
}

When to Use Gateway Mode​

Choose Gateway Mode when:

  • You need the lowest possible latency
  • You want to use your own LLM API keys
  • Your compliance allows client-side LLM calls
  • You're integrating with existing LLM workflows

Choose Proxy Mode when:

  • You need response filtering (PII detection)
  • You require centralized audit capture before the application receives the response
  • You want automatic LLM failover
  • Compliance requires all calls through a proxy

Enterprise Operating Guidance​

Gateway Mode is often the right enterprise pattern when application teams want to keep direct control of LLM calls while platform teams still need centralized governance, audit, and operating discipline.

What Gateway Mode Means In Enterprise​

In Gateway Mode:

  1. your application asks AxonFlow for a pre-check or approved context
  2. your application calls the LLM provider directly
  3. your application reports the audited outcome back to AxonFlow

That gives enterprises a useful compromise:

  • application teams keep provider-level control
  • platform teams still get governed policy checks and audit records
  • migration from existing LLM integrations is usually faster than a full Proxy Mode cutover

When Gateway Mode Is The Best Enterprise Choice​

Gateway Mode is usually the right fit when:

  • a product team already has mature OpenAI, Anthropic, Bedrock, Azure OpenAI, or private-provider integrations
  • the workload is latency-sensitive and full mediation is hard to justify
  • teams want to add governance without rewriting the whole application path
  • the platform team wants to standardize audit and policy checks before centralizing provider execution

Common examples include internal copilots that already call provider SDKs directly, customer-support assistants that need policy approval before sensitive prompts are sent, and financial or healthcare workflows where application teams own downstream provider logic but governance must be standardized.

Enterprise Responsibilities​

Gateway Mode works best when teams are clear about ownership.

ResponsibilityTypical owner
SDK integration and direct provider callApplication team
Policy baseline and approval posturePlatform or governance team
Provider lifecycle and routing defaultsPlatform team
Audit review and evidence exportPlatform, security, or compliance operations

If those responsibilities are blurred, Gateway Mode can create the wrong kind of freedom: teams move fast, but audit quality and governance consistency drift over time.

Deployment Mode Differences​

Gateway Mode behaves differently operationally depending on deployment posture.

Deployment modeEnterprise implication
SaaSGood for team-level adoption where tenant isolation is the primary concern and the platform team does not need broad node visibility
In-VPCBetter for organizations that want stronger network control, broader operations visibility, and tighter internal platform ownership
Self-hostedUseful when the customer wants to own infrastructure, telemetry, provider networking, and upgrade posture directly

Banks, healthcare organizations, and larger platform teams often choose In-VPC or self-hosted postures for the highest-value Gateway Mode workloads because those modes give them stronger control over provider networking, secrets, telemetry, and internal audit posture.

Provider And Connector Planning​

Gateway Mode is easiest to run well when provider and connector choices are deliberate.

For providers, decide:

  • which providers are available to each tenant or business unit
  • how failover or routing is handled operationally
  • how cost metadata, provider health, and credential rotation are maintained

For connectors, coordinate Gateway Mode with connector governance so data retrieval and policy review evolve together. Gateway Mode pre-checks are connector-agnostic; per-connector allowlists, rate limits, budgets, and response scanning belong to the MCP connector path. Use MCP Policy Enforcement when tool access or connector-scoped controls matter.

  1. Start with one high-value workload that already makes direct provider calls and has a clear risk or compliance reason to add governance.
  2. Standardize audit first: make the pre-check and audit paths mandatory, observable, and owned by the right team.
  3. Centralize provider operations as much as the deployment allows, even if applications still call providers directly.
  4. Review graduation paths. Some workloads should stay in Gateway Mode; others should move to Proxy Mode or WCP once governance needs deepen.

Enterprise Review Checklist​

  • Is the workload latency-sensitive enough to justify Gateway Mode?
  • Does the application team have a reliable audit reporting path?
  • Are provider credentials and runtime decisions centrally managed?
  • Are governance owners comfortable with the pre-check-plus-audit model?
  • Does the workload need richer approval, replay, response mediation, or workflow lifecycle controls?

Configuration​

const axonflow = new AxonFlow({
  clientId: process.env.AXONFLOW_CLIENT_ID,
  clientSecret: process.env.AXONFLOW_CLIENT_SECRET,
  endpoint: process.env.AXONFLOW_ENDPOINT,
  mode: 'production'
});

API Reference​

Pre-Check Methods​

getPolicyApprovedContext() / preCheck()​

Both methods are equivalent - preCheck() is an alias for simpler API.

TypeScript:

const ctx = await axonflow.getPolicyApprovedContext({
  userToken: string,      // User's JWT token
  query: string,          // The query/prompt
  dataSources?: string[], // Data sources being accessed
  context?: object        // Additional context for policy evaluation
});

Go:

ctx, err := client.GetPolicyApprovedContext(
    userToken string,
    query string,
    dataSources []string,
    context map[string]interface{},
)

Python:

ctx = await client.get_policy_approved_context(
    user_token="...",
    query="...",
    data_sources=["..."],
)

Response:

{
  contextId: string;      // Unique ID to correlate with audit
  approved: boolean;      // Whether request was approved
  approvedData: object;   // Filtered data (if policies modified it)
  policies: string[];     // Policies that were evaluated
  expiresAt: Date;        // When this approval expires
  blockReason?: string;   // Reason for blocking (if !approved)
  rateLimitInfo?: {       // Rate limit info (if applicable)
    limit: number;
    remaining: number;
    resetAt: Date;
  }
}

Audit Method​

auditLLMCall()​

TypeScript:

await axonflow.auditLLMCall({
  contextId: string,        // From pre-check response
  responseSummary: string,  // Summary of LLM response
  provider: string,         // e.g., "openai", "anthropic"
  model: string,            // e.g., "gpt-4", "claude-opus-4"
  tokenUsage: {
    promptTokens: number,
    completionTokens: number,
    totalTokens: number
  },
  latencyMs: number,        // Call duration in milliseconds
  metadata?: object         // Optional additional metadata
});

Go:

result, err := client.AuditLLMCall(
    contextId string,
    responseSummary string,
    provider string,
    model string,
    tokenUsage TokenUsage,
    latencyMs int64,
    metadata map[string]interface{},
)

Python:

await client.audit_llm_call(
    context_id="...",
    response_summary="...",
    provider="openai",
    model="gpt-4",
    token_usage=TokenUsage(...),
    latency_ms=250,
)

auditToolCall()​

Record non-LLM tool calls (API calls, MCP executions, function invocations) in the audit trail. Useful when external orchestrators like LangGraph or CrewAI invoke tools outside of AxonFlow's proxy mode.

TypeScript:

const result = await axonflow.auditToolCall({
  toolName: "weather-api",     // Required: tool name
  callerName: "my-service",    // Identifies the calling client (see note below); toolType is deprecated
  input: { city: "SF" },       // Optional: tool input
  output: { temp: 18 },        // Optional: tool output
  durationMs: 245,             // Optional: call duration
  success: true,               // Optional: whether call succeeded
  policiesApplied: ["..."],    // Optional: policies evaluated
});
// result.auditId, result.status, result.timestamp

Go:

result, err := client.AuditToolCall(ctx, axonflow.AuditToolCallRequest{
    ToolName:   "weather-api",
    CallerName: "my-service", // Identifies the calling client (see note below); ToolType is deprecated
    Input:      map[string]interface{}{"city": "SF"},
    Output:     map[string]interface{}{"temp": 18},
    DurationMs: 245,
    Success:    boolPtr(true),
})

Python:

from axonflow.types import AuditToolCallRequest

result = await client.audit_tool_call(
    AuditToolCallRequest(
        tool_name="weather-api",
        caller_name="my-service",  # Identifies the calling client (see note below); tool_type is deprecated
        input={"city": "SF"},
        output={"temp": 18},
        duration_ms=245,
        success=True,
    )
)

Java:

AuditToolCallResponse result = client.auditToolCall(
    AuditToolCallRequest.builder()
        .toolName("weather-api")
        .callerName("my-service") // Identifies the calling client (see note below); toolType is deprecated
        .input(Map.of("city", "SF"))
        .output(Map.of("temp", 18))
        .durationMs(245L)
        .success(true)
        .build()
);
toolType is deprecated

toolType / tool_type is the legacy way of tagging which client made the call. On platform v9.11.0+ this is recorded as caller_name — the calling client or integration (for example claude_code, codex, cursor, openclaw) — where the /api/v1/audit/tool-call handler folds a legacy tool_type into caller_name when no caller_name is sent, and records "unknown" when neither is supplied. A dedicated callerName / caller_name request field has landed in SDK v9.0.0 (go / python / typescript / java) — prefer it over toolType. On platforms below v9.11.0 caller_name is accepted but ignored, so keep passing toolType there.

Audit Reliability​

Gateway Mode audit logging provides high reliability through a multi-layer architecture:

Compliance Mode (Default)​

In compliance mode, audit operations are synchronous:

  • Pre-check context is written to database before response returns
  • LLM call audits are written before acknowledgment
  • Gives the caller a stronger persistence signal than asynchronous audit writes
  • Useful when an AI governance or compliance review requires evidence before the request is acknowledged

Performance Mode (Optional)​

For high-throughput systems, enable performance mode:

  • Audit operations are queued and processed asynchronously
  • ~99.9%+ audit reliability with disk fallback
  • Lower request overhead than full synchronous audit writes

Automatic Retry & Recovery​

All audit operations include:

  1. Exponential Backoff Retry - 3 attempts (100ms, 200ms, 400ms delays)
  2. Disk Fallback - If database unavailable, entries written to fallback file
  3. Automatic Recovery - On restart, entries from fallback file are replayed to database

This reduces the chance of losing audit evidence during transient database outages and gives teams a recovery path after restart.

Audit Entry → Queue → Retry (3x) → ✅ DB Success
                              └─→ ❌ Fallback File → Recovered on Restart

Configuration​

# Enable performance mode (optional)
AGENT_PERFORMANCE_MODE=true

# Fallback file location
AUDIT_FALLBACK_PATH=/var/lib/axonflow/audit/audit_fallback.jsonl

Cost Tracking​

AxonFlow automatically estimates costs based on provider and model:

ProviderModelEstimated Cost
OpenAIgpt-4$0.03/1K tokens
OpenAIgpt-4o-mini$0.002/1K tokens
Anthropicclaude-sonnet-4$0.003/1K tokens
BedrockVariousBased on AWS pricing
OllamaLocal$0 (self-hosted)

Comparison with Proxy Mode​

FeatureProxy ModeGateway Mode
Integration EffortMinimalModerate
Latency OverheadHigherLower
System Policies (PII, SQL injection)✅ Yes✅ Yes
Tenant Policies (custom rules)✅ Yes❌ No
Response FilteringYesNo
Audit Coverage100% automaticManual (call audit API)
LLM ControlLimitedFull
Best ForSimple apps, custom policiesFrameworks, performance

See Choosing a Mode for detailed guidance.

Next Steps​

Enterprise customers: Use the Enterprise Operating Guidance section on this page with Enterprise Architecture for rollout planning.

Last updated on