Skip to main content

v9.12 → v9.13 Upgrade Guide

v9.13.0 is the output of a cross-tenant remediation programme. Most of it needs nothing from you: principals and tenancy are now bound from the validated credential instead of the request body, the orchestrator API gained authentication, and nine SSRF egress classifiers were unified. Those land on restart.

Four things do need a decision before you pull the image, and this guide is about those four. Two of them are irreversible in the sense that matters: a migration disables policy rows whose identifiers you cannot recover afterwards, and another stamps rows in a way its own down migration deliberately leaves behind.

Start here: run the preflight​

Do not work through this guide by hand. Every question below is answered, against your own running stack, by a script that already ships with the platform:

scripts/deployment/v9_self_hosted_preflight.sh

Self-hosted partners running the install bundle have the same script as ./preflight.sh in the bundle root — it is a byte-identical copy, kept in lockstep by a CI drift-guard.

DATABASE_URL="postgres://axonflow:<password>@localhost:5432/axonflow?sslmode=disable" \
  ./scripts/deployment/v9_self_hosted_preflight.sh

Three properties are worth stating explicitly, because they are what make it safe to run on a production stack before a maintenance window:

  • It runs against the OLD stack. It never needs the new image, never writes, and never restarts anything. Every statement it issues is a SELECT, a SHOW or an EXPLAIN.

  • It cannot report a verdict it did not reach. If a query fails — a dropped connection, a permission denied — the run FAILs and names the query. An empty result and a failed query are never allowed to look the same. If it cannot read a container's environment it says NOT VERIFIED, never "OK".

  • It runs where the database is. If psql is not installed on the host, point it at the Postgres container instead and it resolves the DSN from inside:

    AXONFLOW_PG_CONTAINER="$(docker compose ps -q postgres)" \
    DATABASE_URL="postgres://axonflow:<password>@localhost:5432/axonflow?sslmode=disable" \
      ./scripts/deployment/v9_self_hosted_preflight.sh

Run it from your Compose project directory so it reads the environment your agent and orchestrator are actually running with rather than the environment of your shell.

Reading the verdict​

Final lineExitMeaning
PASS — ready to upgrade0Every check passed.
PROCEED WITH CAUTION0No blockers, but at least one WARNING. Each is printed with its consequence; read and decide.
DO NOT UPGRADE1At least one check FAILED. Each FAIL names its remediation.
Script error: …2No verdict was reached (no database, no psql transport, a bad argument). Not an all-clear.

--self-test runs the script's own logic with no database at all, which is a quick way to confirm the copy you have is intact.

What changed — the four things that need a decision​

1. DEPLOYMENT_MODE is now validated​

A value the platform does not recognise is no longer widened to the SaaS migration set. Recognised values, matched exactly:

community, community-saas, enterprise, evaluation, in-vpc-banking, in-vpc-enterprise, in-vpc-healthcare, in-vpc-travel, invpc, saas

There is no trimming and no case-folding, so community and Community are both unrecognised. An unset value is unchanged, is still resolved to community for migration selection, and is not fatal.

The two components fail differently, and that asymmetry is the reason the preflight reads both:

ComponentAn unrecognised DEPLOYMENT_MODE
AgentRuns the migration selector. Refuses to boot — the container does not start. Loud.
OrchestratorDoes not run the migration selector. Starts normally, then silently takes the non-Community posture, because the Community check compares against the exact string community.

A stack in that state has an agent that will not start next to an orchestrator that started with the wrong posture. Checking one container is not enough.

Checking by hand is also harder than it looks: docker compose exec axonflow-agent printenv DEPLOYMENT_MODE prints a leading space invisibly. The preflight quotes the value and names the difference — "it is community with leading/trailing whitespace" or "…in the wrong case".

DEPLOYMENT_MODE=enterprise needs no change: it is now an alias for in-vpc-enterprise, and nothing is dropped from a stack that already has the industry tables.

2. core/155 disables policies with an empty tenant_id​

core/155 normalises tenant_id = '' to NULL on both policy tables. On dynamic_policies it also sets enabled = false.

That is deliberate. NULL is the apply-to-all sentinel, so normalising alone would take a row that was enforced for nobody and make it enforced for every tenant — promoting a dormant row, possibly one carrying a block action, to deployment-wide enforcement in the middle of a security patch.

The cost is that the repaired row becomes unreachable through the product. The policy API filters tenant_id = <caller> OR tenant_id = 'global', and NULL matches neither, so there is no portal round-trip available. Remediation is direct SQL — and the migration is the last moment those rows are identifiable, because afterwards no row has an empty tenant any more.

So capture the identifiers first. The preflight prints them and the statement:

UPDATE dynamic_policies
   SET tenant_id = '<your-real-tenant-id>', enabled = true
 WHERE policy_id IN ('<the ids the preflight named>');

static_policies rows are normalised but NOT disabled, and no enforcement changes for them: the static loader's tenant_id = $1 OR tenant_id = 'global' predicate already excluded both '' and NULL for every caller. The preflight reports the two tables separately so they are not confused.

Nothing in the shipped code creates an empty-string tenant — PolicyRepository.Create rejects it and both seeders write 'global' or a real tenant — so on most deployments this check finds nothing.

3. core/156 holds locks on five tables until COMMIT​

core/156 adds NOT NULL plus a blank-string CHECK to org_id and tenant_id on plans, workflows, workflow_checkpoints, execution_summaries and webhook_subscriptions.

Neither operation rewrites a table, so each is a sequential scan rather than a rewrite. But all five ALTERs run in one transaction, so the ACCESS EXCLUSIVE locks are held until COMMIT rather than released per table. Size the window for the sum of the scans, not for the largest table.

Do not guess that number. The preflight prints, per table, the row count and the measured cost of one scan — it runs EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*), which is the same read the ALTER performs — then reports roughly four times the total as an arithmetic lower bound (two constrained columns, two scans each). It is a lower bound rather than an estimate because ACCESS EXCLUSIVE additionally waits for every in-flight transaction on those tables.

Behaviour change. A row with a blank tenancy key was previously readable by every tenant — that is the vulnerability being closed. Every such row is stamped with the inert sentinel __axonflow_unowned__ and becomes reachable and writable by nobody. The migration does not attempt to determine an owner, so no row is recovered to a real tenant.

The practical consequence: an execution that starts before the upgrade and finishes after it will fail its final update instead of being marked completed. That window is the upgrade itself. Drain in-flight executions if you can. The preflight reports both numbers — how many rows you lose access to, and how many blank values the migration's own log will report, which is larger because it logs once per table and column.

budgets is deliberately not constrained: its scope lookup treats an empty organization as apply-to-all, so adding the constraint would silently disable deployment-global spend caps.

4. Cross-origin browser access is denied by default​

With AXONFLOW_CORS_ALLOWED_ORIGINS unset and DEPLOYMENT_MODE anything other than exactly community, the agent and orchestrator deny every cross-origin browser request.

Note the trap: for CORS, an unset DEPLOYMENT_MODE is not the Community posture. The migration selector resolves unset to community; the Community check compares the raw environment value against community and an unset value is not equal to it. Unset therefore lands in the deny branch.

Who is affected: only a browser front-end served from a different origin that calls the AxonFlow API directly. The bundled Customer Portal is same-origin and is unaffected; SDKs, curl and CI ignore CORS entirely.

This one is hard to notice after the fact, which is why the preflight WARNs about it in advance: a blocked preflight request produces no server-side error — the browser blocks it — and the only trace is a single line at startup.

If you do have such a front-end, set the exact origins (scheme + host + optional port, comma-separated). There is no suffix matching: https://example.com does not cover https://app.example.com.

An entry containing * behaves differently from what older documentation claimed: it is honoured, by prefix/suffix match rather than exactly, but credentials are then not advertised for any origin in the list. A front-end relying on cookies will break. The preflight WARNs when it sees one.

Before you upgrade: take a backup​

Both migrations mutate data, and core/156's down migration deliberately leaves its sentinel stamps in place. Rolling the schema back does not undo the data change, and re-pinning the previous image digests rolls the images back exactly while restoring neither a disabled policy nor an un-stamped tenancy key.

A database snapshot is the only rollback for those. The preflight WARNs when it cannot find evidence of one; set RDS_INSTANCE_IDENTIFIER (AWS RDS) or PG_BACKUP_TOOL so it can check rather than guess.

What you need to do — by consumer type​

SDK / plugin / HTTP API enforcement consumers​

Nothing, with one condition: your calls must carry a valid credential that resolves both X-Org-ID and X-Tenant-ID. Callers going through the AxonFlow Agent or the Customer Portal already do — both stamp those headers from the validated credential.

If you call POST /api/v1/process, /api/v1/workflows/execute, /api/v1/plan or /api/v1/plan/execute directly, they now return 401 without both headers, and 403 when the request body names a different tenancy than the headers do (previously the body value was silently corrected).

Self-hosted operators​

Run the preflight, resolve every FAIL, read every WARNING, take a snapshot, then upgrade. That is the whole procedure.

Anyone running MCP tool governance​

content policies now govern MCP tool calls where the dynamic plane is enabled (MCP_DYNAMIC_POLICIES_ENABLED=true). Because content is the default policy type, existing tenant policies of the form {query contains|regex …} + block begin governing MCP tool calls on upgrade, where previously they governed only the LLM, MAP and WCP planes.

Deployments on the default posture (MCP_DYNAMIC_POLICIES_ENABLED=false — both docker-compose and the community-SaaS CloudFormation template) are unaffected until they opt in. Review your tenant policies before enabling: a condition naming a field the MCP evaluator cannot resolve fails to no-match, but a known field with a negated operator can match on an empty value — {user.role not_equals "admin"} is true when the role is unset, which is common on this plane.

Anyone with circuit-breaker notifications or orchestrator webhooks​

Those now refuse internal and reserved IP ranges they previously accepted. If either targets an internal address, re-check it before upgrading. The preflight does not cover this — see What the preflight cannot tell you.

What the preflight cannot tell you​

Written down rather than left to be discovered. The preflight reads the database and the container environment; it does not exercise the running platform, so:

  • Whether a webhook or circuit-breaker notification target is now refused. That needs the new egress range table, which only exists in the new image. Check any internal-addressed target by hand.
  • Whether a direct API caller sends both tenancy headers. The 401/403 change is a property of your callers, not of your deployment, and nothing in the database records it.
  • Whether enabling MCP_DYNAMIC_POLICIES_ENABLED would newly block traffic. Which of your content policies would match an MCP tool call is a question about traffic, not configuration.
  • Anything about a component it could not read. If discovery fails it says NOT VERIFIED and tells you what to run by hand. That is not a pass.
  • Whether your .env matches what the containers are running. It reads the containers, which is deliberate — the running environment is the one that matters — but a .env edited since the last docker compose up will not be reflected until you restart.

Getting help​

If something behaves differently after upgrading, the most useful information for support:

  1. The full preflight output from before the upgrade (it is safe to share — it prints no secrets).
  2. The exact endpoint and HTTP status code you are seeing.
  3. Your DEPLOYMENT_MODE, quoted, on both the agent and the orchestrator.

[email protected] — please mention "v9.13 upgrade" in the subject so it routes correctly.

See also​

Last updated on