v9 → v10 Migration Guide
AxonFlow v10.0.0 is a MAJOR. It removes three fallbacks: one at boot, one on the identity path where a presented-but-invalid per-user token used to be downgraded to a synthetic service identity, and one on the portal's SAML path where a fabricated, per-restart signing keypair stood in for a real one. It makes a credential required on two tools for some organizations and adds a posture that can require one for an entire organization; it adds new fail-closed refusals on eight enforcement planes and two execution-read shapes; and it changes what several read APIs put on the wire when they have nothing to report.
The part of the upgrade that needs a decision before you take it, rather than after, is the database. Two of the eight migrations rewrite audit_logs whole-table, and the migration runner answers a migration error with a fatal exit, so a statement_timeout too tight for them fails the container, which restarts, and fails again. This is the first AxonFlow release in which a migration's runtime scales with your audit history rather than with your schema size.
One-line summary: size your
statement_timeoutagainst your ownaudit_logsrow count before upgrading, setAXONFLOW_DB_PLATFORM_ADMIN_URLif you run the customer portal, removeAXONFLOW_DEBUG_POLICIES, sweep your dashboards for theplane="memory"label, which goes silently empty rather than erroring, check whether any MCP REST caller is relying on an invalid per-user token still being accepted, and confirm every SAML tenant has a working SP signing keypair.
For the full behavior narrative, read the v10.0.0 release notes. This guide covers the operational steps.
The eight migrations
All eight ship to every edition and apply with the usual boot-time runner. From a v9.19.0 baseline, the deploy delta is images plus these eight. core/161 and core/162 need sizing; core/165 is the only one that can change what your deployment ENFORCES; core/165, core/166 and core/167 each hold ACCESS EXCLUSIVE for their duration, so reads are blocked on the tables they touch.
| Migration | What it does | Cost |
|---|---|---|
core/160 | Deletes one seeded dynamic-policy row, high_risk_block, superseded by the tuned near-duplicate sys_dyn_high_risk_block | One row. Instant. A paired down-migration restores it verbatim |
core/161 | Sets audit_logs.response_time_ms = NULL on every historical row holding a fabricated 0 | One sequential scan of audit_logs, plus one new row version per matched row |
core/162 | Sets audit_logs.tokens_used and audit_logs.cost to NULL on every historical row where both coalesce to zero, at least one of the two is not already NULL, and the row carries no provider and no model | One sequential scan of audit_logs, plus one new row version per matched row |
core/163 | Adds organizations.require_user_token BOOLEAN NOT NULL DEFAULT false | Additive. No backfill, no rows rewritten. A paired down-migration drops the column |
core/164 | Widens organizations.license_key from VARCHAR(512) to TEXT, so customer onboarding stops failing on long licence keys | Metadata-only ALTER. No table rewrite and no rows read |
core/165 | Makes org_id NOT NULL and non-empty on static_policies, dynamic_policies and policy_overrides, because it is now the only column that selects a policy row | ACCESS EXCLUSIVE on three config-scale tables, no rewrite. A policy row it cannot resolve an organisation for stops being selectable |
core/166 | Drops the legacy organization_id column from the same three tables | Metadata-only. No deployment data depends on it |
core/167 | Adds hitl_approval_queue.consumed_at, two partial indexes and a widened history CHECK | One transaction. ADD COLUMN takes ACCESS EXCLUSIVE first and Postgres never downgrades a lock, so reads on hitl_approval_queue and hitl_approval_history are blocked until it commits. Both indexes build without CONCURRENTLY, which cannot run inside a transaction block |
core/160: one seeded policy row is deleted
A 2025 migration seeded high_risk_block. A later migration seeded sys_dyn_high_risk_block, a near-duplicate under the same name, description, conditions, priority and tenant, as part of a structured system-policy pass. A migration after that downgraded the action from block to warn, and it tuned only the newer row. The older duplicate has stayed at block the entire time.
Until v10.0.0 that duplication was latent rather than harmful, because risk_score resolved only from the caller's own request body and the condition was unreachable in practice. With risk_score computed by the platform again, the never-tuned duplicate would have started blocking production traffic on upgrade while its intentionally-tuned twin sat at warn right beside it. core/160 deletes the duplicate, not the tuned row.
The threshold this policy pair enforces lives in the surviving row's conditions JSON and is tunable through the policy API and the portal like any other dynamic policy. It is not the dynamic_policies.risk_threshold column both rows also carry, which is read by nothing in the platform.
core/161 and core/162: whole-table backfills of audit_logs
Both are one statement in one transaction, so each takes a ROW EXCLUSIVE lock on audit_logs for its duration and writes a new row version for every matched row. Readers are not blocked. Expect bloat proportional to the match count until autovacuum catches up.
Both are bounded to timestamp < NOW() and pin SET LOCAL TimeZone = 'UTC' so that bound means the same thing on every deployment. "Whole-table" is the description before the bound; the bound exists for correctness, not for speed.
Expect a sequential scan anyway. A timestamp index does exist and timestamp is a predicate column in both statements, but timestamp < NOW() selects essentially the whole table, so the planner sequential-scans regardless. The remaining predicate columns carry no index at all.
Do not assume 162 is the smaller one on your data. On the seeded pre-release history the platform measured, 161 matched 140 rows and 162 matched 100 of those same rows, with none matching 162 alone. The predicates do not imply that relation. 162 never references response_time_ms, so it cannot entail 161's predicate; the narrower true point is that 162's extra guards exclude writers 161 catches, such as the human-review approval writer, which binds a literal 0 into response_time_ms and names neither usage column. 161 was the larger on that measurement and runs first, so it is the natural one to size against, but measure both on your own data rather than taking the ordering on faith.
Both down-migrations are deliberate no-ops, because a fabricated zero is not recoverable from a NULL.
core/163 and core/164: additive schema changes
Neither reads or rewrites a row, so neither enters the statement_timeout sizing above. That sizing remains about core/161 and core/162 alone.
core/163 adds organizations.require_user_token as a BOOLEAN NOT NULL DEFAULT false. The default is load-bearing: every existing deployment is untouched at deploy time, and an organization that never touches the lever keeps its current behavior until someone opts it in, per organization or through the deployment-wide default. There is no backfill. Both the migration and its paired down-migration are guarded on the organizations table existing, so both no-op on a deployment without it. See The require-user-token posture below for what the column controls.
core/164 widens organizations.license_key from VARCHAR(512) to TEXT. The symptom it fixes is worth naming, because it looks like an intermittent onboarding bug: a V2 licence key's length grows with the organization id and the permission grants it encodes, so a key can exceed 512 characters and POST /api/v1/admin/onboard-customer then answers 500 with value too long for type character varying(512), after the key has already passed licence validation. Which side of that cliff a customer lands on depends on the length of their organization name. On PostgreSQL the widening is a metadata-only ALTER, because widening a varchar to text needs no table rewrite and no verification scan. Its down-migration is a deliberate no-op, and for a different reason from 161 and 162: narrowing the column back could truncate a validly stored licence, so the down path declines to do it rather than risk destroying one.
Size your statement_timeout before upgrading
A 161 or 162 that trips a per-session or per-role statement_timeout fails the migration, and a failed migration is a fatal exit. The container restarts and fails again. Size the timeout against your own audit_logs row count before upgrading, or raise it for the migration role for the duration of the upgrade.
The sizing basis, so this is actionable rather than a warning: each migration costs one sequential scan of the whole table, plus one new row version written per matched row, plus the bloat those versions leave until autovacuum catches up. The scan is fixed by your table size and the write cost is fixed by the match count.
All three queries below are read-only and safe to run before the upgrade. Run them as one transaction so the SET LOCAL applies to all three.
SET LOCAL TimeZone = 'UTC';
-- scan size
SELECT count(*) FROM audit_logs;
-- rows migration 161 will rewrite
SELECT count(*) FROM audit_logs
WHERE response_time_ms = 0 AND timestamp < NOW();
-- rows migration 162 will rewrite
SELECT count(*) FROM audit_logs
WHERE COALESCE(tokens_used, 0) = 0 AND COALESCE(cost, 0) = 0
AND (tokens_used IS NOT NULL OR cost IS NOT NULL)
AND (provider IS NULL OR provider = '') AND (model IS NULL OR model = '')
AND timestamp < NOW();
Size the statement_timeout against the larger of the two match counts and against a full-table scan. If audit_logs is large, take an actual timing on a restored copy rather than extrapolating. A statement_timeout that already suits a full-table scan of that table will suit these two.
Do not re-run core/161 by hand after upgrading
core/161's predicate is response_time_ms = 0, which is precisely the shape the new writers produce for a genuine sub-millisecond decision. Before v10.0.0, a zero in that column meant "nothing was measured"; after v10.0.0, it means "measured, and it was under a millisecond". Running the migration again after the new writers are live would erase real measurements.
There is a related rule worth stating once, because it applies to every future release and not only to this one. A migration edited in place after it lands is silently skipped wherever it already applied. The runner's applied-migration lookup keys on the version and name and never compares a checksum, so a database that already recorded a migration will not re-run an edited file. That is harmless for this release, because no edited migration here is in a released tag and every deployment will apply the final text once. The rule to carry forward is: once a migration is in a released tag, correct it with a new migration, never in place.
Per-tenant policy targeting is dropped
Before v10.0.0, which policies applied to a caller was decided by tenant_id - the username half of the Basic-auth credential. Whatever the client typed. Nothing validated it against a directory, a credential record or the licence. So a caller chose the policy set it was governed by, and a username that named no policy was governed by no tenant-tier policy at all.
From v10.0.0 the platform selects on the organisation id from the signed licence, which the caller cannot choose.
Does this affect you?
If your organisation has one tenant - the usual case - no. Exactly the same policies apply before and after. This matters only where one organisation owns several tenants and gave them different policy rows, and only your database can answer that.
./preflight.sh check 23 answers it. It names the rows rather than counting them, because you cannot re-author a number.
What changes where it does apply
Two parts, with opposite directions:
-
Policy rows. A policy authored for one tenant starts applying to every tenant in the same organisation. The direction is over-blocking: a rule applies more broadly, and nothing stops being enforced.
-
Override rows (Enterprise
policy_overrides) do NOT become organisation-wide, deliberately. An override downgrades an action -blocktowarn- or disables a policy outright, so applying one across an organisation would be the only part of this change that LOOSENS enforcement, which is why it is the one part that was not made organisation-wide. Override selection still narrows to your own tenant plus the organisation's own rows, on the reasoning that an override is an exception granted to a caller rather than a policy targeted at one../preflight.shcheck 23 still lists organisations that authored divergent per-tenant overrides, because that is worth knowing before a tenancy change, not because anything about them changes on upgrade.One related fix ships with it: that read never filtered
revoked_at, while the session-override matcher always has, so a revoked override went on being applied. It is filtered now, which makes revoking work the way you would already have assumed it did.
The re-authoring path
For each row check 23 names, before you pull the image:
- A rule that genuinely belongs to a subset of your people becomes segment-scoped. Governance segments are the platform's verified sub-org dimension: membership resolves from your SCIM directory, not from a string the client sends. Set
segment_idon the row and remove the per-tenant intent. Segment authoring has no portal surface yet, so this is a directUPDATEtoday. - A rule whose per-tenant scoping was incidental - two tenants carrying different
tenant_idvalues but wanting the same governance - needs no change. It becomes organisation-wide, which is what it already meant. - Overrides need no action for this change, since they do not become organisation-wide. Check 23 lists divergent ones so you can confirm that is what you intended.
Roll the agent and the orchestrator together
The agent now sends the organisation id on its internal policy-evaluation call, and an orchestrator on this release refuses that call with 403 when it arrives without one. That refusal is deliberate: evaluating without an organisation would silently drop every tenant-authored dynamic policy behind a 200, and a governed tool call must fail closed rather than quietly stop being governed. If you deploy the two services separately, expect refusals for the duration of any window in which they are on different versions.
Policy rows with no organisation key stop firing
core/165 makes the organisation key mandatory, because it is now the only thing that selects a row. It resolves what it can, in this order, and stamps anything still unresolved with an internal sentinel the platform refuses on both sides of every comparison:
- the
globalwildcard rows, which already applied to everyone; - on
dynamic_policiesonly, rows carrying no tenant at all are mapped toglobalrather than to the sentinel. On that table an absent tenant is the apply-to-every-tenant shape, so these rows go on governing everybody and are not part of the population below; - your
tenantsmapping, which is the only step that can move a row from one organisation to another; - the legacy "organisation id equals tenant id" collapse, for any row with a non-empty tenant;
- on
policy_overridesonly, the organisation is resolved from theorganizationstable through the row's legacyorganization_id. Org-scoped override rows carry that column withtenant_idunset, so steps 2 to 4 cannot see them at all; - anything left is stamped.
The order matters for reading check 24's output: a row named there has failed every one of these, not just the first.
Step 5 is narrower than it reads, and the difference decides which override rows survive. It matches organizations.id against the row's legacy organization_id, and those were never the same kind of value: organizations.id is an auto-incrementing integer, while organization_id was declared as a UUID and only became free-form text in core/133. An integer can never equal a UUID, so step 5 rescues only integer-shaped legacy values; a UUID-shaped one falls through to the sentinel. Check 23 counts the rows step 5 can rescue and check 24 reports the ones it cannot, so between them nothing is dropped - but do not read a clean check 23 as covering both.
One warning during the upgrade is expected and is not about lost scope. core/166 drops the legacy organization_id column and raises a warning naming how many rows still carried a value in it, because the down migration cannot restore them. A row can appear in that count and still carry a perfectly good org_id; some shipped policy bundles populate both. Treat it as an inventory of what the drop discarded. What stops being enforced is check 24's list, which is a different question.
A stamped policy row stops being able to fire. This is the one change in v10.0.0 that removes enforcement rather than widening it. The migration raises a warning naming those rows, but it raises it during the upgrade, in the agent's boot log, which is the wrong moment to find out. ./preflight.sh check 24 reports the same rows read-only beforehand, using the migration's own resolution chain.
For each row it names: UPDATE the org_id column with the owning organisation if the rule should keep applying, or accept that it stops. A rule with no owner was already unreachable under row-level security on any app-role deployment, so for most deployments accepting is correct - but that should be a decision, not a discovery.
Why this was dropped rather than rebuilt on a verified key
Keeping per-tenant targeting would mean maintaining two sub-org dimensions - one verified, one forgeable - with every enforcement plane forever answering which of them wins. Per-application policy on a verified key is not foreclosed: the building block is a real credential-to-tenant binding validated at authentication time, which is recorded as unblocked and unscheduled.
tenant_id is not removed. It stays on every row and in every audit record as attribution - which credential produced a decision. What it no longer does is select policy.
Environment and configuration changes
AXONFLOW_DB_PLATFORM_ADMIN_URL becomes load-bearing
If you run the customer portal on a deployment using the application database role, which is the default since v9.0.0, set this variable before upgrading.
Be exact about how the app-role posture is turned off, because the two variables on this page do not share a vocabulary. AXONFLOW_DB_USE_APP_ROLE is disabled by exactly four literal values, false, FALSE, False and 0, and the match is not case-insensitive. Every other value leaves the posture on, including no, off, f, FaLsE and a value with a stray leading space. AXONFLOW_REQUIRE_USER_TOKEN, tabulated twenty lines below, does accept true / 1 / yes and false / 0 / no case-insensitively, and carrying that vocabulary across is the mistake to avoid: setting AXONFLOW_DB_USE_APP_ROLE=no intending to disable the posture leaves it on, and if AXONFLOW_DB_PLATFORM_ADMIN_URL is blank the orchestrator then crash-loops.
The portal's organization-wide Executions read is now refused with 500 when the application role is in use and no bypass-RLS admin pool was installed. That read would otherwise be filtered to zero rows by row-level security, which restores exactly the confident-empty page the change exists to remove, so it fails loudly instead.
The orchestrator refuses to boot without it under the app-role posture, and that is the failure you will actually meet. When the app-role posture is on and AXONFLOW_DB_PLATFORM_ADMIN_URL is blank after trimming, the orchestrator logs a fatal line naming the variable and exits, so the container crash-loops instead of starting and failing on one route. The posture counts as on unless it is set to one of the four literal false values above, which means a deployment that has never set either variable is within its scope.
A second boot guard sits behind the first, and it is what makes the inference below sound. Where the posture is on and AXONFLOW_DB_PLATFORM_ADMIN_URL is non-blank but yields no usable admin pool, the orchestrator also refuses to boot: the pool is opened during startup, the connected role is asserted to be the platform-admin role rather than the master or owner role, and a nil pool or an open error at that point is fatal rather than a fall back to the ordinary pool. So the posture being on and the container running means both that the variable is set and that what it points at actually works.
Qualify that second guard by one condition. It sits inside the startup branch that runs only when the orchestrator has a usage database wired, so it is skipped entirely in the degraded, database-less state the orchestrator is deliberately built to boot in. An orchestrator that came up without a database has not passed the second guard and proves nothing about the admin pool.
Neither guard is new in v10.0.0. Both are present in v9.19.0 as well, and this release does not change them. One consequence is worth drawing out, because it narrows who has work to do: a deployment whose orchestrator boots today with the app-role posture on already satisfies both guards, so it already has the variable set and already has a usable admin pool behind it. That claim rests on the guards' stated behavior rather than on the variable merely being present. The upgrade step matters most if you are also turning the app-role posture on, or deploying fresh.
That makes the 500 above defence in depth rather than the arm most deployments will meet. It is genuine behavior and it is what the route does; it is simply unreachable in any configuration that boots, and it exists so the read refuses rather than fabricating an empty page should the boot guard ever be relaxed. A one-time warning is logged where the fallback is taken on a pool on which it is harmless.
The require-user-token posture
(Enterprise)
core/163 adds organizations.require_user_token, defaulting to false. With it on, an enterprise caller that presents no validated per-user token is refused at authentication rather than served under a synthetic, organization-scoped service identity. This is the lever that makes a segment-scoped policy something a caller cannot shed by omitting a header, so read it together with the segment guidance below.
Two controls set it, and they compose in a defined order:
| Control | Values | Default |
|---|---|---|
organizations.require_user_token (per organization) | true / false | false, set by core/163 with no backfill |
AXONFLOW_REQUIRE_USER_TOKEN (deployment-wide default) | true / 1 / yes, false / 0 / no, case-insensitive | false |
An explicit per-organization value wins over the deployment-wide default in either direction. An organization can opt out of a true default exactly as it can opt in over a false one. Where no organization row is set, the deployment-wide default applies.
Before you turn it on, three things to plan for:
- Provision per-user tokens for every enterprise caller first. With the posture on, a token-less caller is refused at six gate points:
POST /api/v1/decide, the MCP-server session-authentication plane, and the four MCP REST routesPOST /mcp/resources/query,POST /mcp/tools/execute,POST /api/v1/mcp/check-inputandPOST /api/v1/mcp/check-output. Every gate point answers401, including the MCP-server plane, but that plane differs in two ways worth wiring for. Its401is not the platform's ordinary error envelope: session initialization and every subsequent tool call return a JSON-RPC error body carrying aWWW-Authenticateheader, and the session-delete route returns a bare401with no body. And it writes nouser_token_requiredaudit marker, so a search for that marker turns up/api/v1/decideand the four REST routes and nothing else; on the MCP-server plane the401itself is the only signal. One more thing to expect on that plane: turning the posture on does not end the sessions already open. The gate runs during session authentication, and a request carrying an established session id is resolved from the session cache without re-authenticating, so a session opened before you switched the posture on keeps working until it falls out of that cache, which is a 24 hour idle timeout. Plan for the change to take full effect over that window rather than immediately. - Set
AXONFLOW_REQUIRE_USER_TOKENonly to a value the platform recognizes. On an agent wired to a database, an unparseable value such asenabledrefuses the boot, deliberately, rather than falling back to a default and silently turning the control off for a deployment whose admin set the flag intending the opposite. A deployment that never sets the variable is unaffected. The guard is database-conditional: it runs inside the same startup step that wires the posture cache, and that step returns early on an agent with no database, so there an unparseable value does not refuse the boot and resolution falls back to the environment default silently. The posture is unreachable on such an agent anyway, but do not read a clean boot there as confirmation that the value parses. - Know which way resolution fails. Where the posture cannot be read at all, because the database is unreachable or the column is absent, resolution returns "required" and caches that for at most 15 seconds. Two cases are deliberately not failures and fall through to the deployment-wide default: an organization row that is genuinely absent, and a deployment with no database wired at all.
Resolution never issues a per-request query. A successful read is cached for AXONFLOW_REQUIRE_USER_TOKEN_TTL_SECONDS, which defaults to 60 seconds and is clamped to between 5 and 600. Lower it if you need a posture change to take effect faster; the clamp is the floor.
Revoked and invalid per-user tokens now fail on the MCP REST routes
No configuration change is needed for this one, but it can break a caller, so audit for it before upgrading rather than after.
On the four MCP REST routes, a presented token that fails validation, whether malformed, expired, signed with the wrong algorithm, carrying a bad signature or revoked by identifier, used to fall through to the same synthetic service identity as a caller who sent nothing. It is now audited as user_token_rejected and refused with 401. The practical consequence is that revoking a per-user token now actually revokes it on that plane. Any caller still working on a token that should have stopped working will stop on upgrade. The absent-token and rejected-token causes carry separate audit markers, so user_token_required and user_token_rejected can be told apart when you go looking.
SAML tenants need a real SP signing keypair before you upgrade
(Enterprise)
Do this check before the upgrade window, not during it. The customer portal loads a deployment-default SAML SP signing keypair from the secret store at startup. Until v10.0.0, any failure to load it, a missing secret, a denied read, a misconfigured region or a secret-store outage, made the portal mint a throwaway RSA keypair and install that as the deployment-default SP signing identity. That fabricated keypair was inherited by every tenant whose SSO configuration carries no sp_private_key of its own.
It made SAML look healthy while it was already broken. Metadata served, assertions were signed and logins worked, but the SP certificate was regenerated on every portal restart, so an identity provider that pins it, through signed authentication requests, encrypted assertions or strict metadata pinning, rejected logins with a signature mismatch reported at the identity provider, mid-login, uncorrelated with a boot warning from days earlier.
v10.0.0 deletes that fallback and refuses the affected tenant instead. The refusal happens when the tenant's SAML configuration is loaded, per tenant, not at boot: failing the whole service at startup would have removed SAML for every tenant, including those that store their own keypair and were never on the fabricated one. So:
| Tenant's SSO configuration | Deployment-default keypair | Result on v10.0.0 |
|---|---|---|
Stores a valid (parseable) sp_private_key + sp_certificate | Either | Unaffected. SAML works, publishing the tenant's own certificate |
| Stores neither | Loads from the secret store | Unaffected. The tenant inherits the real deployment default |
| Stores neither, or stores a PEM that does not parse | Cannot be loaded | SAML refused for that tenant: login initiation, the assertion callback and SP metadata all fail |
The word "valid" in the first row is load-bearing. A stored PEM that does not parse is discarded with a log warning and the tenant is treated as storing nothing, so it falls to the deployment default and is refused alongside the tenants that stored nothing at all.
A deployment whose tenants have no per-tenant keypair and whose secret store the portal cannot read loses SAML login on upgrade, for exactly those tenants. Say it plainly to whoever owns the change window, because the fabricated keypair made that same deployment look healthy beforehand.
Two checks answer this on your current deployment, and they answer it together:
-
The portal boot log states whether the deployment-default SP keypair loaded. Your current portal already logs that failure; it is the line the fabricated keypair made easy to ignore. If the keypair loads, no tenant is affected and there is nothing further to do.
-
A direct read of
sso_configurationslists the tenants that store no keypair of their own, which is exactly the set that loses SAML login when the deployment default is unavailable. It applies the same predicate the platform applies:SELECT tenant_id, org_id, enabledFROM sso_configurationsWHERE provider_type <> 'oidc'AND (COALESCE(sp_private_key, '') = '' OR COALESCE(sp_certificate, '') = '');sso_configurationscarries FORCE row-level security keyed onorg_id, so run this as the platform-admin role behindAXONFLOW_DB_PLATFORM_ADMIN_URLrather than as the application role, or the result is silently narrowed to a single organization and reads as a clean sweep.
Two more surfaces report it per tenant once you are on v10.0.0:
-
POST /api/v1/sso/config/test, the session-authenticated SSO configuration test in the portal settings, reports that SAML would be refused for a tenant before anyone attempts a login. Scope the instruction to your deployment mode, because the route takes no tenant parameter: it tests the configuration belonging to the calling session. On an in-VPC deployment there is one platform-wide SSO configuration, so a single call covers the whole deployment. On a tenant-isolated deployment the call resolves to the caller's own tenant, so one operator cannot sweep every tenant with it and each tenant's own credential holder has to run it. That is why the database read above, not this route, is the sweep on a tenant-isolated deployment.Two more things to get right before you read a result as reassuring. The route requires the
sso:configurepermission, not merely an admin session; a session without it is answered403, and a403is not an unaffected result. And an affected configuration answers400with a message opening "SAML login would be refused for this tenant", while the secret's name travels in the response'serror_detailsfield rather than inmessage. Match on the message text, because the same route answers400for ordinary configuration errors too, among them "Either metadata URL or SSO URL is required", "Invalid metadata URL" and "Entity ID is required". An OIDC configuration, and a SAML one that stores a parseable keypair, are reported unaffected. -
The portal request log records the cause on every refused login.
Both pre-upgrade checks share one false negative, and it is worth knowing before you trust a clean result. The check treats a stored sp_private_key and sp_certificate as sufficient when both are merely non-empty. It never parses them. The runtime does parse them, discards a value it cannot read, and then refuses the tenant for storing nothing. So a tenant holding a truncated, mangled or placeholder PEM is reported unaffected before the upgrade and refused at login after it. Treat a stored keypair you have not re-provisioned or re-verified yourself as unconfirmed rather than as clear.
The pre-authentication responses deliberately do not carry that detail. Login initiation, the callback and tenant metadata answer a generic message, because the detailed text names the secret and wraps the raw cloud SDK error, which on an access-denied shape carries the portal's own principal identifier and account id, and that does not belong in a body served to an anonymous caller. Do not build your check on the anonymous response. Every other SAML error keeps its existing wording unchanged.
Two remedies, either one is sufficient per tenant. Restore the portal's read access to the SAML SP keypair secret and restart the portal, or store a per-tenant sp_private_key and sp_certificate on that tenant's SSO configuration through the SSO configuration API, which accepts both fields on create and on update. The second is the one that does not depend on the secret store at all, but it changes the certificate that tenant signs with, so re-register the new SP certificate at the identity provider or you have traded one signature mismatch for another.
Two rules about writing those fields, because getting them wrong is how a tenant acquires a stored-but-unusable keypair in the first place. Always send the full PEM for both fields on every update that touches either. And never round-trip sp_certificate from a GET: the read API returns it masked, so a client that reads the configuration and writes it back stores the mask as if it were a certificate, which is non-empty, unparseable, and reported unaffected by both checks above. The two fields do not behave the same way on a blank value either: a blank sp_private_key preserves the stored one, so a later edit to unrelated fields does not silently clear it, while a blank sp_certificate clears it.
AXONFLOW_DEBUG_POLICIES no longer exists
Remove it from your deployment configuration. It only ever controlled the verbose logging of the in-memory dynamic-policy engine, and that engine is deleted. Setting the variable now does nothing; leaving it in place implies a behavior that is gone.
Nothing fails if you leave it set. That is the reason to remove it deliberately: the platform will not tell you.
Sweep your dashboards and alerts for plane="memory"
The plane="memory" label value of axonflow_policy_condition_unevaluable_total is no longer emitted, because the engine that emitted it is deleted. The remaining label values (database, mcp, policy_test) are unchanged.
Be clear about the failure mode: nothing can detect this for you. A recording rule, dashboard panel or alert matching plane="memory" does not error, does not warn, and does not disappear. It goes permanently and silently empty, which on most alerting stacks reads as "healthy". A panel showing a flat zero line and an alert that never fires again look exactly like a well-behaved system.
The sweep is manual and it is yours to do:
- Search your dashboard definitions, recording rules and alert rules for
axonflow_policy_condition_unevaluable_total. - For each hit, check whether the query pins
plane="memory"or enumerates plane values in a way that includes it. - Repoint those to the remaining planes, or drop the label selector entirely if the intent was "any plane".
- If an alert existed specifically to watch the in-memory engine, delete it rather than repointing it. There is no in-memory engine any more, so there is nothing for it to watch.
Four new metrics report the surviving engine's health directly and are the right thing to build replacement panels on: axonflow_policy_set_source (whether the engine is serving built-in defaults or a database-loaded set), axonflow_policy_cache_age_seconds, axonflow_policy_refresh_failures_total and axonflow_policy_zero_row_loads_total.
Review the two risk-score thresholds
risk_score is a platform-computed signal again rather than a value read out of the caller's own request body. Two seeded, enabled dynamic policies have therefore been unable to fire on a real signal since January and begin evaluating on upgrade:
| Policy | Condition | Action |
|---|---|---|
sys_dyn_high_risk_block | risk_score > 0.8 | warn |
sys_dyn_anomalous_access | risk_score > 0.6 | alert |
Neither newly blocks. Both are allow-but-annotate: warn records a warning entry and alert records a structured line plus a required-action entry that reaches the audit trail. What changes is that requests which previously passed unconditionally now get annotated, so expect a step in annotated volume rather than in denials.
The weights the score is now computed from are:
| Signal | Contribution |
|---|---|
| An SQL-injection pattern, via the shared scanner | +0.9 |
| A word-boundary-anchored sensitive-data keyword match | +0.7 |
A select * query | +0.3 |
Review both thresholds against these weights before rolling out, because the thresholds were not tuned against them. Two properties of the calculator are worth knowing while you do:
- The sensitive-data pattern is anchored, so a question like "what is a monkey" or "tell me about tokenization" no longer scores merely for containing "key" or "token" inside a longer word.
- Role no longer contributes. Role is an authorization signal rather than a risk signal, and as previously built it meant the more trusted the caller, the more likely they were to be blocked.
The explicitly namespaced context.risk_score condition field is unaffected and continues to resolve the caller-supplied value under its own name.
The running score is also now clamped to the zero-to-one range once, after the policy loop rather than inside the risk-modifying action, so several risk-modifying policies compose additively and are then clamped, instead of a value above 1 flowing verbatim into audit rows and the compliance evidence export.
API response shapes an integration must handle
These are wire changes. They apply to every client, including all five SDKs, because the SDKs forward the platform's shapes verbatim.
| Surface | Before | After |
|---|---|---|
POST /api/v1/audit/report, POST /api/v1/audit/summary | avg_latency_ms a non-nullable float, coalesced to 0 | Nullable. Explicit JSON null when nothing was measured, plus a new latency_sample_count |
GET /api/v1/audit/session-summary | avg_latency_ms non-nullable | Nullable, plus a per-bucket and per-tool latency_sample_count |
GET /api/v1/usage, GET /api/v1/usage/summary | Latency fields returned 0 when unmeasured | Nullable, plus latency_sample_count and latency_bucket_count on the summary |
POST /api/v1/audit/search, POST /api/v1/audit/export, GET /api/v1/audit/{id} | tokens_used, cost and response_time_ms always present, 0 when unrecorded | Omitted when unrecorded. CSV writes an empty cell rather than 0 |
POST /api/v1/audit/report | A scan failure was logged and skipped, returning a silently short top-policy table | 500, with a body stating explicitly that no report was produced. Also bounded by a 15 second aggregation timeout that fails the same way |
POST /api/v1/audit/summary | (same defect) | Deliberately does not fail. Degrades with a new top_policies_unavailable flag |
POST /api/v1/sebi/audit/export | status always completed, compliance_score always computed | completed, partial or failed. compliance_score absent rather than null or 0 on an incomplete export |
POST /api/v1/compliance/reports | pending / processing / completed / failed | Unchanged. Never returns partial |
POST /api/v1/workflows/execute | Run ids prefixed wf_ | Prefixed wfe_. Ids minted before this release keep the old prefix; no row is rewritten |
GET /api/v1/unified/executions (list) | An unscoped list carrying neither a tenant nor an organization key returned 200 over every organization's rows | 401 |
GET /api/v1/unified/executions (organization-wide list) | Returned a confident empty page under the application role | 500 where no bypass-RLS admin pool is installed. See AXONFLOW_DB_PLATFORM_ADMIN_URL above |
| The four MCP REST routes, with a presented-but-invalid per-user token | Downgraded to a synthetic service identity and served | 401, audited as user_token_rejected. A revoked token now actually revokes |
POST /api/v1/decide and the four MCP REST routes, with no per-user token | Served under a synthetic service identity | 401, audited as user_token_required, only where the organization's require_user_token posture is on. Unchanged while it is off |
The four MCP REST routes and POST /api/v1/decide, verified caller whose segment resolution errors | Evaluated organization-only and passed | 403 with the guard identifier segment_resolution_failed |
Three things to be precise about:
- A genuine measured
0still renders as0. What changed is that an unmeasured value is absent rather than fabricated. A consumer that indexes those keys unconditionally, or that treats a missing key as an error, must handle absence. - Usage response and export row counts can rise on upgrade. The list endpoint's scan loop used to skip rows it could not decode into a non-nullable integer, and the OpenTelemetry metrics writer never names the latency column, so every metric-sourced row was silently dropped from both the response and the CSV behind a
200 OK. Those rows now appear. - The measured-latency predicate relaxed from "greater than zero" to "not null", so a genuine sub-millisecond decision is now a sample rather than a discarded row.
The two SEBI routes behave differently
This is the change most likely to be discovered by a regulator rather than by an operator, so be precise about which route your integration uses.
POST /api/v1/sebi/audit/export (legacy, synchronous) is the route that gains partial. The roll-up is conditional, not universal, and it has three outcomes:
partial: some but not every requested section could be served, or a served one carries a scope gap.summary.compliance_scoreis then absent.failed: no requested section could be produced. A request whose sections all failed isfailed, notpartial.completed, with a score: still the outcome for a subset request that hits neither a failure nor a scope gap. That case is the documented example in the public API specification and is unchanged.
An all-types request reports partial on a stock deployment, because the human-oversight and PII-redaction sections read stores no migration in the platform creates. A request for only those two sections returns failed. Handle all three values, move off status === "completed", move off a non-optional compliance_score, and read summary.report_state and summary.sections instead.
POST /api/v1/compliance/reports (the asynchronous facade) never returns partial. Its status vocabulary is exactly pending, processing, completed and failed, and v10.0.0 does not widen it. On an Enterprise build the portal always takes this route, so every SEBI pack lands as completed, including the incomplete ones.
If you poll the facade, do not wait for a status change. Completeness on that route is carried inside the document, in a section titled "Report completeness". The presence of that section is the signal, and so is its absence: it is prepended only when at least one section failed or carries a scope gap, so a complete pack does not contain it at all. Test for the section, not for a sentence.
The MCP-server tools may now require X-User-Token
Where an organization holds an enabled segment-scoped policy for the matching phase, X-User-Token becomes mandatory on the MCP-server check_policy and check_output tools. X-User-Email is explicitly refused as a substitute, and a token naming a shared synthetic identity is refused as well.
Two refusals exist on that plane and they size differently:
segment_resolution_failedis unconditional. A caller presenting a validated per-user token whose segment resolution fails is denied whether or not the organization holds a single segment-scoped policy, and regardless of detection posture. If your MCP callers hold per-user tokens, a resolver outage denies them.segment_identity_unresolvedis conditional. A caller with no validated per-user principal is denied only when the effective policy set for that organization and phase holds an enabled segment-scoped row, or when that policy set could not be read at all. That second trigger means a database or policy-plane outage also refuses these callers, even on an organization holding no segment-scoped policy.
Both refusals arrive as an HTTP 200 JSON-RPC result carrying an unallowed verdict and a blocked_by identifier, not as an HTTP error. Match on the identifier, never on the human-readable reason text: the reason strings in that family are not punctuated consistently across the planes that emit them, so a string-equality check written against one plane's wording silently misses another's.
Skipping releases: upgrading from 9.16.x or earlier
v10.0.0 is cumulative. An operator on 9.16.x crosses 9.17, 9.18 and 9.19 in one boot; an operator further back crosses more. The migration runner applies every unapplied migration in order in the same startup, so plan the maintenance window for the sum of them, not for core/160 through core/166 alone.
Here is the full set from core/155 forward, with the release each landed in, so you can work out which apply to your baseline:
| Migration | Landed in | What it does | Cost |
|---|---|---|---|
core/155 | v9.13.0 | Normalises tenant_id = '' to NULL on both policy tables, and on dynamic_policies also sets enabled = false. The disable is deliberate: NULL is the apply-to-all sentinel, so normalising alone would promote a row enforced for nobody into a row enforced for every tenant | Small, but read the v9.13.0 upgrade guide first: a repaired row becomes unreachable through the product, and the migration is the last moment those rows are identifiable |
core/156 | v9.13.0 | Makes the tenancy keys NOT NULL and adds a blank-string CHECK on plans, workflows, workflow_checkpoints, execution_summaries and webhook_subscriptions, after stamping orphaned rows with an inert sentinel | Scans rather than table rewrites, but all five run in one transaction, so the ACCESS EXCLUSIVE locks on all five tables are held until commit. Size for the sum of the five scans, not the largest table |
core/157 | v9.14.0 | Adds a nullable static_policies.segment_id, NULL on all existing rows | Additive. No rows rewritten |
core/158 | v9.16.0 | Adds two optional OIDC client-credential columns to the SSO configuration table | Additive. No-ops on deployments without that table |
core/159 | v9.17.0 | Adds a nullable dynamic_policies.segment_id plus a partial index, using IF NOT EXISTS on both, so it is idempotent | Additive. No rows rewritten, and a policy with no segment keeps matching exactly as before |
core/160 | v10.0.0 | Deletes the superseded high_risk_block seeded policy row | One row |
core/161 | v10.0.0 | Nulls fabricated zero response_time_ms on audit_logs | Whole-table. Size your statement_timeout |
core/162 | v10.0.0 | Nulls fabricated zero tokens_used and cost on audit_logs | Whole-table. Size your statement_timeout |
core/163 | v10.0.0 | Adds organizations.require_user_token, default false | Additive. No backfill, no rows rewritten |
core/164 | v10.0.0 | Widens organizations.license_key from VARCHAR(512) to TEXT | Metadata-only ALTER. No table rewrite |
core/165 | v10.0.0 | Makes org_id NOT NULL and non-empty on the three policy tables | ACCESS EXCLUSIVE, no rewrite. A policy row it cannot resolve an organisation for stops being selectable - run ./preflight.sh check 24 first |
core/166 | v10.0.0 | Drops the legacy organization_id column from the same three tables | Metadata-only |
core/167 | v10.0.0 | Adds the HITL approval-grant column, two partial indexes and a widened history CHECK | Blocks reads on hitl_approval_queue and hitl_approval_history until commit |
Two behavior changes on that path have no schema change and are easy to miss on a multi-release skip:
- v9.19.0 excluded policies stored with an explicitly empty conditions list from evaluation. A policy with no conditions applies to everything, so leaving those rows in place would have turned each one into a match on every governed request. The v9.19.0 release notes carry the query that finds them, and running it before you upgrade is worthwhile on a long-lived database.
- v9.19.0 converged five separately maintained policy-condition matchers onto one evaluator, which changes what some stored policies match. Five named behaviors converge, and six operators that were silently inert on the tool-call, connector and content-type planes begin enforcing there.
Read the intermediate release notes for anything else on your path. This table is the migration inventory, not a substitute for them.
What you need to do, by consumer type
Self-hosted operators
- Run the sizing queries above against your own
audit_logsand setstatement_timeoutaccordingly, or raise it for the migration role for the duration of the upgrade. - Set
AXONFLOW_DB_PLATFORM_ADMIN_URLif you run the customer portal on the application database role. - Remove
AXONFLOW_DEBUG_POLICIES. - Sweep dashboards, recording rules and alerts for
plane="memory". - Review the
0.8and0.6risk-score thresholds against the new weights. - Decide whether you want per-user tokens required. If you do, provision tokens for every enterprise caller first, then set
require_user_tokenon the organization orAXONFLOW_REQUIRE_USER_TOKENdeployment-wide. Leaving both alone changes nothing. - Audit your MCP REST callers for expired, malformed or revoked per-user tokens, which stop being downgraded to a service identity and start being refused with
401. - If any tenant logs in through SAML, check the portal boot log for whether the deployment-default SP keypair loaded, and run the
sso_configurationsread in SAML tenants need a real SP signing keypair before you upgrade to list the tenants that store no keypair of their own. Those two together are the affected set, and the read is the sweep becausePOST /api/v1/sso/config/testtakes no tenant parameter and only ever tests the calling session's own configuration. On an in-VPC deployment there is one configuration and one call to make; on a tenant-isolated deployment each tenant's ownsso:configureholder has to make it, and a403from a session lacking that permission is not an unaffected result. Restore the portal's access to the secret, or store a per-tenantsp_private_keyandsp_certificateand re-register the new SP certificate at the identity provider, before upgrading. A tenant with neither loses SAML login on upgrade, and so does one whose stored PEM does not parse, which neither check can see. - Run
./preflight.shchecks 23 and 24 and act on what they name: re-scope any per-tenant POLICY rows you are not willing to have apply organisation-wide, and stamp an organisation onto any policy row that has none. Check 23 reports two shapes, and the second is the one to read carefully - an organisation whose policy rows target only some of its tenants carries no divergence among its own rows and still starts governing every sibling. The override rows it lists need no action: overrides are deliberately not made organisation-wide, so revoking one is not a remediation step here. See Per-tenant policy targeting is dropped. - Plan to roll the agent and the orchestrator together. A version-skewed pair fails closed on the internal policy-evaluation call rather than silently dropping tenant dynamic policies.
- Take the usual pre-upgrade snapshot, then pull the v10.0.0 images and restart. The migrations apply on boot.
- After the upgrade, confirm the portal Executions page returns rows rather than a
500, confirmaxonflow_policy_set_sourcereports a database-loaded set once the first successful load has happened, and add an alert onaxonflow_segment_subject_org_mismatch_total, whose non-zero value means a segment-scoped policy is not restricting the members it names.
Enterprise deployments authoring segment-scoped policies
Segment-scoped policies now enforce on eight planes: the gateway pre-check plane, the workflow control plane's HTTP step-gate, the MCP-server check_policy and check_output tools, the four MCP REST routes, and POST /api/v1/decide. A policy that blocked a member elsewhere begins blocking them there.
On the four MCP REST routes and POST /api/v1/decide there is a second new outcome to plan for: a caller with a validated per-user principal whose segment resolution errors is refused with 403 and the guard identifier segment_resolution_failed, so a resolver outage now denies those callers rather than passing them organization-only.
Provision per-user tokens for any MCP caller before upgrading. Then read the deliberate disclosure in the release notes, because coverage is now wide but neither complete nor unconditional, and the two gaps have different remedies.
Two enforcement routes still evaluate organization-only, and their scopes differ:
- The OpenAI-compatible endpoint,
/v1/chat/completions, is the wider of the two. It passes no segment set into the general request-phase evaluation, so a segment-scoped policy of any category is excluded there. It mirrors OpenAI's wire shape and carries no user-token field, so a caller on it has no validated per-user principal from which segments could be resolved. - The response plane,
POST /api/v1/process, is narrower, and it is an enforcement gate rather than an observability read: a policy whose effective response-phase action isblockwithholds the LLM response there and the caller is answered403. It too passes no segment set. It serves proxy, Gateway and multi-agent modes, not Gateway Mode alone. Its exposure is narrower because that plane evaluates only the PII and sensitive-data categories your detection configuration has enabled, and skips evaluation entirely when none are, so what goes unenforced is a segment-scoped policy in one of those enabled categories whose effective response-phase action isblock.
The REST MCP handlers and POST /api/v1/decide belonged on this list before this release and now resolve segments.
On six of the eight planes, enforcement is conditional on identity, and require_user_token closes it on exactly those six. A caller presenting no validated per-user token is evaluated organization-only and is not refused for it, so a member can shed a segment-scoped restriction by simply not sending a token. The posture's gate points are the MCP-server session-authentication plane, the four MCP REST routes and POST /api/v1/decide; setting require_user_token on the organization, or the deployment-wide default, is what closes that on them. See The require-user-token posture above.
The remaining two planes are not covered by the posture, in opposite directions. The gateway pre-check already requires a validated per-user token unconditionally on an enterprise deployment: a caller that presents none, or one that fails validation, is answered 401 before any policy is evaluated, so there is nothing for the posture to add there. The workflow control plane's HTTP step-gate is the other way round. It reads X-User-Email, degrades to organization-only when the header is absent, and has no posture remedy today, so on that plane the organization-scoped policy below is the whole of the answer.
Until you have covered what you can, keep an organization-scoped policy behind any segment-scoped one so the organization-only evaluation still refuses on both remaining routes, and, if you rely on segment scoping as a security boundary, gate the OpenAI-compatible endpoint at the edge as well. An edge gate does not reach the response plane, and the reason is worth being exact about, because it is not that the plane is unreachable from outside. POST /api/v1/process is registered on the agent's own public router, so it is separately addressable. But gating that path would not close the plane: POST /api/request, a second public agent route, forwards to the same orchestrator handler, so both paths land on the same response processing. A path-based edge gate is therefore incomplete rather than impossible, which is why the paired organization-scoped policy is the remedy that actually covers this plane.
Programmatic API consumers
Work through the response-shape table above. The short version: handle nullable latency, handle absent token, cost and response-time keys, handle 500 from the audit report endpoint, handle partial and failed on the legacy SEBI export, and stop prefix-matching declarative workflow run ids on wf_.
Compliance and regulator-export consumers
Expect an all-types SEBI pack to be incomplete on a stock deployment, and expect that to be visible now where it was not before. On the legacy route, that is a partial status with no compliance_score. On the asynchronous facade, the status stays completed and the news is in the document's "Report completeness" section, whose presence is the signal.
Retention rows for the SEBI data types now report an unknown status with a not_configured cause where no retention configuration row exists, rather than reporting compliant over the compiled-in default. The 1825-day figure is still reported, because the default really is what would be applied, and it now travels with a retention_configured boolean so a reader can tell a configured period from a compiled-in one.
SDK and plugin users
No client release accompanies this train and minimum-version floors are unchanged, so clients below the recommended versions keep working. The recommended versions stay at Go 9.1.1, Python 9.1.0, TypeScript 9.1.0, Java 9.1.0 and Rust 0.8.2. Every SDK forwards the platform's response shapes verbatim, so the handling above is application-side work regardless of which client you use.
Getting help
If a call or an export started behaving differently after a v10 upgrade, the most useful information for support:
- The exact endpoint and request body or query string.
- The HTTP status code and response body, including any
blocked_byor policy identifier it carries. - Whether the deployment runs the application database role, and whether
AXONFLOW_DB_PLATFORM_ADMIN_URLis set. - For a migration failure: the container log line naming the migration, and your
statement_timeoutandaudit_logsrow count. - For a
401on a decision or MCP call: whether the organization'srequire_user_tokenposture is on, whetherAXONFLOW_REQUIRE_USER_TOKENis set and to what, and whether the audit row carriesuser_token_requiredoruser_token_rejected, which distinguish an absent token from a rejected one. On the MCP-server plane neither marker is written, so send the response body and headers instead. - For a SAML login that stopped working: the portal boot line reporting whether the deployment-default SP signing keypair loaded, the
POST /api/v1/sso/config/testresult for that tenant, and whether that tenant's SSO configuration stores its ownsp_private_keyandsp_certificate. The anonymous login response is deliberately generic and will not tell you which.
[email protected] with "v10 migration" in the subject so it routes correctly.
See also
- v8 → v9 Migration Guide for the previous major upgrade, which canonicalized the audit decision vocabulary and the decisions read API.
- v10.0.0 Release Notes for the full behavior narrative behind every change here.
- v9.13.0 Upgrade Guide if your baseline is v9.12.x or earlier, for the
core/155andcore/156preflight. - Failure Modes And Recovery for degraded-provider, connector, approval, and runtime behavior.
