FL Fredrik Lindstrom

Resource

A zero override rate is not a quiet system.

Most boards are told a human reviews the AI output. Almost none are told whether that human has ever changed one.

Override rate is the share of AI outputs a reviewer changes, rejects or flags before the output ships, over a defined window, per workflow. It matters because of what it cannot be. A sign-off can be performed in two seconds and leaves the same record as one that took an hour. An override cannot be faked into existence — it is the rare oversight number that cannot be rubber-stamped. That makes it the anchor signal of the governance column, and it converts “we have a human in the loop” from a sentence anyone can say for free into a number someone has to produce.

It also has to be read backwards from how most metrics are. A near-zero reading is not the good news it looks like. Security already learned this one: an analyst whose alert-escalation rate is zero rarely means a quiet network, it usually means the alerts stopped getting read. The mechanism specific to this domain is worth one line to a board — a fluent wrong answer looks identical to a fluent right one, and reviewers trust polished output.

The fourth row on the sheet is the one no interpretation band can catch, and it is the most dangerous because it satisfies every other control. Where the oversight role has a name against it but no authority ever exercised, the name itself suppresses the asking-around. With no owner at all, “who decides this?” stays live and somebody eventually escalates. With a name and no exercised authority, the organisation has an answer, the escalation never starts, nothing arrives at the review step, and the dashboard goes quiet. The column has an owner, a cadence, a reporting line and a number. The number is zero and the number is correct.

Which is why the rate is unreadable on its own, and why tightening the bands does not help. Read it against one decision the owner actually exercised in the trailing quarter — a hold placed, a budget refused, an escalation raised past the sponsor. Attendance at a governance forum is not evidence.

A four-row table of readings for the AI Oversight Override Rate: near zero, to investigate rather than celebrate; low to moderate, where the feedback loop is plausibly real; sustained high, where the deployment needs re-scoping rather than more review; and zero against a named owner who has never exercised the authority, which no interpretation band can detect. Each row names the lever that applies.
Download the sheet PDF, text selectable, prints for board packs. No email required.

Four readings, and the lever for each

  1. Investigate before celebrating

    Near zero (~0–2%)

    Rubber-stamping or automation bias — a fluent wrong answer looks identical to a fluent right one. An analyst whose escalation rate is zero rarely means a quiet network. Lever: sample the overrides — is the model right, or is nobody looking?

  2. Plausibly real

    Low–moderate (~15–30%)

    The feedback loop is plausibly real — plausibly, not proven. The cheapest way to hit a number is a trivial edit on every output. Lever: log the type and severity of each override, not the count, and track it per reviewer.

  3. Re-scope, don’t re-review

    Sustained high (>50%)

    Not an oversight problem — the output is poor enough that review has become production. Lever: re-scope the deployment, not the review step. Read it against the capability tier first; a new high-stakes workflow and a mature narrow one differ.

  4. This should stop a board meeting

    Zero, with a named owner

    The ghost owner — no band detects it. The name stops the asking-around: there is an answer to “who decides this?”, so nothing escalates and nothing arrives to be overridden. Zero, and correct. Lever: one decision the owner exercised last quarter.

Two disciplines carry with this metric wherever it goes. The bands are heuristics, not benchmarks. No benchmark data exists for override rate; the ranges are signal levels to investigate, and presenting them as certified thresholds would be the overclaim this toolkit exists to correct. And it is a diagnostic, never a target. Any reading outside its band — high or low — tells the owner to sample the overrides and find out whether the model is right or nobody is looking. The moment the metric is given a target it acquires a number people manage toward, and the thing it detects disappears. One further boundary, which is a reporting rule rather than a footnote: where review is delegated to another agent, this metric does not apply to the delegated step. An agent reviewing an agent produces a figure measuring agreement between two systems, not oversight. Govern the envelope instead — the thresholds, the escalation route, the standing exceptions granted and revoked — and measure the human overrides there. Either way, the row says which seat it is.

Three things to settle before the number goes in the pack

  1. 01

    Is the reviewing seat human? Delegate review to another agent and this metric does not apply to that step — it measures agreement between two systems, not oversight. Govern the envelope instead. Either way, the row says which seat it is.

  2. 02

    Name one decision the column owner exercised last quarter. Not a meeting attended — a hold placed, a budget refused, an escalation raised past the sponsor. If the row is empty, the override rate cannot be read at all, whatever it says.

  3. 03

    Take ten overrides claimed this quarter. Can each be tied to a change in a system whose owner has no stake in this number? Most of ten and the metric has a substrate. Fewer, and it is self-attesting — worth saying to a board before a general counsel says it for you.

A sign-off can be performed in two seconds. An override cannot be faked into existence.

Source: Fredrik Lindstrom, Cybersecurity Metrics Toolkit — AI Oversight Override Rate metric card, the anchor signal of the governance column in Columns Not Layers (11 August 2026). The metric and its framing are the author’s. The interpretation bands above are heuristics, not benchmarks. No benchmark data exists for this metric; they are signal ranges to investigate, and presenting them as certified thresholds would be the overclaim the toolkit exists to correct. Human-review scope: Singapore IMDA’s Model AI Governance Framework for Agentic AI (v1.5, 20 May 2026) requires human approval for irreversible, high-stakes or atypical actions, and states the preference for structural controls over prompt-layer ones. Oversight effectiveness: EU AI Act Article 14 requires that oversight can actually be exercised rather than nominally assigned, and Article 26(2) puts the competence, training, authority and support to exercise it on the deployer; the Article 4 AI-literacy duty has been in force since 2 February 2025 — whoever reviews must be competent to disagree, or the rate measures compliance with a rota. This is not a date-driven metric and nothing here turns on a compliance deadline. Maps to: NIST AI RMF MEASURE and MANAGE; ISO/IEC 42001 human-oversight provisions and Clause 9. Row groupings and verdict labels above are this sheet’s reading of the cited source, not the source’s own framing.

Related

A Venn is a claim about overlap — the diagram that files a responsibility matrix and the oversight it exists to assign as separate concerns — sorting by topic is what produces an owner nobody ever asks to decide anything.

Governance is not the bottom brick — the column this metric is the anchor signal for, and why it has to hold on every floor rather than being finished once at the bottom.

The Governance Memo carries this work monthly for boards and CISOs — one breach post-mortem and two or three governance items.