A model produces a draft contract, a summary of a patient record, a credit memo, a set of figures for a board pack. The output is good, frequently better than the hurried human version it replaced, and it goes into a process that was designed on the assumption a person made it.
Nothing in that process asks who is answerable for it. That question stays unasked until the first output that is confidently, plausibly wrong reaches somebody who acted on it.
Review is not the same as reading
The instinct is to say a human reviews everything, and in most deployments that is formally true and practically hollow. A reviewer presented with fluent, well-formatted, correct-looking output at volume does not audit it; they scan it. This is not laziness — it is what happens to anyone checking work that is right ninety-odd per cent of the time, and it is well documented in every other field that has tried to use humans as a backstop for automation.
So the sign-off exists and the scrutiny does not, which is the worst of both arrangements: the organisation has a named accountable person and no actual control.
The organisations handling this seriously have generally done two unglamorous things. They have defined which outputs are consequential enough to require genuine review rather than assent, which is a much shorter list than everything. And they have made the reviewer's job possible by requiring the system to show its inputs — which returns to provenance, and to the fact that companies still cannot reliably tell whether their AI works at the level of a single answer.
Contractually the ground has moved faster than internally. Corporate buyers pushing for standard terms have secured indemnities and audit rights from vendors, which addresses the supplier relationship and does nothing about the employee who approved the output. Liability that stops at the vendor's door still lands inside the company.
Regulated sectors are ahead by necessity, and their answer is instructive because it is so old-fashioned: a named individual, a defined scope, a record of what was reviewed and on what basis, and a periodic sample audited by someone who did not do the original review. None of that is novel. It is how model risk has been governed in banking for years, applied to a broader class of output.
The disclosure question sits underneath all of it and is mostly unresolved. A reader, patient or counterparty receiving a document has an interest in knowing how it was produced, which is the argument for disclosing AI use the way companies disclose their auditors — a position that is easier to hold in the abstract than at the moment of putting a line on the bottom of a letter.
Topics aigovernancerisk
