In April 2019, I asked a simple question. When an organization gets breached because they did not bother to implement basic controls, and someone’s identity is destroyed for the next seven years as a result, how should the organization be penalized?
The data I cited then: 16.7 million Americans had their personal information used illegally in 2017, at a direct cost of $16.8 billion. Roughly $1,000 per victim, against the $480 worth of credit monitoring most breached organizations were offering as compensation. I argued that organizations should be required to cover all expenses for everyone they harmed, for at least seven years.
That position aged well. It also turned out to be the position regulators were quietly arriving at, just slowly. GDPR penalties got there in part. The FTC got there in select cases. The SEC’s 2023 cyber disclosure rule got there in another way. None of it has been enough to match the actual harm being inflicted.
Now the same question is about to apply to AI failures, and the answer the regulators arrive at will define what AI looks like in the enterprise for the next decade.
The Promise
There is, finally, a serious framework for thinking about this.
Three things have changed in the last 24 months that did not exist when I wrote the original piece in 2019.
First, the EU AI Act creates explicit categories of prohibited and high-risk AI uses. The regulatory architecture itself is built around the idea that some AI failures are categorically more serious than others. Social scoring is prohibited. Biometric categorization based on sensitive attributes is prohibited. AI used in critical infrastructure, education, employment, law enforcement, and migration is high-risk. Each tier has a corresponding obligation structure and a corresponding penalty structure.
Second, the NIST AI RMF gives organizations a framework that, when adopted, creates a defensible record of reasonable care. This is the analog of the old “we followed CIS Top 20” defense for cyber. It does not eliminate liability. It creates a documented basis for arguing that the organization did the work.
Third, U.S. state-level AI legislation is filling in the federal gap. Colorado’s 2024 AI Act. The New York City automated employment decision tools rule. California’s algorithmic discrimination provisions. The federal regulatory environment may be uncertain, but the state-level environment is creating a patchwork that, in aggregate, looks a lot like the federal environment that emerged after Sarbanes-Oxley.
The promise here is that the question I asked in 2019 — how should organizations be penalized for failures they could have prevented — finally has the regulatory architecture to support a serious answer. The penalty structure is being built right now. The question is whether organizations are going to participate in shaping it or wait for it to land on them.
The Risk
The risk is two-sided, and both sides of it matter.
On one side, under-penalization. If AI failures continue to be priced as a regulatory cost rather than as harm to actual people, the same cycle that produced the data-breach economy will produce an AI-incident economy. Hallucinated outputs in medical contexts. Biased outcomes in hiring. Discriminatory pricing in insurance. AI-generated content that defames someone who has no recourse. The pattern from 2017 was that breached organizations rationally calculated that the breach cost less than the prevention cost and behaved accordingly. If AI failures get priced the same way, we will see the same behavior.
On the other side, over-penalization that targets the wrong actor. The natural temptation in regulation is to penalize the most visible actor in the chain, which is usually the deployer rather than the developer. But the EU AI Act is doing reasonable work distinguishing between providers, deployers, importers, distributors, and authorized representatives. The U.S. patchwork is doing less reasonable work on this distinction. The risk is a regulatory regime in which a small business using a third-party AI tool ends up bearing penalties intended for the foundation model provider, while the foundation model provider continues to operate with the limited liability of an open-source disclaimer.
Both failure modes have a shared structural cause: the difficulty of attributing AI harms to a specific decision by a specific actor. This is the technical problem that needs to be solved for the regulatory architecture to actually function. Until it is, the penalty regime will be either too weak to prevent harm or too blunt to be fair.
The Verdict
Seven years ago I argued that organizations should be required to make their victims whole, for as long as it takes those victims to recover. That position still holds. The mechanism for getting there has changed.
For data breaches, the mechanism turned out to be a combination of GDPR-style direct penalties, sector-specific regulation, FTC consent decrees, state attorney general settlements, and class action litigation. Not pretty, not consistent, but ultimately effective at making the breach economics worse than the prevention economics for serious actors.
For AI failures, the mechanism is going to be similar in structure but tighter in timeline. The EU AI Act provides the direct-penalty piece. The state AI laws provide the sector-specific piece. The plaintiff’s bar is already preparing for the litigation piece. The piece that is missing — and that needs to arrive faster than it did for cyber — is a clear regulatory expectation that AI deployers must demonstrate they did the work, and that this demonstration must be documented before the harm occurs, not reconstructed after.
The original question still applies. How should organizations be penalized for AI failures they could have prevented? The honest answer is: enough that prevention becomes the rational economic choice. We are not there yet. But for the first time since I wrote the original piece, the regulatory architecture exists to get there. The next 24 months will define whether it actually does.
After 25 years in this work, the pattern is unmistakable: the cost of building governance ahead of the regulator is always lower than the cost of being on the wrong side of the enforcement curve. AI is not an exception. It is the next iteration.