The Data You Cannot Clean

By Fredrik Lindstrom · ~11 minute read · October 2026

50 petabytes prices out between $60 million and $150 million over three years, and most of that is people and storage rather than tooling. I worked that estimate up, and it holds. The question sitting underneath it does not.

Here is the arithmetic, roughly. 50 petabytes is 50,000 terabytes, and it does not reduce to a per-terabyte rate, because it is three different populations wearing one number.

Those per-terabyte rates are modeled from program shape rather than taken from a benchmark, so treat the shape as the finding and the precision as approximate. Tooling and processing lands near $15 million. The costs that dominate sit outside the table entirely. 60 stewards for three years is roughly $29 million, which is a department you stand up and then disband. Hot object storage for 50 petabytes runs about $14 million a year at prevailing rates, or $39,000 a day to hold data nobody has touched. One full migration across clouds adds $2.5 million in egress before a single record is examined.

Then the part that should stop the meeting. The AI deployment that triggered the estimate will turn out to need two domains. Call it 300 terabytes: six-tenths of one percent of the estate, and the other 99.4 percent was never in the decision.

So before anyone prices a cleanup, someone at the table should ask which parts of the estate have a clean version to reach, and which parts the decision actually touches. In a surprising number of domains, the answer to the first question is none.

Where cleaning removes the evidence

Take three builds a board would recognize from its own capital plan.

Pharmaceutical research wants a responder-prediction model pooling twelve trial sites. Pooling means harmonizing, and harmonizing means batch correction, which removes the site effect that was carrying the signal. Patients at different sites differ systematically, because sicker patients get referred to different centers. The compound does not disappear. It gets normalized into background, and the failure surfaces three years later as a trial that will not replicate. A 2018 study in JAMA Internal Medicine estimated the pivotal trial behind Novartis’s sacubitril–valsartan, 8,442 patients, at around $347 million. That is the price of a single late-stage trial, and the order of magnitude on finding out late. Andrew Witty, then chief executive of GlaxoSmithKline, made the point about the industry’s economics years ago: If you stop failing so often, you massively reduce the cost of drug development. Failing late is the expensive version.

Physics wants an anomaly-detection model to find what nobody thought to look for. It gets trained on what the trigger kept. At the Large Hadron Collider almost every collision is discarded in microseconds and nothing is stored, so the model can only surprise you inside a boundary drawn before the data existed. CERN runs on roughly 1.5 billion Swiss francs a year, and a decision taken in microseconds sets how much of that any model can reach.

Finance wants an agent that reconciles entries and proposes corrections. Level 3 fair value has no true number behind the estimate; it is the output of judgment, and two competent valuers working the same transaction reach two defensible figures. The agent returns a point. The estimate was a range, and the range was the disclosure. Get the correction itself wrong with intent and Section 802 of the Sarbanes-Oxley Act reaches 20 years. Notice that the third cost is the only one of the three not denominated in money.

Three costs, three currencies, no exchange rate between them. Any program that reports all three as one data quality percentage has already made the error.

The standard that travelled from somewhere else

Nobody in those three stories is being careless, and that is the part worth sitting with. Someone is hitting a completeness target, or remediating an anomaly the profiling tool flagged, following a standard written by competent people for a domain where cleaning made sense. The destructive act looks like diligence, which is why it clears review.

The failure classes are distinct, and they call for different responses:

I have watched this structure before, in security telemetry. You cannot clean a log estate, because the malformed packet you would discard as garbage is the attack, and deduplicating events destroys the timing that made the pattern visible. Every SOC that learned this learned it by tuning out the noise and losing the intrusion inside it. AI teams are rediscovering it now, in domains with longer feedback loops and larger bills.

The regulation has already conceded the point, which is the part most boards have missed. EU AI Act Article 10(3) requires training, validation and testing data sets to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete. That qualifier is a reasonableness standard, and it is there because the drafters knew a strict one was unmeetable. Article 10(2) enumerates the data-preparation operations that have to be governed, cleaning among them. Under the Digital Omnibus on AI, Regulation (EU) 2026/1744, in force since 27 July 2026, these obligations apply to stand-alone Annex III high-risk systems from 2 December 2027, and to embedded high-risk systems from 2 August 2028. The artifacts an auditor will ask for cannot be reconstructed after the model is trained.

The data that is not yours to clean

The last row on that table is different in kind, and it is the one every enterprise has.

Six of the seven are impossibilities. There is no clean version to reach, or the law forbids reaching it. The seventh is organizational. A clean version exists in principle; you simply have no authority to produce it. The marketing team builds a propensity model on finance’s customer master. The risk function trains on operations telemetry. The AI team has read access to eleven systems and write authority over none of them.

So the team does the only thing available. It copies the data into its own environment and cleans the copy.

That copy is a fork, and it starts diverging the week it is made. The source keeps changing under its real owner. The model runs on a world that stopped updating, and the two drift apart until somebody notices in a board pack that operations and the model disagree about how many customers exist. Nobody owns the reconciliation, because nobody registered the fork. No owner, no retention schedule, no lineage back to the system of record. Shadow data with a business case attached.

The blocker here is authority, not technique. A data quality platform will happily clean the fork and report a rising score while the divergence widens underneath it.

Which makes this the only one of the seven that governance can straightforwardly fix, and the fix is the ordinary one. Someone owns the source, with authority to change it, budget to do so, and an escalation path when a downstream consumer needs something different. Absent that, the enterprise has two honest options: fix the data at source, or run on the source as it stands and carry its known defects into the model’s documentation. Forking silently is the third, and it is the one taken by default.

This is why data quality belongs in a governance column running alongside every deployment rather than in a stage that a project passes through and exits. A cleaning stage completes. Ownership of a shared data asset never does.

What you can still build here

None of this argues for leaving AI out of these domains. It argues against one particular move, and the alternatives are ordinary engineering.

Model the defect instead of removing it. The pharmaceutical case fails because batch correction deletes the site effect. It does not have to be deleted. A hierarchical model carries site as a term, so the variation stays in the model and comes out the other side as a wider interval on the estimate. The finding survives, honestly labeled as less certain than a corrected pipeline would have claimed. Mixed-effects modeling is decades old and every biostatistics group already knows how to do it.

Treat missingness as an input. An absent lab value records a clinician’s decision not to order the test. Feed the model an indicator for absence alongside the value, and the judgment becomes a feature instead of a hole someone imputes away. The same applies to sensor dropouts and telemetry gaps, where the gap is often the most informative thing in the window.

Query the source rather than copying it. This is the answer to the ownership problem, and it is the one most teams skip. A model that retrieves from the system of record at inference time never creates a fork, never goes stale, and inherits the upstream owner’s corrections for free. It is slower per call and it requires an integration nobody wanted to build. It also removes the entire class of failure described above.

Let the system scope the cleanup. This is the inversion, and it is the one that saves the three years. You cannot produce a reliable inventory of what needs cleaning by auditing the estate, because the estate does not know which of its defects a given decision can tolerate. Build the system first. Instrument where it copes, where it works around, and where it escalates, and the list of what actually has to be remediated writes itself — in production, against real decisions, at a fraction of the volume anyone estimated. 300 terabytes, not 50 petabytes, and you learn which 300 by running.

Make the output carry the range. If the model returns a confidence interval and the interface rounds it to a number, the engineering was wasted at the last inch. The interval has to survive to the person acting on it.

None of these is exotic. What they have in common is that they preserve the uncertainty rather than resolving it early, which costs precision on paper and buys correctness in the decision.

What boards already know how to do

Here is the part that should be reassuring rather than alarming.

Boards have governed judgment-constituted data for well over a century. They did not call it a data quality problem, because it never was one. They called it materiality, auditor independence, disclosure of estimation uncertainty, and a signed opinion attached to a range rather than a point. Every director reading this approves that machinery annually and knows what a qualified opinion means.

None of it involves cleaning anything. All of it is governance.

So the transfer is available, and it does not need a new instrument. It needs the one that already governs every other cross-cutting function: a named owner with halt authority. Applied to data, that reads as a specific person who can change the source, and who can stop a deployment running on a version of it the decision cannot support.

Data quality is a column, not a stage. A stage completes and the project moves past it. Ownership of a shared data asset never completes, which is why cleaning keeps getting funded as a project and keeps failing as one.

Three questions establish whether the column is load-bearing, and a director can ask all three in one meeting. Who owns the source, and can they halt a deployment running on it? What did the pipeline remove, and who authorized the removal? And does the model’s output still carry its uncertainty by the time a human acts on it, or has it been rounded to a number on a dashboard?

The third is the one that gets skipped, and it is the one this argument turns on. Everything upstream can be sound and the governance still fails at the last inch, when an interval becomes a figure on a slide. Pair it with the AI oversight override rate, which needs a companion signal anyway, because an override rate near zero can mean the governance is working or that it is theater.

Notice what is not on that list. Nobody is asked whether the data is good. In an irreducibly dirty domain that question has no answer, and an instrument demanding an unobtainable input produces a fabricated one.

In my experience, a data quality program reporting 94 percent clean arrives in the boardroom the same way a green risk dashboard did in 2014, and it should be read the same way. It is not evidence that the estate is clean. It is evidence that somebody chose a definition of clean that could be hit, and then hit it.

The next twelve months will produce a run of AI deployments in these domains, priced against cleanup estimates built on the wrong denominator. Some will be paused when a regulator asks what was removed and nobody can answer.

And there is a cost that never appears on the estimate, because it has no invoice. While the cleanup runs, nothing ships. Three years of remediation produces a clean estate and an organization that has learned nothing about deploying AI, because that knowledge is only ever acquired by deploying.

The competitor who scopes an agentic system instead can ship in four months. Emphasis on system, because what gets built is the retrieval and the checkpoints around it, and the data is one input among several. It works with the data where the data holds, works around where it can, escalates where needed, and in running produces the only trustworthy inventory of what actually has to be cleaned. 300 terabytes, found by deploying rather than by auditing. None of that transfers from a vendor, and none of it can be bought back at the end of the program.

That is the real price of the data-first sequence. The hundred million is recoverable. 36 months of institutional learning that never started is not, and the competitors are 36 months into theirs.

The boards that do well will be the ones that stopped asking how clean the data is, and started asking what it cost to make it look that way.

Common questions

Does this mean we should not invest in data quality at all?

No. Most of an enterprise estate is ordinary operational data where deduplication, validation and access remediation pay for themselves. The distinction is domain-by-domain. The point is that a single quality standard applied uniformly across the estate will do real damage in the minority of domains where the anomaly is the finding.

How do we tell which of our data is un-cleanable?

Ask what the cleaning step assumes. If the remediation assumes there is a correct value the record should have held, and the domain cannot supply one, that is the signal. Genomics, longitudinal clinical records, sensor and telemetry estates, and any Level 3 fair value estimate all fail that test for different reasons.

We pull data from another department and cannot change it. What do we do?

Copying it so you can clean it creates a fork, and the fork begins diverging from the source almost immediately. There are two governed options. Fix the data at source, which requires an owner over there with authority and budget. Or run on the source as it stands and carry its known defects into the model's documentation and its output. Pick one deliberately and record which, because the default in most organizations is to fork quietly and discover the divergence in a board pack.

Who owns this question, the CDO or the board?

The Chief Data Officer owns the pipeline. The board owns whether anyone has halt authority over it and whether the uncertainty survived it. Those are different questions, and the second one does not delegate, because it is the same fiduciary question as approving an estimate with a range attached to it.

Does EU AI Act Article 10 require us to clean our training data?

EU AI Act Article 10(3) requires data sets to be relevant, sufficiently representative and free of errors to the best extent possible, in view of the intended purpose. The qualifier matters. What is enforceable is the documentation: what you did to the data, why, and what gaps you identified. Those records cannot be produced retroactively once the model has been trained.

What is the one thing to ask management this quarter?

Ask for one deployed model, and ask what the pipeline removed from its training data and who approved the removal. If nobody can answer inside a week, the answer is already informative.