Your vendor can't explain the model either. That's the part people miss.

There's a version of the AI procurement conversation that goes like this. The buyer asks for documentation on how the model works. The vendor sends architecture notes, an evaluation summary, maybe a model card. Everyone treats the transparency question as closed. It isn't, because most of what makes a modern model opaque was never the vendor's to hand over. Jenna Burrell made this distinction back in 2016, and it still sorts out about eighty percent of the confusion I see in governance discussions: opacity comes in three forms, and only one of them is a property of the technology itself.

• Withheld

  A contract problem

  The provider won't share weights, training data or evaluation results, commercial confidentiality, sometimes national security. Negotiate it, escrow it, or regulate it away.

• Unreadable

  A hiring problem

  The information is there, but your organisation doesn't have anyone who can interpret it.

• Constructed

  Nobody holds the explanation

  Not you, not the vendor, not the researchers who trained it. The behaviour came out of an optimisation process, not a design document.

Only the third one is a property of the technology, and it's the one that shows up in incident reviews. None of the reasons it persists are fixable with a patch. Neurons aren't concepts: networks routinely encode more features than they have dimensions, packing them as overlapping directions in activation space, so one unit takes part in a lot of unrelated computations and looking at individual components tells you very little. There's no specification to check against either. Normal software traces from requirement to code; a model has only a training objective, a dataset and an optimiser, so the behaviours were never specified anywhere, and there's no document to verify against, only a distribution to test on.

The model's explanation isn't a record of what it did, either. Ask a language model why it answered something and it will tell you, fluently. Turpin and colleagues showed in 2023 that when you plant a biasing cue in a prompt, models change their answers to match it while producing reasoning that never mentions the cue, the explanation is generated by the same process as the answer, an output, not a log. And the same input doesn't reliably produce the same output: sampling temperature, floating-point ordering and batch effects under server load all shift results, so reproducing an incident needs the input, the model version, the sampling parameters and sometimes the serving configuration, and most production logging captures one or two of those.

Opacity isn't the same thing as complexity, which is worth saying because it gets conflated constantly. The 2020 Ofqual grade standardisation model in England was statistically modest and published in outline. It still behaved as a black box for every student affected, because none of them could reconstruct how their own circumstances produced their own grade, and around thirty-nine percent of A-level grades were adjusted down from centre assessment before the model was pulled. A simple system with no explanation pathway is still opaque to the person on the receiving end.

The failure mode that actually costs money starts with calibration. Models are badly calibrated, and calibration gets worse under distributional shift while confidence scores stay high, Guo and colleagues documented the overconfidence in 2017, Ovadia and colleagues showed the degradation under shift in 2019. Read that plainly: a model that has started failing does not report that it has started failing. You only catch it from outside.

• $304M

  Zillow Offers' inventory write-down in Q3 2021, after its valuation model priced acquisitions fine in a stable market, then didn't

• 25%

  of Zillow's headcount cut after the business line closed on 2 November 2021

• 26,000

  families flagged by the Dutch tax authority's fraud-risk system on grounds they couldn't see or challenge

• 1,000+

  children removed from their homes before the Cabinet resigned in January 2021

The number I keep coming back to is Anthropic's circuit tracing work on Claude 3.5 Haiku, which produced a satisfying explanation for roughly a quarter of the prompts examined. That's the state of the art, published by a lab with every incentive for the figure to be higher. Elsewhere, sparse autoencoders trained on identical data learn different features across runs, dead-feature rates above seventy percent have been reported on some models, and DeepMind's interpretability team published negative results on downstream applications in March 2025 and deprioritised the line of work. Mechanistic interpretability is genuinely useful for investigation and debugging. It is not yet a compliance artefact, and no programme plan should assume it becomes one on a convenient timeline.

Meanwhile the law didn't wait. In Dun & Bradstreet Austria, C-203/22, decided by the CJEU in February 2025, handing over a complex algorithmic description was held not to discharge the GDPR Article 15(1)(h) duty, the information has to let the person actually understand and contest the decision. In the UK, section 80 of the Data (Use and Access) Act 2025 replaced Article 22 with Articles 22A to 22D, commenced 5 February 2026: solely automated significant decisions are now permitted for ordinary personal data, conditional on safeguards, information, representations, human intervention, a route to contest. Notice what that does. The question stops being whether an exemption applies and becomes whether you can evidence that contestability actually works, which is a logging and systems-design question, not a legal drafting one. The EU Digital Omnibus, Regulation 2026/1744, deferred high-risk Annex III compliance to 2 December 2027, but the Article 50 transparency obligations applied from 2 August 2026 and the underlying duties are unchanged. The deadline moved. The design work didn't.

Stop trying to make the model explain itself. Constrain what it's allowed to decide, and instrument everything around it.

1. 01

   Keep opaque components off the actuation path

   If a model informs a control action, put a deterministic, independently verifiable envelope between it and the plant, rate limits, physical bounds from first principles, interlocks, reversion to a conventional controller. Assurance then attaches to the envelope, which you can verify properly.

2. 02

   Monitor the input distribution separately from the model's own confidence

   Mahalanobis distance, energy scores, PSI or KS statistics on input features. The signal doesn't come from the same computation that produced the prediction.

3. 03

   Use conformal prediction and let the system abstain

   Distribution-free coverage guarantees, applied to any underlying model. Turns an overconfident predictor into one with a measurable escalation rate.

4. 04

   Give counterfactual explanations for individual decisions

   The minimal input change that would have flipped the outcome. Actionable for the person, requires no disclosure of internals, and maps onto the contestability duty directly.

5. 05

   Log at decision level, not artefact level

   Input payload, model ID and weight hash, feature or prompt version, sampling parameters, output, confidence or conformal set, any human intervention and who made it. Model cards describe the tool. They don't reconstruct a specific decision three years later when someone challenges it.

Scale the requirement to consequence, not complexity. Document routing needs drift monitoring. Benefit determination needs interpretability by default, evidenced safeguards and full lineage. Anything touching physical actuation gets the model in an advisory role only, behind a verified envelope, with independent out-of-distribution monitoring and a documented reversion mode. The same model can be fine in one row of that table and disqualifying in the next.

An opaque model isn't disqualified from a system. It's disqualified from being the only thing standing between a decision and its consequence. That's an architecture decision, made at the point where you define how much authority the model has, not a research problem you're waiting on someone else to solve.