Why We Trust Engines We Don’t Understand and AI We Can’t Explain
Dec 13, 2025
There is a clamor today for the "opening of the black box" of Artificial Intelligence. As algorithms increasingly influence our lives—from loan approvals to medical diagnoses—the reflexive societal demand is for radical transparency. The assumption is straightforward: if we can see how the machine thinks, we will trust it.
This assumption is flawed.
While well-intentioned, the drive to expose the inner workings of neural networks to the general public misunderstands the fundamental psychology of technological trust. Trust is rarely built on a forensic understanding of mechanics. As the adage goes, people do not trust their cars because they understand the thermodynamics of an internal combustion engine.
We trust these complex systems because they are presented to us through intuitive, predictable abstractions. We trust the steering wheel, not the transmission linkage. The urgent challenge for AI adoption is not teaching the world data science; it is designing abstraction layers that feel natural and predictable.
The Psychology of "Functional Trust"
Sociologists and psychologists have long studied how humans interact with complex "expert systems." We rely on vast networks of expertise—pilots, surgeons, structural engineers—without possessing their knowledge.
We bridge this knowledge gap through what sociologist Anthony Giddens calls "functional trust" (or relying on "abstract systems"). We don't look for the how; we look for consistency between action and outcome. If I turn the key, the car starts. If I drag a file to the trash, it deletes.
When a technology becomes too transparent, it can actually decrease trust for a layperson. In a 2021 study on explainable AI (XAI), researchers found that for non-expert users, detailed technical explanations often led to "information overload," causing users to doubt their own ability to use the tool. Presenting a non-expert with raw probabilistic weightings of a neural network is akin to a doctor handing a patient their raw blood work spectrography instead of a diagnosis. It doesn't empower; it overwhelms.
Trust comes from the perceived reliability of the interface, not the inspectability of the code.
The Abstraction Layer as the Locus of Trust
If raw mechanics are overwhelming, the abstraction layer—the interface—is where trust is won or lost. In technology, an abstraction is a simplified model that hides underlying complexity to allow for easier interaction.
The Graphical User Interface (GUI) is perhaps the greatest trust-building abstraction in history. When you drag a file into a folder on your desktop, you aren't actually moving physical items; you are executing complex commands to alter bits on a storage drive. The "desktop" metaphor is a useful fiction that allows you to predict the outcome of your actions reliably.
Don Norman, author of The Design of Everyday Things, argues that users develop "conceptual models" of how systems work based on their interfaces. If the system’s behavior aligns with the user’s conceptual model, trust is established. If an AI interface behaves unpredictably—even if its internal logic is sound—the conceptual model breaks, and trust evaporates.
The goal of AI design, therefore, must be to create interfaces that allow users to form accurate conceptual models without needing to understand the underlying math.
The Audience Divide: Regulators vs. Users
This is not to argue that transparency is obsolete. It is merely misplaced. We must distinguish between "mechanistic transparency" (how it works) and "behavioral transparency" (what it does and why).
For Regulators (Mechanistic Transparency) | For Everyday Users (Behavioral Transparency) |
|---|---|
There is an absolute need for deep, technical transparency for a select group of experts. We need auditors to ensure models are not illegally discriminatory and safety inspectors to verify autonomous vehicle code. Frameworks like the EU AI Act rightly demand technical documentation for high-risk systems. This is the equivalent of automotive safety standards—necessary for the system to exist, but irrelevant to the daily driver. | The average user needs to understand the system’s intent, capabilities, and limitations. They need to know, "Why did the AI recommend this movie?" (e.g., "Because you watched 'The Matrix'"), not "What were the node activation weights?" |
Designing for Probabilistic Systems
The challenge in designing these abstractions for AI is significantly harder than for cars. A car engine is deterministic; if it is working, turning the wheel left always turns the car left.
Generative AI is probabilistic. It makes guesses based on patterns. A chat interface that looks authoritative but hallucinates facts is an example of a failed abstraction layer—it promises reliability it cannot deliver.
Building trust in probabilistic systems means designing interfaces that communicate uncertainty naturally:
Confidence Signaling: Instead of a definitive answer, an interface might signal, "I’m 90% sure about this," similar to a weather forecast.
Guardrails as Interface: When ChatGPT refuses to answer a dangerous query, that refusal is an abstraction layer functioning correctly, demonstrating predictability and adherence to social norms.
Closing Thoughts
The pursuit of universal transparency is a dead end for mass adoption of AI. It attempts to solve a psychology problem with an engineering solution.
If we want society to reap the benefits of AI, we must stop trying to force everyone to become a mechanic. Instead, we must focus on building better dashboards. Trust is not built by revealing the chaos of the engine room; it is built by the smooth, predictable hum of a machine that does what its interface promises it will do.
Co-authored with Gemini 3.0