Humans are causal creatures. We explain the world with because: this lever moved that outcome; this decision created that consequence. Statistical learning systems—including large language models—are extraordinarily good at capturing patterns in data. Patterns are not causes. That gap is easy to forget when the interface speaks fluent English.
This post expands on a simple warning I returned to as ChatGPT-class tools spread: do not confuse probabilistic association with understanding of cause and effect—especially when systems may not recognise the consequences of being wrong.
The problem: fluency hides the epistemic gap
When a model answers confidently, it feels like reasoning. Sometimes the answer is even right. The mechanism underneath is still, at core, prediction over patterns learned from data (and whatever tools we bolt on). Correlation-heavy training signals can produce useful behaviour without producing a stable causal model of the world.
That matters when users ask for:
- Medical, legal, or financial guidance
- Root-cause analysis in operations
- Strategy recommendations
- Explanations of why something happened
A system can complete a plausible story that matches common co-occurrences in text—while missing confounders, reversed causality, or the one detail that changes the decision.
Why it matters in the ChatGPT era
Natural-language interfaces democratise access to analytical-sounding output. More people can ask sophisticated questions. Fewer people may feel the old friction that used to force careful method: collecting data, stating assumptions, designing controls, testing interventions.
Prompts become the new method section—and prompts are often underspecified. Without context, the model fills gaps with priors from training data. Those priors may be popular rather than true for your situation.
Bad data makes this worse. If your retrieval corpus or fine-tuning set encodes spurious correlations (seasonality mistaken for causal lift, selection bias in CRM fields, labels from broken processes), the system will repeat them with polish.
There is a second-order risk: systems may not recognise the consequences of wrong answers. A human expert often feels the cost of a bad call in their bones. A model does not stake reputation the same way unless we design workflows that force verification and accountability.
Cause-effect thinking vs probabilistic generation
Human cause-effect reasoning (at its best) seeks interventions: if we change X, what happens to Y? It looks for mechanisms, alternative explanations, and tests.
Probabilistic AI (at its common default) seeks coherence with patterns: what completion is likely given the prompt and context?
Both can be valuable. Mixing them up is how organisations get “AI said so” decision culture.
Useful mental model:
- Use generative systems to propose hypotheses, drafts, and candidate analyses
- Use causal method, domain expertise, and experiments to decide what is true enough to act on
- Use software guardrails to prevent high-impact actions from running on unverified narrative alone
Prompts and context are not a cure-all—but they matter
Better prompting and richer context reduce some failure modes:
- State the decision, constraints, and what would change your mind
- Provide relevant data instead of asking the model to invent it
- Ask for alternative explanations and what evidence would discriminate between them
- Require uncertainty and “what is missing” sections in outputs for high-stakes work
Still, prompting cannot create causal identification out of thin air. If the information needed to establish cause is not in the context—or not knowable from observational text—the model cannot responsibly provide it. The failure is when it provides it anyway.
Practical recommendations
- Label the epistemic status in products: “draft,” “retrieved,” “verified,” “recommended action.”
- Separate narration from action. Generation can draft; execution needs policy and often a human.
- Demand evidence links for factual claims in enterprise assistants (RAG with citations—and teach users citations can still be wrong).
- Watch for spurious metrics in analytics copilots; build checks for confounders and segment mix.
- Train teams on correlation vs causation with examples from your domain—not abstract statistics only.
- Design consequences into the workflow. Review, sampling, and accountability for decisions influenced by AI.
What this means for builders
If you build AI products, resist the temptation to market them as understanding engines. Market them as leverage for experts and operators—with clear limits. Instrument where users accept answers without checking. That metric is a risk indicator.
If you advise leaders, push for decision rights: which calls may be AI-assisted, which require human sign-off, which require experimental validation.
Closing
Correlation does not imply causation. The sentence is old because the mistake is perennial. Generative AI makes the mistake more seductive by wrapping associations in sentences that sound like explanations.
Use the tools. Enjoy the leverage. Keep the causal discipline. And remember: a system that does not bear the consequences of error will not, on its own, be careful enough for the decisions that do.