Here is the uncomfortable part. In a study by researchers from three French and Italian universities, access to AI advice collapsed people’s willingness to say “I don’t know” from 44% to 3%. At the same time, their accuracy dropped from 27% to 9%.

And then comes the twist that makes this more than a curiosity. Confidence rose from 30% to 76%. As the researchers summarized it, “People became much worse, the accuracy was only one third, but they were twice as confident.” So the system did not just nudge beliefs. It changed how people behave when they are uncertain: they stopped admitting uncertainty, even as they got more wrong.

If you manage teams, products, or risk, this is the kind of effect that can quietly poison an organization. Many modern workflows already rely on some mix of human judgment and AI assistance: customer support drafts, analytics explanations, forecasting tools, underwriting guidance, coding copilots, and internal search. The headline numbers are blunt, but the mechanism is intuitive. When people feel empowered by an answer, they treat that answer as permission to stop thinking. “I don’t know” disappears. Guessing becomes normal.

This matters because confidence is not the same thing as correctness, and organizations often treat the two as if they travel together. People with higher confidence are more likely to escalate decisions, sign off on outputs, or defend them in meetings. That means an AI system that increases confidence can increase error rates at the exact moments you need caution the most. In the study, accuracy went from 27% to 9%, which is a dramatic drop, but the equally dramatic change is behavioral: uncertainty disclosure fell from 44% to 3%.

In plain English, the AI advice seems to have re-trained the users’ internal safety switch. The willingness to pause and say “I don’t know” is one of the most important defenses against making confident mistakes. Remove it, and the user’s confidence becomes a management problem. Not because people are irrational, but because the incentives for decisiveness and speed can turn a confidence boost into an operational shortcut. Faster decisions are not always better decisions, and the study suggests AI advice can make people more likely to commit.

Now layer in how boards and regulators have been thinking about AI recently. Even where rules differ by jurisdiction, the repeated theme is governance: systems should be monitored for performance, audited for risk, and deployed with clear accountability. Confidence inflation is a governance red flag because it can distort human oversight. If an AI tool changes what users report about their uncertainty, your internal QA and risk controls might look fine on paper while accuracy deteriorates in practice. You could see high “user satisfaction” or low “I’m not sure” frequency, while error rates quietly rise.

There is also a second-order implication for how you measure success. Many teams evaluate AI copilots with metrics that capture usage and subjective trust, like adoption, perceived usefulness, or user confidence. But this study suggests you should track accuracy and uncertainty disclosure explicitly. The “I don’t know” drop from 44% to 3% is a metric you can build into testing. If your pilots show that people stop signaling uncertainty, you may be watching a reliability risk take root.

So what should decision-makers do with this? Start by separating confidence as a psychological state from confidence as a decision quality signal. The study shows confidence can rise even as accuracy collapses, going from 30% to 76% while accuracy falls from 27% to 9%. That is exactly the mismatch that can lead to overreliance. In real deployments, especially high-stakes ones, you want guardrails that keep uncertainty visible. If your UI or process encourages users to pick an answer rather than admit uncertainty, you are likely to reproduce the effect.

Strategically, this is a wake-up call for executives deploying AI advice in customer-facing, operational, or compliance-adjacent workflows. It also matters for capital allocators and product leaders who shape governance expectations across companies. The future of AI at work is not only about improving answers. It is about preserving the human ability to recognize when the model might be wrong. Because when “I don’t know” disappears, the organization becomes more confident and, in the study, less accurate. That is a recipe for preventable mistakes at scale.