OpenAI Research Finds That Even Its Best Models Give Wrong Answers a Wild Proportion of the Time

Hello!

BS Generator

OpenAI has released a new benchmark called “SimpleQA” to measure how accurately its own and competing AI models answer factual questions.

The results highlight serious limitations. In 2026 evaluations, OpenAI’s most advanced model at the time, o1-preview, achieved only a 42.7 percent success rate. This means the leading large language model was more likely to give an incorrect answer than a correct one—an outcome that raises concerns as AI tools become integrated into daily workflows.

Why Accuracy Matters

Other frontier models performed even worse on the same benchmark. Anthropic’s Claude-3.5-sonnet reached just 28.9 percent accuracy. While the model often admitted uncertainty and declined to answer—arguably a safer behavior than confident errors—the overall numbers remain low.

OpenAI also observed that its models frequently overestimate their own knowledge. This overconfidence can lead to fluent but entirely fabricated responses, commonly referred to as “hallucinations.”

Real-World Impact

Despite these known shortcomings, organizations continue to adopt LLMs at scale. Students use them for assignments, developers rely on them for large codebases, and institutions deploy them in sensitive environments. One hospital transcription system built on OpenAI technology was recently found to introduce frequent inaccuracies when summarizing patient conversations.

Law-enforcement agencies across the United States are also exploring AI tools, prompting warnings about potential false accusations and the amplification of existing biases.

OpenAI’s SimpleQA findings underscore that current large language models still struggle to deliver reliable factual answers. The prudent approach remains to treat any LLM output with caution and verify key claims against primary sources.

Whether scaling training data further will close the accuracy gap continues to be an open question for researchers and industry leaders alike.

Thank you!
Join us on social media!
See you!