Government Test Finds That AI Wildly Underperforms Compared to Human Employees

Hello!

Generative AI continues to lag behind humans when it comes to summarizing complex information, according to the findings of a 2026 trial conducted for the Australian Securities and Investments Commission (ASIC) and reported by Australian outlet Crikey.

The trial, run by Amazon Web Services on behalf of the corporate regulator, served as a proof-of-concept exercise to explore how generative AI might perform in real business environments. The results, however, painted a sobering picture of the technology’s current limitations.

Clear Performance Gap

In blind evaluations, AI-generated summaries of official government documents achieved an aggregate score of just 47 percent against the trial’s assessment criteria. Human summaries prepared by ASIC staff, by contrast, scored 81 percent—highlighting a substantial quality difference that echoes broader concerns about the reliability of today’s generative AI tools in professional settings.

Signature Shoddiness

The experiment used Meta’s open-source Llama2-70B model. Although not the most recent release available in 2026, the 70-billion-parameter system remains a robust large language model. It was tasked with summarizing submissions to a parliamentary inquiry, focusing specifically on references to ASIC and including accurate page citations. ASIC employees produced parallel human summaries for comparison.

Five independent evaluators then assessed both sets of summaries without knowing their origin. Even after the exercise concluded and participants learned an AI had been involved, three of the five evaluators said they had already suspected they were reading machine-generated text—a telling indicator of the outputs’ distinctive flaws.

Sucks On All Counts

Across every evaluation criterion, the AI summaries underperformed their human counterparts. Most notably, the model failed to supply correct page references for the information it extracted—an issue the report acknowledges could be mitigated through further prompt engineering or retrieval-augmented techniques. More fundamental problems included an inability to grasp nuance or context and a tendency to emphasize irrelevant details while overlooking key points.

The AI outputs were also frequently described as “waffly” and “wordy,” containing redundant or off-topic information. Evaluators concluded that relying on such summaries would likely create additional work downstream, primarily because of the extensive fact-checking required. This outcome directly challenges the often-cited advantages of generative AI—namely, cost and time savings—suggesting that, for many organizations, the technology may not yet deliver practical value in document-intensive workflows.