Scientists Preparing “Humanity’s Last Exam” to Test Powerful AI

Hello!

​AI experts are inviting submissions for what they describe as the “hardest and broadest set of questions ever” — an ambitious effort to challenge today’s most advanced AI systems and those still on the horizon.

A New Benchmark for Advanced AI

As Reuters reports, the initiative — titled “Humanity’s Last Exam” — is being crowdsourced by the Center for AI Safety (CAIS) and Scale AI. The data-labeling company raised $1 billion over the summer, reaching a valuation of $14 billion.

The call for submissions opened just one day after OpenAI released its o1 model preview. CAIS executive director Dan Hendryks observed that o1 has already “destroyed the most popular reasoning benchmarks.”

From 2021 Tests to Today’s Challenges

Back in 2021, Hendrycks co-authored two papers outlining AI evaluation methods that would test whether models could outperform undergraduates. At the time, systems responded nearly at random. Today, as Hendrycks notes, current models have “crushed” those earlier benchmarks.

Abstract Reasoning and Confidentiality

While the 2021 criteria focused mainly on math and social studies, “Humanity’s Last Exam” will emphasize abstract reasoning to increase difficulty. To prevent answers from entering future training datasets, CAIS and Scale AI plan to keep the final test confidential.

​Submissions are due November 1. Specialists across diverse fields — from rocketry to philosophy — are encouraged to propose questions that would challenge even experts outside their own domains. After peer review, selected contributors will receive co-authorship on the associated research paper and prizes of up to $5,000, sponsored by Scale AI.

The organizers have made one clear exclusion: no questions related to weapons will be accepted, citing safety concerns about AI access to such information.

---

Also read:

• Your Quick Post-LASIK Surgery Guide: From Day 1 to Long-Term Care
• Brett Favre Says He Has Parkinson’s, Likely From Countless Concussions
• The Role of License Plate Lookup in Reducing Fraud​​​​​​​

---

Thank you!
Join us on social media!
See you!