AI benchmarks are broken. Here’s what we need instead.

AI Summary

A Technology Review article published March 31, 2026, authored by Angela Aristidou — a professor at University College London and faculty fellow at the Stanford Digital Economy Lab and Stanford Human-Centered AI Institute — argues that current AI benchmarking methods are fundamentally flawed and misrepresent real-world AI performance. Aristidou, who has studied real-world AI deployment since 2022 across small businesses, health, humanitarian, nonprofit, and higher-education organizations in the UK, the US, and Asia, found that AI systems achieving high benchmark scores — such as FDA-approved radiology models boasting superior accuracy over expert radiologists — routinely underperformed or caused inefficiencies once embedded in actual hospital workflows across California and London. The core problem identified is that benchmarks evaluate AI on isolated, individual tasks with clear right-or-wrong answers, while real-world deployment involves multidisciplinary teams, evolving workflows, and extended time horizons — conditions no current standardized test captures. This gap leads to what Aristidou calls the 'AI graveyard,' where adopted AI tools are abandoned after failing to deliver promised productivity gains, resulting in wasted financial and technical resources and eroding organizational and public trust. As an alternative, Aristidou proposes 'HAIC benchmarks' — Human–AI, Context-Specific Evaluation — which reframe testing across four dimensions: shifting from individual to team-level performance, extending time horizons beyond one-off tests, expanding outcome measures to include coordination quality and error detectability, and accounting for upstream and downstream system effects. A UK hospital system evaluated between 2021 and 2024 and an 18-month humanitarian-sector case study are cited as early examples of organizations already piloting this longitudinal, workflow-integrated approach.