Machine Learning Street Talk · 2026-05-04 · 1h 53m
Beth Barnes and David Rein on the one graph that ate the AI timelines discourse, and why the two people who built it are the most careful about how you read it.**SPONSOR**Prolific - Quality data. From real people. For faster breakthroughs. Barnes and David Rein from METR on the one graph that ate the AI timelines discourse, and why the people who built it are the most careful about how it gets read.Beth founded METR after leaving OpenAI alignment. David is first author on GPQA and co-author on HCAST and the METR Time Horizons paper. Together they built the measurement Daniel Kokotajlo called the single most important piece of evidence on AI timelines: the log-linear line of "how long a task a frontier model can complete at 50% reliability" vs release date.The conversation opens on reward hacking. Current models can articulate in chat why a behaviour is undesired and then execute it anyway as agents. From there: construct validity, Melanie Mitchell's four-problem taxonomy, and the ARC-AGI 1-to-2 collapse as a worked example of adversarially-selected benchmarks regressing once labs target them. Beth's counter: METR deliberately does not adversarially select.
The Benchmark With No Instructions - ARC-AGI-3 (winning team!)85 / 100The Thermodynamic AI Computing Chip - Thomas AhleHe won a Nobel here for AlphaFold. Then he left. - John JumperWhen AI Decides You're a Threat - Brad CarsonIntelligence is collective, not artificial - Prof. Michael I. Jordan (UC Berkeley / Inria)