Training Data · 2026-07-29 · 49 min
Jerry Tworek led reasoning at OpenAI, convinced that scaling reinforcement learning was the path to AGI. Rohan Anil co-led Gemini pre-training and built the Shampoo optimizer. Now they've teamed up at Core Automation on a contrarian premise: the transformer has carried us as far as it can, and the bottleneck to smarter systems is no longer scale - it's the architecture itself. The missing capability is continual learning, models that adapt at test time, which transformers can't do. In-context learning taps out fast (Codex needs compacting after ~20 minutes) and fine-tuning invites catastrophic forgetting. Rohan argues pre-training and RL should be optimized end-to-end, and that transformers spend computation inefficiently. They lay out why the largest labs won't chase alternatives while locked in the coding-agent race, and why building the world's most automated lab starts with automating kernel generation - the one place frontier models still lose to a high-taste human. Hosted by Sonya Huang and Pat Grady, Sequoia Capital
Chai Discovery's Bitter Lesson: Drug Design Is Another Scaling Problem95 / 100
Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis95 / 100
Factory's Matan Grinberg: The Coming ‘Dark Factory’ Where Software Builds Itself
Anthropic's Katelyn Lesse & Angela Jiang: Building an Ecosystem, not a Walled Garden
Inside Zipline's Autonomous System: 140M Miles, Zero Incidents