AI's Unsung Hero: Data Labeling and Expert Evals

Episode notes

Labelbox CEO Manu Sharma joins a16z Infra partner Matt Bornstein to explore the evolution of data labeling and evaluation in AI - from early supervised learning to today’s sophisticated reinforcement learning loops. Manu recounts Labelbox’s origins in computer vision, and then how the shift to foundation models and generative AI changed the game. The value moved from pre-training to post-training and, today, models are trained not just to answer questions, but to assess the quality of their own responses. Labelbox has responded by building a global network of “aligners” - top professionals from fields like coding, healthcare, and customer service, who label and evaluate data used to fine-tune AI systems. The conversation also touches on Meta’s acquisition of Scale AI, underscoring how critical data and talent have become in the AGI race. Here's a sample of Manu explaining how Labelbox was able to transition from one era of AI to another: It took us some time to really understand like that the world is shifting from building AI models to renting AI intelligence.

Related episodes across the Index

Other episodes covering the same guests and topics, from across The B2B Podcast Index.

• The Race to Build the Next Trillion-Dollar AI Chip CompanyTechSurge: Deep Tech Podcast · on Artificial intelligence90 / 100
• Hiring top-tier talent, the importance of adopting an evolutionary mindset, and taking critical feedback to solve technical problems w/ Sergiy Nesterenko @ QuilterEngineering Founders · on generative AI88 / 100
• What Marketers Can Control When AI Changes Everything with Nick Wedewer, VP of Growth Marketing at Hims & HersThe Partnership Economy · on generative AI87 / 100
• Hiring top-tier talent, leveraging open source models, and staying competitive in the age of AI w/ Benny Chen #267The Engineering Leadership Podcast · on generative AI86 / 100
• Less about Models; More about ArchitecturePractical AI · on Artificial intelligence85 / 100
• 321 | How to Build an AI App Without Coding: From Idea to App Store with Replit & Claude by Bryce Rattner KeithleyLeveraging AI · on Artificial intelligence80 / 100

• How AI Is Rewriting the Rules of Cybersecurity | Truffle Security & Socket95 / 100
• Building Self-Accelerating AI with Mirendil65 / 100
• Inside vLLM: The Engine Powering Open-Source AI
• Ideogram’s Open-Weights Image Model and the Future of AI Design
• Building Search for AI Agents with Exa CEO Will Bryk