Value Driven Data Science

The episode is packed with substantive ideas about decision-making under uncertainty, measurement inversion, the lens model, and expert aggregation methods. However, it occasionally retreats into broad conceptual territory without drilling into new specifics - e.g., the discussion of AI personas is exploratory rather than evidence-based, and some segments recycle well-known principles (e.g., consensus being worse than averaging). Most claims are supported with reasoning, but the density could be higher if more concrete examples were threaded through.

“Highly uncertain variables are actually easier to reduce uncertainty on. Because they're uncertain. It's convenient. It's the neat little convenient aspect of all of this that the highest information value variables are actually also easier to reduce uncertainty on because they're so uncertain.”

“The measurement inversion. We've observed this, I think, in every industry we've ever done consulting it. It's almost like everybody's systematically measuring all the wrong stuff.”

Hubbard's Applied Information Economics framework and the lens model are genuinely original contributions to decision science, and the framing of measurement inversion as a systemic problem across industries is compelling and non-obvious. However, much of the supporting material draws from well-established research (Kahneman, decision theory, calibration studies) and the episode doesn't push into truly contrarian territory - it's more a careful reframing of existing ideas than a radical departure. The AI personas discussion hints at originality but remains speculative.

“I don't know how it doesn't affect the GDP. Organizations making major decisions are spending more time measuring things that are statistically less likely to actually improve a decision while ignoring the most uncertain things that would have the biggest sway on the decision.”

“If you know almost nothing, almost anything will tell you something. That's what the math actually means about this which is contrary, I think, to a lot of intuition.”

Douglas Hubbard is exceptionally well-calibrated for this episode. He is the founder of Hubbard Decision Research with 35+ years of management consulting experience, author of multiple seminal books on decision-making and measurement, and brings both practitioner credibility and rigorous academic grounding. He's not a career podcast guest but a genuine practitioner and thought leader who has deployed these methods at scale across industries. His willingness to discuss unpublished experiments (the 2 million virtual teams analysis, recent AI calibration work) demonstrates active, ongoing practice.

“We've done this a lot for a lot of different kinds of decisions with lots of variables in the decision model dozens or sometimes a few hundred variables in a decision model.”

“We have a bunch of data on that. The answer is, it can be about as good as a human and forecasting future events.”

Hubbard provides concrete examples (duplicate pair analysis in project estimation, the 2 million virtual teams experiment with ~2,400 cases matching a specific pattern, 50/50 base rate for trivia questions, the temperature parameter for AI consistency, solar panel ROI optimization) that ground his claims. However, many examples remain illustrative rather than deeply quantified - exact ROI figures, specific companies, or detailed metrics are sparse. The discussion of how individuals answer trivia questions differently (the 20% unexplained variation) is precise, but broader claims about organizational measurement practices lack named cases or hard numbers.

“By the time you get down to 91, you forgot you already answered that exact combination of quantities and those parameters, and you probably put down a slightly different answer. We find that based on how much of the variation in judgment... you can explain about 20% of the variation in your judgments. Just as personal infancy.”

“So there was plenty of data on these things. Most of these combinations had a few hundred examples. There'd be rare examples where one person said it 10% likely, and two other people said it was 90%.”

Hayes asks strong framing questions and demonstrates genuine intellectual engagement - she confirms her own experience against Hubbard's framework and pushes on specific topics like the lens model and AI integration. She creates natural follow-ups and shows she has read Hubbard's work carefully (bookmarks reference). However, the conversation rarely becomes adversarial or deeply probing; Hayes is largely validating rather than challenging. She could have pressed harder on the practical friction of implementing AIE (e.g., how do you convince skeptical stakeholders to measure uncertain variables first?), the limits of his solar panel example, or gaps in the AI persona reasoning. The flow is warm but not sharp.

“Before you were saying, why is data science so focused on the applications where basically we're swimming in data? I think part of that happened because of the evolution of data science.”

“How to measure, anything was written before the big AI wave. One thing that struck me while reading this was that the, a i a approach is something that firstly, it would be much harder to automate using AI than machine learning would, but at the same time, it's something that I can see AI supporting very well”