Value Driven Data Science · 2026-06-17 · 17 min
Key moments - from our scoring
Substance score
73 / 100
Five dimensions, 20 points each
Most data scientists default to machine learning for every problem, but the most critical business decisions often involve sparse data where ML fails entirely. Douglas Hubbard, author of How to Measure Anything, demonstrates counterintuitive Bayesian techniques that work with minimal samples. The episode covers the urn-and-marbles problem (showing a single sample can predict majority outcomes at 75% accuracy), Laplace's rule of succession (estimating probabilities from small counts using the formula: 1 + hits / 2 + sample size), beta distributions for proportion estimation, and the rule of five (which shows 93.75% confidence that a population median falls between the min and max of five random samples). Hubbard emphasizes that asking "has this been measured already?" is often more valuable than collecting new data - using Google Scholar, ResearchGate, or AI to find existing research and methods before building custom solutions. This approach is most relevant for data scientists supporting high-stakes decisions with limited historical data, cybersecurity risk assessments, product launch predictions, and automation ROI calculations.
Using Bayesian logic on a uniform prior (0-100% of marbles could be red), if you draw one red marble, there's a 75% chance the urn contains a majority of red marbles, because the probability density is higher in regions where red dominates the urn's composition.
Laplace's rule is 1 + hits / 2 + total sample size, and you use it to estimate the probability that the next occurrence will match your observed hits. For example, if 2 data breaches occurred across 10 companies over 2 years (20 company-years), the chance the next company experiences a breach is 3/20 or 15%.
If you randomly sample five items and measure a continuous value (like minutes spent on a task), there is a 93.75% probability that the true population median falls between your smallest and largest samples, regardless of the population size.
If a Laplace estimate feels wrong, you likely have additional information beyond your immediate observations - such as industry benchmarks, longer historical records, specific risk mitigations, or known threat trends - which you should layer in to refine the estimate.
Ask whether the problem has already been measured: search Google Scholar, ResearchGate, and call authors directly; existing studies often provide methodology you can adapt or findings that make new data collection unnecessary.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode packs multiple concrete statistical concepts into 17 minutes: Laplace's Rule of Succession, the Urn problem, Beta distributions, and the Rule of Five. Each technique is explained with worked examples (data breach estimation at 15%, median estimation at 93.75%). However, some runtime is spent on foundational setup and repetition, and the practical applicability to typical B2B data scientists could be more explicitly connected.
if I drew a single marble out of that urn and it was red, what's the probability that the majority of marbles in that urn are red? Actually, the way the math works is it's 75%.
one plus hits over, two plus hits, plus misses. If I had two breaches out of those 10 companies over a two year period, that means I'd have one plus two hits over two plus 18 misses. So it'd be three over 20 or 15% chance.
The core insight - that small samples can yield actionable probability estimates through Bayesian methods and distribution theory - is genuinely counterintuitive and contrarian to the prevailing ML-first dogma in data science. However, these techniques (Laplace's Rule, Beta distributions) are classical statistics, not novel. The value lies in repositioning them as practical tools for sparse-data scenarios rather than reinventing theory.
Data scientists are trained to reach for machine learning almost by default, but machine learning automation works best when data volume is high. The problem is that for the most critical decisions that organizations have to make, data volume is extremely [limited].
machine learning falls apart and data scientists often struggle to support business leaders. But there are techniques the data scientists can use to create value in these situations.
Douglas Hubbard is a recognized applied measurement expert and author of foundational texts on measuring intangibles and decision theory. He demonstrates real-world application experience (crash testing, cybersecurity) and practitioner insight rather than pure academia. However, he's primarily known for theoretical frameworks rather than operating at scale in a specific industry domain, limiting caliber slightly.
Douglas Hubbard, author of How to Measure Anything, finding the Value of Intangibles in Business and the Failure of Risk Management, Wyatt's Broken, and How to Fix It.
Suppose I'm considering implementing some new controls in cybersecurity...You go around, you randomly select from the office map or something. Locations...And you find five or six people.
The episode uses concrete worked examples: crash testing, missile failure rates, cybersecurity post-it notes (5 samples, 1 hit), data breaches (2 hits in 20 company-years = 15%), time-spent estimation (5 samples: 15 - 40 minute range). These are specific scenarios with actual numbers and probabilities. However, examples remain somewhat hypothetical; there are no named companies, real case studies with outcomes, or actual business impact data.
if I'm trying to crash test cars. Or I'm trying to work out the launch failure rates of missiles. And suppose it's a brand new design and I have as much uncertainty as the earn a mystery example where I think the failure rate could be anywhere between 0% and a hundred percent.
you go around and you gather information. How many minutes per day people spend in this activity? Out of five people, the lowest number was 15 minutes, and the highest number was 40 minutes.
Dr. Hayes asks generally solid follow-up questions (e.g., 'Can you give us an example in business context?', 'What's your go-to technique?') and demonstrates engagement by offering geometric intuition on the rectangle-diagonal visualization and citing her own teaching experience. However, the conversation lacks pushing back, challenging assumptions, or exploring limitations. Hayes affirms Hubbard's points rather than probing edge cases or business friction.
You can say that because if you [draw] a rectangle which would be your uniform distribution...split it in half diagonally and colored in which bits were red and which bits are green...the side that's got more red than green, that's gonna be 75% colored in red.
Out of all the techniques that you've mentioned, I've actually found that one to be the most useful in practice. What's the one technique that you find to be your go-to technique?
Computed from the transcript - who did the talking, and the words that came up most.
In high-stakes decision-making, waiting for more data is often not an option. Yet many data scientists assume that without a large dataset, meaningful analysis is impossible. The good news is that rigorous, quantitative analysis is possible with far less data than most data scientists realise - in some cases with just a single datapoint. In this Value Boost episode, Douglas Hubbard joins Dr Genevieve Hayes to share practical techniques from How to Measure Anything that data scientists can start using right now to support high-stakes decisions when observations are scarce and every data point counts. In this episode, you'll learn: Why a single observation reveals more than you think [01:58] How Laplace's Rule of Succession lets you estimate probabilities from tiny samples [08:25] The Rule of Five and what it reveals about small sample statistics [12:08] The simplest and most overlooked technique for reducing measurement uncertainty [14:07] Guest Bio Douglas Hubbard is the founder and president of Hubbard Decision Research and the creator of Applied Information Economics.
Transcribed and scored by The B2B Podcast Index.
Dr Genevieve Hayes: Hello and welcome back to Value Driven Data Science, where data professionals become strategic experts. I'm Dr. Genevieve Hayes, and I'm here again with Douglas Hubbard, author of How to Measure Anything, finding the Value of Intangibles in Business and the Failure of Risk Management, Wyatt's Broken, and How to Fix It. Last week, Doug and I discussed how data scientists can add value when data is limited, using Doug's applied information economics framework Today in this Value Boost episode, we're gonna dig even deeper into that topic and discuss simple techniques that you can start using right now to get the most out of limited data.
Welcome back, Doug. Douglas Hubbard: Yeah, thanks for having me back. Genevieve. Dr Genevieve Hayes: Data scientists are trained to reach for machine learning almost by default, but machine learning automation works best when data volume is high.
The problem is that for the most critical decisions that organizations have to make, data volume is extremely. In these situations, machine learning falls apart and data scientists often struggle to support business leaders. But there are techniques the data scientists can use to create value in these situations. We touched on these techniques briefly in last week's episode, but in today's episode we're going to dig deeper and discuss some simple techniques that you can start using right now to get the most out of limited data.
Now this is something you discuss in your book, how to Measure Anything. Can you give us some examples of some of these techniques that data scientists can use when data is so sparse that it seems like there's no way in which they can support decision makers? Douglas Hubbard: There's some ideas that might seem counterintuitive to even the data scientists who've had a lot of training in their field. Possible that I could estimate a probability based on a single sample well.
Yeah, it, you can change the probability of something being true based on a single sample. Here's an example of this. We call it the earn a mystery example. So you've got a warehouse full of thousands of urns, and these urns are filled each with a million marbles.
And each of the marbles in an urn are either red or green. What varies from urn to earn is the proportion of red marbles. In fact, they're uniformly distributed between 0% red to a hundred percent red, and every proportion is equally likely so I randomly select a single earn. I randomly select a single marble out of that urn.
Now, if I ask a room of pretty sophisticated people, including biostatisticians, sometimes actuaries, sometimes data scientists, what's the probability of that marble that I do at random being read? They'd say 50% and their intuition is correct in that case. All right. Now the next question is, if I drew a single marble out of that urn and it was red, what's the probability that the majority of marbles in that urn are red?
Some will say things like some infinitesimal amount over 50%, because it's just a single marble out of a million. Actually, the way the math works is it's 75%. Now, one way to prove that is a very simple little Bayesian proof, but some people actually sat down and wrote simulations and then realized after the simulation. Yeah, that's right.
The single marble actually correctly predicts the majority color 75% of the time. Dr Genevieve Hayes: You can say that because if you. Are a rectangle which would be your uniform distribution. And then split it in half, diagonally and colored in which bits were red and which bits are green.
And then draw a line at the 50% mark. Douglas Hubbard: Right. Dr Genevieve Hayes: On the side that's got more red than green, that's gonna be 75% colored in red. So that's how it works, isn't it?
Douglas Hubbard: That's right. Dr Genevieve Hayes: Can you give us an example of how you could use that single sample approach in a business context? Douglas Hubbard: Yeah, sure. Suppose I've got really expensive samples, like I'm trying to crash test cars.
Or I'm trying to work out the launch failure rates of missiles. And suppose it's a brand new design and I have as much uncertainty as the earn a mystery example where I think the failure rate could be anywhere between 0% and a hundred percent. That would be a huge amount of uncertainty. Now, fortunately, when they're crash testing cars or missiles, usually they've reduced a lot of uncertainty beyond that.
But when you have these really expensive samples and the first one flies. That does tell you something about the likelihood the next one will fly, or that the next car crash will survive a front end collision, et cetera, and the crash test dummies will have lived or something, that's a real life business example. Another one might be if you went past even just one sample, but to a few samples. Suppose you're considering implementing some new controls in cybersecurity.
And the benefit of these new controls depend on how well people follow certain manual requirements. Right now, nobody should be putting their password on a post-it note on their screen. So you go around, you randomly select, from the office map or something. Locations.
You don't just wander around. And you just look at them and you find five or six people who. You saw their screens and out of five people, one of them had a post-it note on their screen. Does that tell me anything at all?
It would, if I had lots of uncertainty. If my uncertainty really was zero to a hundred percent of the entire office building has post-it notes on their screen, that sample of five told me something. So that sample of five is actually a beta distribution problem. You know how in beta distributions,, they've got the alpha and beta parameters that describe the shape.
So I call them hits and misses. So you have these two parameters in a be distribution. And if you have a uniform distribution from zero to one it's like you get a free hit and a free miss. You set alpha and beta each to one.
So you start with a free hit and a free miss. That actually draws that uniform distribution on a beta distribution. If you sample five things and you get one hit and four misses. So one post-it note on a screen and four screens, it didn't.
So the hit. Gets added to the alpha and the misses get added to the beta. So that tells you a probability distribution of the proportion of people in the office with post-it notes on their screens. Now if I wanted to ask a different question, and this is really very relevant for a wide variety of problems.
What if I'm not just estimating the proportion of screens with the post-it notes on the screen? Instead, what I'm trying to estimate is the chance that the next one I randomly select will have its post-it note on the screen. Now, this is actually a pretty interesting problem that applies in a lot of situations. If I say that I looked at 10 companies over a two year period, so my company and nine competitors in my industry over a two year period.
And I'm looking at the number of data breaches that have happened in those 10 companies over two years. And what I'm interested in is the chance that my company will have a data breach next year. So I've got a sample of 20 company years of data, now. Of those 10 companies, over a two year period, I had, let's say, two data breaches.
So two hits and 18 misses. So two plus my free hit of one and 18 plus my free miss of one. So I get three and 19. That would be my beta distribution if I estimated the population proportion.
But if I just wanted to estimate the chance that my firm or any given firm will have a data breach next year, I would use Laplace's rule of succession. Now, the beta distribution has this whole big formula. There's a bunch of math behind it, but the mean of the beta distribution couldn't be simpler. It's laplace's rule.
It's one plus hits over, two plus hits, plus misses. If I had two breaches out of those 10 companies over a two year period, that means I'd have one plus two hits over two plus 18 misses. So it'd be three over 20 or 15% chance. Now that's what we would call a naive Bayesian.
Those all start with the assumption that before that observation, I knew almost nothing about the. Proportion of companies that would be hit by a data breach in the following year. And I just started with the idea that it's somewhere between zero and a hundred percent uniform distribution. That's almost like saying, I really have no idea so it's a very conservative assumption about a prior or what they might call an uninformative prior.
So laplace's rule, you can do that in your head. One plus hits over, two plus sample size. That's the chance that the next one will have this. Situation.
If I drew eight marbles in a row out of an urn in that previous situation, the urn and mystery problem, and I drew those eight marbles and they were all green, I know that doesn't mean that there's no chance of the next marble being red, what does Laplace's rule tell me? Laplace's rule never gets to a hundred percent or 0%. It can approach them, but it never gets to them. Based on my prior, if I drew eight green marbles in a row.
If there's a 10% chance that the next one I'd draw be red. Now if I keep drawing green marbles, I'll get closer and closer to zero, but it'll never actually reach zero. That's how Laplace's rule works. Dr Genevieve Hayes: Yeah, it reminds me of a problem I used to set my students when I was teaching them Bayesian statistics.
What is the probability that the sun will rise tomorrow given that it's risen every day forever? Douglas Hubbard: Yeah that's actually the example Lalas used was the sunrise problem. That was the Laplace's role of succession. He was using the sunrise tomorrow.
And that's an interesting one because now we know a lot more about mechanics of the solar system, so we actually have other information. But if you literally knew nothing. Other than you've observed the sun coming up and you had no idea about oral mechanics or what the sun was or anything. You just saw that every day.
Then, Laplace's rule would be the approach to take. What's interesting is that when people say LA Place's rule is too naive, what they mean is they actually have even more data. So they can improve on that and say, we already made a naive estimate and if you're saying it doesn't sound right, it's because you have more data than just your observations of the last two years of these 10 companies or whatever it is, if you haven't had over an eight year period in your company a data breach, and you can.
Collude that there's a 10% chance of it happening next year, and you say, that doesn't sound right, that sounds way too high or way too low. I would argue that the only reason it seems that way is because you actually have more data than your own immediate observations of the last eight years. You have information about how frequent it is in other companies. You have data that goes back.
More than eight years, 'cause your colleagues have been there longer. You have information about specific risk mitigations that have been implemented in the last eight years. You have information about an increase in certain kinds of attacks in your industry. All of those actually are additional information that cause you to further modify the naive estimate.
So that comes from Laplace's rule of succession. If I were to estimate something that's not a discrete probability of a yes, no kind of outcome, but let's say a continuous value, I had this tool called the Mathless Table in the first book. Dr Genevieve Hayes: Yeah. It's really useful.
Douglas Hubbard: The first row on the mathless table is the rule of five. If you randomly sample five of something, suppose you're in the same office building and you're trying to figure out how much time people spend in some activity because it's something you're trying to automate, and if you automate it, you're trying to work out the benefits of this automation and it's gonna be this expensive investment. So you need to figure out how much time people are spending in this one thing that you're automated.
, So you go around and you gather information. How many minutes per day people spend in this activity? Out of five people, the lowest number was 15 minutes, and the highest number was 40 minutes. And you randomly selected those five people out of the entire organization.
It could have been 20,000 people in the organization. Doesn't matter. What's the chance that the median of the entire population of 20,000 or 200,000, whatever it is, would fall between the smallest and the largest of that random sample of five? Dr Genevieve Hayes: And I know from reading your book that it's a very high probability.
Douglas Hubbard: Yeah, it's 93.75%. That's right. And the reason to think about it is you go through this process.
You think if I randomly select a person out of the entire company, what's the chance that they would be below the media? By definition, 50%. The only way that the median of the entire population wouldn't be somewhere between the largest and smallest random sample is that if all five happen to be under the median, or all five happen to be above the median, that's just like flipping a coin five times in either getting all tails or all heads. Now that's possible, but there's only a 3.
25% chance of either of those occurring, Dr Genevieve Hayes: Out of all the techniques that you've mentioned, I've actually found that one to be the most useful in practice. What's the one technique that you find to be your go-to technique that's delivered you the greatest return on investment over the course of your career? Douglas Hubbard: Asking the question, has it been measured already? That's probably the easiest one.
What's interesting is even in data science, people tend not to look outside of their company. They. Tend to treat every measurement problem as if they're the first people to ever measure that thing. So they want to figure out how much time do we spend in X And they do this by doing, machine learning analysis of a bunch of logs of logging into systems and stuff like this.
And they try to figure out how do I make these inferences from my database? Maybe somebody wrote a whole dissertation on exactly that problem and they surveyed 200 companies to do it often there is a study like that and I'll point it out to a company that is measuring this really consequential measurement and I'm the first person to bring up this study that's been around for years. So no one there even thought to look for it. I don't have any special academic access.
I'm just using Google Scholar and, and ResearchGate and that's it. And I just download PDFs or I call up the author and say, Hey, can you send me a copy of this? I can see the abstract, but I don't have access to it behind the paywall. Or sometimes I pay 35 bucks for the article.
Yeah, they already measured it. So , it's just odd that it doesn't happen more often. Dr Genevieve Hayes: And with ai, you can get around that because I've had situations where I'm writing something and I want a citation. For it, so I'll just put into um, ai I need a citation for this.
Can you please help me find a reference for it, and it'll tell you where the reference is. So you could use that for finding someone who's done similar research. Douglas Hubbard: Yes, you're exactly right, even if the study doesn't measure exactly what you're talking about, the methods can be revealing. Oh yeah, they measured it that way.
Maybe I can implement that, maybe I can try that method myself. Dr Genevieve Hayes: So that's it for today's conversation with Doug. If you haven't already listened to our previous episode where Doug and I discussed how data scientists can create business value with limited data by reducing decision making uncertainty, if you'll find it at value driven data science.com or on your favorite podcast platform.
Thanks for joining me again, Doug. Douglas Hubbard: You bet. Thanks a lot for having me. Dr Genevieve Hayes: And for those in the audience, thanks for listening.
I'm Dr. Genevieve Hayes, and this has been Value-Driven Data Science.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
Episode 112: [Value Boost] Lies, Damned Lies and Stakeholders61 / 100Episode 111: Building Your Defences Against AI Misinformation83 / 100Episode 109: How to Measure Anything and Make Better Decisions98 / 100Episode 108: [Value Boost] How to Use AI Without Losing Your Edge75 / 100Episode 107: Building a Virtual Empire of AI Specialists78 / 100
