#296: Avoiding Major Oopsies: Twyman's Law, Intuition, and Valuing Accuracy Over Precision

Narrator: Foreign. Analytics topics covered conversationally and sometimes with explicit language.

Tim Wilson: Hi everyone. Welcome to the Analytics Power hour. This is episode number 290. Oh, wait, no, wait, let me check that. Uh, this is episode number 296. That was a close one. I mean, how embarrassing would that have been? We're all set to have a discussion about data accuracy and getting business partners to trust the data. And I almost whiffed on something as simple as the episode number already. Though perhaps I've undermined your trust in me. Tim Wilson sitting in the host chair that is usually occupied by Michael Helbling. Perhaps I have. Let's ask my co hosts, Mo Kiss from Canva. Welcome to the show. What do you think? Did my little gaffe destroy my credibility?

Moe Kiss: Yeah, trust is completely lost. No, I'm joking. You nailed it.

Tim Wilson: It's been a loss. It was lost years ago with you. So, uh, and Julie Hoyer from Further. Have I already destroyed our audience's confidence in me?

Julie Hoyer: What's a mistake among friends, right? I'm just your pre read.

Tim Wilson: Okay. I like it. Well, that's the topic for this episode. I mean, sort of all data can be digitized and an overwhelming amount of it is. And digitized data breaks down into like discrete little ones and zeros somewhere down the chain. It should be cold and objective and to use it effectively, we need to get it to be as accurate and precise as possible, right? I mean, well, maybe so. Joining us for a discussion on just that topic is Eric Friedman. Eric started his career as a software engineer, then took a turn and did a, uh, PhD in computer science, focused on, on privacy preserving data mining, which is a tongue twister. Then went on to work a few years as a program manager for Microsoft R and D. Then he popped back over to research for a while and now he's been at atlassian for over 10 years where he is currently a senior principal data scientist focused on product analytics. And today he is our guest. So welcome to the show, Eric.

Eric Friedman: Thank you very much. Long time listener, happy to be here. And I assure you all of these career moves made a lot of sense. It made a lot of sense at the time.

Tim Wilson: It was a logical progression at the time.

Eric Friedman: Absolutely.

Tim Wilson: I went architecture, technical writing, Marcom analytics. So, um, I sympathize and actually I don't know that any of those made any sense, but, uh, that'll be something for me to ponder on my deathbed. But. So Eric, you presented at the last Measure Camp in Sydney and uh, in that presentation, I think you had a fun. Or maybe it's relatable at least. Or maybe it's a tragic story about, uh, an active users metric that looked almost too good. The results were brilliant. Everyone was thrilled, but you had this nagging feeling that something was off and you couldn't really nail down the specific problem. So maybe that's a good way to sort of start the show because I think it gets to something really human about our relationship with data. So maybe we'll kick things off by having you walk us through kind of what happened there.

Eric Friedman: Sure. So, like, this is a story from a while back, but, you know, at the time the product team was working on a product feature, a big change in the user interface, and a lot of investment went into that. So, so we put a lot of work on actually testing things, you know, a B testing. I worked with another data scientist to make sure that we got things right. And I remember back then, you know, us sitting in the boardroom with all the PMs and it looked like really a big change. Like it moved the needle quite significantly. So that was a big deal. Everybody were happy. And then as we went on, um, and rolled out this feature, we saw what we expected to see. Active users went up. Uh, but then, you know, over time, I'm starting to develop this odd feeling. You know, you look at data over time, you get a sense of how it looks, how growth looks like. And I was looking at the curves and they looked a bit suspicious. Right. Like it's not supposed to go, like. And I started looking into that. Okay. Because it felt a bit odd. And I remember trying to go down to specific trace logs, trying to find what went wrong. I found like, everything checked. So after putting some time into that, I said, everything looks good. We actually have a reason to believe that active users goes up. All is fine. Right. I cannot say anything about this. I don't have a smoking gun. Fast forward. Um, sometime later, another data scientist and a software engineer, of course, we keep looking into this and they actually found a bug. It turned out that because of a bug, it caused inflation of active usage monitoring. And that was an unpleasant surprise for the product team. So for me, uh, it definitely caused a lot of thinking. How come I didn't find that? What could I have done, done different? And that, that, you know, that's. That causes a lot of reflection.

Tim Wilson: Well, and did you, did you go tell them? Um, like immediately, abruptly, did you ease it out? Like, how did you actually communicate? Like, what, what's the rest of the story on that?

Eric Friedman: Yeah, so I, I think that's probably the biggest mistake I made at the time because, you know, I didn't find an issue. And, you know, I. At least back then I thought, you know, we are the data people, we are the evidence people, and if we don't have the data to back up what we say, we should shut up. And, uh, what I learned from that is actually, you know, our intuition, our, uh, you know, that's part of our expert opinion, and we should sometimes do just go with it. And I think there are a lot of things we can do ahead of time to prevent mistakes or to check things. But at least for me, for this story, when I look, what could I have done different? Actually, I knew to do all the checks. And eventually, when your intuition is your last line of defense, and sometimes you just have to go with that.

Moe Kiss: So sometimes I don't know. Firstly, I just want to say thank you. I'm really grateful that you're sharing, you know, straight off the bat, an experience of where you made a mistake that's incredibly humble and wonderful for folks to be able to learn from. So a very big thank you. Um, I suppose I just. I'm curious to understand this intuition piece. Right. Like, I feel we all have it. And I know you and I have had conversations previously about, like, when we're making decisions, you know, we have data, we have intuition, we have, you know, maybe previous experience. We have all these different factors, and part of our role is helping pull those things together. But I'm curious to understand, so how has this changed, uh, how you would show up? Like, let's say this happens next time. You can't find a smoking gun, you can't find a data point, but you had your, you know, your intuition in your gut. That's like, something's not right here. Like, do you think this time you would say something and how would you frame it?

Eric Friedman: Yeah. So I think, first of all, it's about being more opinionated. And in. In the, like, over time. Ah, that's part of my growth journey. I think in the past I would. I. I tended to be a bit more, you know, impartial. Right. We let the data speak. Like, we're just there to give the data its voice. And I think over time I feel a bit more confident, you know, to be opinionated. Like, I have my own opinion of things. And I think we're actually expected to be opinionated. So I think that's. That's one thing. Just like, be more confident in our own expertise. And then I think it's probably, I mean, the Opinion on its own is not enough, Right? Uh, probably even if we have just this intuition, we will still be expected to, okay, dig into that. Can you actually find the evidence? But I think it's the start of a conversation, right? This is what I think. Maybe the data looks a bit odd and we maybe want to dig a bit more into that and then the business can decide, you know what? No, we did all our checks and balances. It's good. We did our due diligence. Let's go on. Or they could say, let's take more time and actually give it more space to dig into that. And at the time, I basically made my own decision and said, I looked enough into that and I kept it with myself. And this is something where we need to be more open and just share what we think with our business partners about that.

Tim Wilson: So I will sometimes find myself not doing this necessarily with a ton of structure and rigor, but trying to think what I expect to see before I actually look at the data. Or sometimes asking if I really don't think I have context. And my business partner says, well, I think it'll go up. I'm like, what does up mean for this metric? Or what is. What would that mean? Like, I don't. Give me. Use it as an opportunity to try to get a little more context about their intuition about their business. Like there's. There's some psychological trick or something where you. You sort of force yourself to set what you think you're going to see as opposed to waiting until you see it and then you immediately rationalize why it makes sense. Like, I had an example from years ago where a, uh, product marketing manager came racing in because he had already told the whole company that this tiny little change he'd made to a webpage had, like, drastically driven the track traffic up, which made no sense. And that was one where I dug in and found that it was, it was a bot. There was a company that was selling us software, and they pointed literally at his page to gather some data for the pitch in a few days. Um, but it was one where it was like he saw it, he was good news, kind of like the active users. But if I'd gone back, if I known he was making that change, I would say, what do you realistically think this is going to do? Because if it goes way past that, maybe we want to apply a little bit of Twyman's Law or something to it. But I think there's. I don't know, I feel like that's the whole It's a challenge, especially for people just entering a field or the space, to have that intuition and to have the confidence on top of it. Like you said that a few times, that it's like as you get more experience, the more faith you have in your intuition and your conviction. And that's like, how do you, how do you develop that other than letting time pass?

Eric Friedman: I think a lot of this is getting to live a bit in your business partner's world, like getting to speak the language and getting to know what they care about, what they're after. I think that adds a lot of context. And for example, I remember at some point I started hearing PMs talk all the time about JTBD, JTBD this, JTBD that, Jobs to be done. I had no idea what it was all about. And I started digging into that, reading a bit about that and what I found. I didn't really get what they were talking about. So getting into the world, learning about that, I think helps create this common language and thinking about it from their perspective. So I think that definitely helps us, uh, know, to get, so to speak, like in their head, when that was

Tim Wilson: one where you actually, if I. I had this happen with okrs once where I was working with someone who was. And I knew what OKRs were. I'd lived in kind of an OKRs training, but it was like, oh, this client is really into it. I'm going to go get a book. And didn't you do that? Didn't you wind up going and like read like whatever the jt JTBD Bible is like, okay, if we're going to use this, I want to understand it. I mean, you probably found out that they actually weren't using it correctly. It's like agile or any other number of things that use the acronym but maybe aren't applying the process.

Eric Friedman: Yeah. So I went on and I actually read about Jobs to be done. And it was actually weird because, like, what I was reading and what I was hearing from the PMs were not exactly the same things. And at the time, like in our confluence, like internal wiki, I wrote a blog post, hey, like, are we doing JTBD wrong? And it was a bit of a clickbait. But, uh, it started a conversation because, I mean, then I started, like, through the conversation with them, I started to find out, oh, actually there are different kinds of definitions of JTBD out there. And having this conversation and trying to understand, you know, what they really mean when they talk about that that really helped. And I did it from the perspective of, you know, how I can be more useful as a data scientist and what is it that they actually after when they talk about understanding the jtbd. So, uh, I think it was first, you know, just to catch up with them, but also to see, you know, how can I answer their question, their questions, and how can I better understand the question and, you know, improve myself as a data scientist? That helps them.

Tim Wilson: Michael, I have news. The AI analyst is emerging.

Speaker F: Uh oh, that's big coming from the quintessential analyst. What do you mean like a cryptid?

Tim Wilson: Well, I mean more like a. More like a job promotion, but you know, with more existential dread. You know how foundation models created the AI engineer role?

Speaker F: Yeah. Developers got all these cool titles and analyst. Scott, can you pull this by end of day?

Tim Wilson: Exactly. But now we're watching the birth of the AI analyst. Someone who uses LLMs to multiply their capabilities without, you know, multiplying their stress rash.

Speaker F: Nice. So an analyst but with superpowers and fewer open tabs.

Tim Wilson: Exactly. And the tool for that is Prism by Ask y AI.

Speaker F: Yeah, Prism is basically the interface between what I mean and the 900 steps I don't want to do. You ask in plain English. And it helps you get from question to analysis really fast.

Tim Wilson: And it doesn't forget your world. Prism's jam memory, that's J A M. Their jam memory remembers your definition like what year org means by conversion, which table is the source of truth and that July data. Don't forget it's, you know, cursed.

Speaker F: Yes, thank you. I am tired of re explaining our metrics like it's the extended MCU universe.

Tim Wilson: Plus you can capture like repeatable workflows as skills portable expertise like, you know, clean utms or fix GA4 channel grouping or standardized campaign naming and reuse those across different data centers.

Speaker F: I like that because I do a lot of copy paste. This is can't continue type of feeling. And I feel like it would be nice to be like run skill. Look smart, drink water.

Tim Wilson: Nice. Want to become the AI analyst before your co worker does go to ask why AI and join the wait list.

Speaker F: Yeah, and use code APH and ask why pushes you to the top of the wait list. That's ask dash y the letter Y AI and use code aph. Yeah, the AI ah, analyst is here. This product is in beta, but you can get in on the ground floor.

Tim Wilson: And it's coming for your busy work, not your job. So you know, relax, chill out.

Moe Kiss: So Eric, you and I have spent a lot of time talking about this whole, like, accuracy versus precision playoff. And it's something that has just really resonated with me because I would say I always lean to kind of, like the best possible answer with the time that we have in the business to help make a decision. I suppose I would say I'm, like, quite pragmatic. But a lot of what I hear coming from data scientists is, um, this number is wrong. This one's right. Um, we can't do it this way. We have to do it this way because this is the right way. And I guess I just wanted to hear your framing about this playoff between accuracy and precision. And I don't know, you have such an elevated way of thinking about it.

Eric Friedman: Yeah. So, uh, I think straight from primary school, the way we're taught things is that accurate and precision are like, being accurate and precise are the same things, right? So at school you get this big equation. And in the data science world, you can think about a business question, like, what kind of features correlate with user satisfaction? Or something like that, or how can we predict those kind of things in parallel? At school, you might get a question like, how much is 2124 times 3926? You get this big equation, and what you're taught is that you need to go through the methods. You have to multiply this digit by that digit, carry over. If you get anything off, any single digit is off, you lose all your marks. So if your answer is not precise, it's not accurate, you lose all your points. And I think we kind of carry on this mindset with us into our jobs. And in the business domain, actually, precision and accuracy are not the same things, because these big numbers, it's about 2k times 4k, it's about 8 million. And, uh, about 8 million is accurate. It's not precise, but it's accurate. And I think that also, like, when we get business questions, there are many ways we can go and approach and solve them. We can throw them, I don't know, like hidden Markov models or clustering algorithms, all this arsenal of all these methodologies that we learned. And that's kind of like even part of our pride in the craft, right? Like, we want to show that we know to do all those things, but sometimes you can get quite a good answer just by writing a very simple SQL query. And it's maybe not the best answer, but it's good enough and it's faster. And, uh, there is this quote from John Tookey from he had this paper about the future of data analysis, and this is from the 60s. And he says far better and approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise. And I think it really captures well that first of all it's better to answer the right questions. And I think that part of our job is to help get to those right questions. But also when we answer, it's not just about precision. It's, you know, the answer can be accurate without being precise. And I think that's a way for us to be more fast and effective and be focused on business impact.

Tim Wilson: But there's a challenge, right, that our business partners, accessing tools and data, um, they're working with spreadsheets, they're working with dashboards because they're in this digital world and processing and multiplication is, is, is cheap. Like they are conditioned to see things that are precise. I mean, I love the, like the, the accuracy versus precision, like uh, dart, uh, um, archery thing that sort of shows accurate. The grid, the 2 by 2 of accurate and precise. Accurate, not precise. Precise, not accurate, not accurate, not precise. And it shows like the spread around the target. Um, and I think that's good for us to be aware of. And part of it is, I mean, can that be handled a little bit with the language of saying try to get yourself out of the boat of providing precise answers. Like get comfortable with. Even if you have a precise answer, better to kind of back off the precision of it a little bit so that somebody isn't looking at, you're not arming them with ways to, to find another. It may be too reasonably accurate. But the precision means that um, the first three digits kind of differ. Like there's a challenge with us understanding that and our business partners thinking that way and the levers we can pull to try to help, um, them think in language of this is, this is accurate. You know, it's about, it's about 8 million, you know, or whatever the answer is. And they're probably not going to object to that. They're not going to come back and say, what do you mean about a million? I need that down to the, to the first digit. Right. I mean it, like it's a fine line to walk.

Eric Friedman: Yeah. And I think it really depends on the context and the question being asked because in some contexts, like if you deal with financial data, you definitely want to be precise. It's important. But most, uh, like a lot of the business questions are more direct, direct in nature. And when you deal with directional questions, it doesn't matter as much if you're precise. And actually precision can kind of draw the attention to the wrong thing because you don't care about the precision necessarily in these kind of questions. So this is about getting to the bottom of what are the decisions that they are about to take and what really matters for answering those decisions.

Moe Kiss: So one of the things I wanted to challenge and have a discussion on, so I'm trying to find a specific. And of course I can't, but there was some research that was done with pricing and Uber, um, and I often quote it when I'm counseling, um, people on buying engagement rings. Buying a 2 karat diamond ring is more expensive than a 1.999 carat ring. Right? Like we have this bias and heuristic to think that the two carat, the rounded number is better.

Tim Wilson: So hold on, just wait, wait a minute. How many people are you counseling on buying? Like what is your world?

Moe Kiss: I. My team are quite young and so a lot of them are going through that life stage.

Tim Wilson: Um, those are just one on ones. They're like, hey, what's going on? I could really use some help anyway. Okay, um, carry on.

Moe Kiss: So also, I don't know who's buying a two carat diamond ring. It's a, it's a. Uh, anyway, is that.

Speaker F: That's a.

Tim Wilson: That's a big one.

Moe Kiss: Two carat is quite big.

Julie Hoyer: Yes.

Moe Kiss: That, that's okay. Not for America, but for Australia. Americans tend to buy much bigger rings. Anyway, back to the point of the story.

Julie Hoyer: Well, and now the lab grow. You should see these rings people are throwing around.

Tim Wilson: Sorry M. You guys are living in a different world.

Julie Hoyer: Um, but.

Moe Kiss: All right, what I was going to.

Tim Wilson: Okay, back to your point.

Moe Kiss: Back to my point. One of my concerns is like, I hear you and I agree with everything that you're saying. Right. But especially, okay, the same thing happens when you're trying to counsel people on like a day rate. Sometimes like people will intentionally like make a number not around number because people are more inclined to believe it that you, you've put more work into it. Because we have this bias that say if you say sixteen hundred dollars a day that like it's not, it's you, you've just picked a number from the sky versus having put thought into it. So like I'm just trying to think about how that plays off with our discussion here about not necessarily always wanting that precision.

Julie Hoyer: So you're saying if we don't go precise enough, they won't Trust the number?

Moe Kiss: Yes. So if we say, oh, the number is 8 million versus 8 million 2097, will people just be like, is, is that bias going to come to life where they're like, oh, maybe that number is not right and like, I can kind of disregard it because it's come from nowhere or like, do you see what I'm saying about like, does the precision give any extra credibility or build more trust?

Eric Friedman: Right. So in those cases, like precision sometimes is kind of like the signal that you did the work. Right. And again, I think these kind of things are context dependent because ideally if you have established a good trust relationship with your business partner, then they trust your judgment and said you made the call, the right call about what's a good methodology to approach this. And ah, in some cases maybe precision does matter. Maybe it's a signal that's important to them. And in that case, yeah, maybe you do need to put a bit more work and give that kind of signal. So that definitely is context dependent and

Julie Hoyer: part of the context. I'm just thinking like, um, Tim, like back to some of our discussions we've had with clients. When they have really low volume, right? The difference of one is greater than if you have a large volume. The difference of 10 is like minuscule or you know, like you can kind of ignore it. So I do feel like too, it's like, are you speaking of a really small N? Because then you don't want to round, um, compared to when you're speaking larger volumes, you can round to 8 million and it's probably okay. The other thing with precision that I think is interesting is we get sucked into a lot is when um, the stakeholders want to compare systems, numbers from systems which we know we're not going to give you the exact same one. Like directionally they should be close enough. But I do feel like that is a scenario where they really get stuck on like why is this 1% different? And it's like, well, it's okay, it's like 8 million, it's all right. And we're not talking dollars and you know, which one's your source of truth. So we should be okay to continue to use, you know, the other um, number in our decision making. But that's really hard, I think for stakeholders and even for analysts like navigate.

Tim Wilson: But I think we tend to make, I mean if it's like you're going to err on one side or the other, I mean I look at, somebody makes a uh, line chart to show the results and they label every single data Point down to the first value. And that's not so like there's there, there are choices made in the presentation of the information where I think you can say there's precision there. I'm showing a dashboard, but this is rounded to the nearest million or the nearest thousand. I'm not giving you 8000000. I'm saying 8 million or 8.1 million. Give them a decimal place. So I think there are choices. I mean certainly arc to your point that the context. And actually Julia, your point as well, the context does matter. When does it matter? But I, I feel like there's like the, the trap we can fall into on the analytics side is we have the position precision. Might as well show it. You know, our P value is not only is it less than 0.05, it's every model spits out. It's the p value is 0.00013472. Like please don't put that. You know, just copy paste. Yeah, P value is less than 0.05.

Eric Friedman: And uh, Julie, I think by the way, it's a great point because it matters a lot like why the numbers are different and scenario one, the numbers are different. We have no idea why. Okay. That's not a good place to be. And that's a place where you probably do want to ask questions. Why do these things don't match versus a different scenario where we actually expect those numbers not to be perfectly the same because maybe we measure slightly different things or there are known reasons why this should mismatch. So I think usually it's more about the confidence. We know what's going on here. And Tim, to your point, maybe if we think that this will just distract and raise questions that are not relevant, then maybe it's better off to reduce the precision to just avoid that altogether.

Tim Wilson: So I think the other, I think business, our business partners and then data teams get sucked into it when it's the different systems aren't. Aren't going to reconcile and ar. I think to your point, like you should understand not down to the perfect. We can net out everything. But say these are the, the big movers of the differences. A lot of times there's the ability to say over time, look, these move in the same direction. I watch companies say, well we just need to pick the system of record for that metric. Which feels like the hairs on the back of my neck go up. Like that's a cool idea that somebody in the C suite or one level down said we have the solution. We Just pick our system of record. But the reality is all those systems exist for different reasons and it gets back to context. So I think there's a part that says we need to put this to rest at some point by doing a little bit of an analysis of saying these differ. These are the main reasons they differ. Let us show you that they generally move in the same direction over time. And now we're just going to have our standard little footnote that these move differently. But I think also we're going to go look at the system that gives us the most what we need, because it's often kind of like upstream system in some process has this data, has this data hands off to another system which goes downstream from that. So you have to pick the system where you can get the slice. I mean, I'm thinking in a CRM digital analytics, CRM sales world, you can't necessarily track the marketing channel all the way through to a sale. And in the middle you've got something like a lead that the downstream ability to follow it comes out of one system. The upstream to follow it comes from another system and heading off and understanding. And maybe this gets back. Arik, to your point earlier, you're like, you really need to be figuring out, is the question being asked, very clear and precise and let that drive everything as to where you look, what level of precision, how you pull it, what the intuition is. And now it just went on a bit of a tirade.

Julie Hoyer: Well, no, but you bring up a point that, you know, maybe this is me still like wrapping my head totally around the, uh, accuracy versus precision. But I do keep thinking of what you were calling out, like the, um, target. So we were talking about people being most comfortable with precision of systems matching, but we just discussed, you know, thoroughly, like, why that's. We know it doesn't happen. But to your point, Tim, if they are, if we can explain in at least broad strokes, Eric, as you said, like, why these things are differing and if we then can say, well, as long as we're pointing this at the right, like, target, the right question, the right problem to be focused on, then can we be comfortable with accuracy without precision between the tools? Like, am I taking us way off? But you're saying, you know, like, if they're always about 1% off, they're all hitting the right target, maybe not closely bunched together, that would make people happy with accuracy and precision.

Eric Friedman: Yeah. So I think the data science teams usually have a good opportunity to collaborate and aim to standardize measurements. And one example I've seen in the past where growth teams and product teams had different measurements of impact. And that can be very confusing because when one team says, oh, we had that impact and then the other team kind of interprets it in their own language. So, uh, I think definitely for data scientists, we can be in a role where we coordinate between our teams to see can we actually standardize and measure things the same way and ensure that we actually get the same numbers all over the place. But as you said, sometimes it's not possible, right? Sometimes there are actually very good reasons why things should be different. And maybe then we should just use different names for them. So it's very clear that when this team talks about something, it's not the same thing that the other Tim talks about. So I think we definitely have a role to play with this.

Tim Wilson: So can we shift a little bit? Because some of this talking about kind of getting trust in the data and it feels like there is a, um, there's a big one we covered which is like we'll get people to trust in the accuracy we've covered two things. If I'm going to mid show recap, which we don't really do recaps, and I don't know what the hell I'm doing. But we've got kind of the develop and trust your intuition. We've got the think about accuracy and precision differently. We haven't actually covered like, how do you not push bad info out? And the intuition is kind of like one hook into that. But, uh, the frenetic pace everybody in modern business is working in. There is a drive to get the task completed off my desk, over the wall and into the hands, which just like begs for mistakes to be made. And the cost of making a mistake with the query with the report is of damaging like trust. So like how. What other like ways do we have? We briefly referenced Twyman's law, which really goes to the intuition and doing checks. But what are other ways to maintain trust in the data from our business partners? How do we prevent ourselves from undermining trust? It's like a long ramble with a broad question.

Eric Friedman: Yeah, I uh, think first thing and something that I definitely would like to see people do more is just stop and ask, does this make sense? And I don't think people do that enough. And at every stage, right, like, um, you look at the raw data, does it make sense? Is there anything off there? Ah, you apply methodology. Does it make sense? In this business context, you get a certain outcome. Does this outcome make sense? So I Think this is kind of like a first line of defense. Does it make sense?

Tim Wilson: But aren't there, There are times, there are times where it makes sense, but it's actually wrong if you're like, oh, it made sense, therefore I'm going to go full steam ahead. And then you find out later that oops, uh, just because it passed that hurdle, it was plausible. But,

Moe Kiss: but you're making the best decision you have with the information available at the time. I feel like the dust doesn't make sense actually is just a, like it's a spot check. Right. I, I actually think the thing that we need to do better at uh, is does it make sense with our stakeholders, like actually bringing them into some of those checkpoints versus kind of like, does it make sense to me individually?

Eric Friedman: Absolutely. I mean I, I guess like this is the first check that you need to do with yourself before you engage with others. And I think that's already a checkpoint that I think can probably spot some of the issues, ah, behind that. Yeah, I mean involving other people in this thought process and like, in general you want to do peer review process for everything. And by the way, I'm um, coming from a computer science background, my brain is wired as a software engineer. So basically anytime that I do anything that involves statistical modeling or math, I will always put someone else that has uh, strong math jobs and statistical modeling. Hey, check my work. Did I actually apply the methodology correctly? Because I know that they think about this in a different way than I do and they can spot things that I want.

Tim Wilson: Will the question come up in that context? Because you said like, does the raw data make sense? And from um, like a, the EDA of saying, I've made my initial query now I've got a table of data that has say 25 columns and 150,000 rows. Do. Have I gone through and checked that one? Is 150,000 rows seem like the right number of rows. Like that's like a sand. Like is the, does the size of the data make sense? But going kind of checking for gaps, distributions, the values, like checking the columns, doing um, that step of saying this should be the right data. But have I actually done, have I checked all the values, all the variables to see if the distributions and values are reasonable? Like not just trust the query because that adds time and it's probably a judgment. It depends on how solid and simple the SQL is. Does that make sense? Uh, and would somebody, if you're having, if you're running it by, is this the right model? How likely are they to say, did you double check that the data that you're feeding into this model is the data that you think it is?

Eric Friedman: Yeah, and I think it's probably both those things. And it depends also if it's like is this the first time I'm answering this kind of question or not? And I'll give us an example. Like there were cases at some point we worked on say internal developer effectiveness metrics and we tried to understand flow of work and things like that. And this was the first time we went to calculate these kind of things. So I worked on this. There was another data scientist calculating the same metric in parallel and we got completely different results. And the thing is that none of us was wrong. I mean technically we didn't make any mistakes or technical mistakes in the process. But we started walking through the calculation step by step and we realized said, oh, we actually made different assumptions about what the data meant. We made different assumptions about what we wanted to measure. So just by working through this process until the numbers matched, it allowed us to align on the assumption and it gave us the confidence that we were actually measuring the right thing. And so, uh, it's definitely not something you will do with any metric or any calculation that, that you do, but definitely the first time that you do something, it's good to have this cross check. And once you got this right, once, okay, the next time you already know what you're doing and you don't need the same level of due diligence.

Tim Wilson: So Julie's lighting up on the like, oh yeah, yeah, assumptions. That's another thing. We haven't, we've talked about it in the past, but the discipline of documenting the assumptions that you made, and it may be like the list of assumptions I made were these 20. If I'm having another analyst review it, I'm going to show them all 20. When I present this to my business partner, maybe I even pre read it. I'm like, these are the three or four that I want to make sure they understood that I made these assumptions. We've talked about that as a practice of analytics of you are making these decisions and just making an assumption assumption and moving ahead is like pretty dangerous actually having in the process a place to write down those assumptions. One from a repeatability, somebody checking your work, but also to actually go back to the business and actually include that in the results. Not the exhaustive here's how we made the sausage. But hey, I just want you to know going into this and if that's your opening slide and they Say, well that's a bad assumption. Then you actually screwed up because you should have done a pre read with hey, before we present this, these are like the assumptions we made and here's why we made them.

Moe Kiss: But how much, how much do you think the business do that, that they actually like, I feel like assumptions are included. They're often like at the bottom of the slide in the, in the text somewhere and like the business just glosses over them. Like I would love to a business stakeholder to be like. And I've had that at previous companies where like I've had very senior folks like the founders or CEO be like, I don't agree with this assumption. Like, let's, let's debate this, let's talk it out. But I don't see. And I would love to see more of that.

Tim Wilson: I mean I, I find it as a, as a way to, on the building trust to not have that in the delivery. But it's like, I mean say there's an executive you're presenting to, but there's this business partner that you're actually collaborating with. It's a great way to throw in slack. Say, hey, is it safe to assume this? And have them say yes, is it safe to assume that? And then I, I sometimes will put those assumptions like front and center. Like I want to be confident that and, but, but I probably wouldn't say I made the assumption. I would say we made the assumptions and hey, just so you know, we assumed, um, you know, that that holiday has no impact on our bottom line. Yeah, no, that's, that would be a bad assumption to make. Um, but I mean it's not just like a CYA of like throw this in the, in the footnotes. It's. I think it's as much of a show that you're trying to understand the context and the operating environment, which is building trust upstream, which also means that there will be more trust in the output.

Julie Hoyer: No, honestly though, I feel like sometimes that type of documentation and assumptions are like definition stuff too. I'm with you Mo. Like it's refreshing and encouraging when your stakeholders key in on it and understand the value of it or want to debate it and can call it out if it's wrong. But like, honestly I think it, a lot of times it is more useful for yourself or others to replicate the work later on or if you have to build on the current work. Like I think sometimes if you're not kind of obsessed with like documenting all of those things. I mean I've been the analyst that's been given a spreadsheet with random numbers supposedly where they pulled it. And I have no other definitions or assumptions. And they're like, oh, recreate this and do it again for the new time period. And you're like, f my life, I'm never gonna get this. And the person that made it originally is gone. Um, so all that to say, like sometimes if it's not building trust directly with stakeholders, I do think it helps maybe the analytics team broadly or the data scientist team broadly. Um, more for the replicability is how I see it.

Eric Friedman: Yeah, I think definitely documenting the analysis, all the assumptions, that's probably not the document that you're going to eventually show as your final result. But it's probably good to always have here is the technical. A document with all the details, all the assumptions. So it's available there and so you can reproduce this. And yeah, but referring to most points, I think this conversation with your business partner is part of the process. Right. Like you get your question, like here is how I understand it, this is my interpretation, this is how it translates to the methodology. So you definitely want to be sure that you know you're on the same page and you're actually answering the same question.

Julie Hoyer: The other thing I think of too, like if it does get put in the footnotes of the document or this um, the presentation like sent to your business stakeholders. Mo like I have spent so much time talking to people and on my team or being so worried about myself, what happens when I give this deck to my stakeholders and then they pass it to their stakeholders and they pass it to their stakeholders and then they start asking follow up questions and nobody knows to ask me for the clarification. So like, honestly I think my um, safeguard and anxiety is what makes me put a lot of that in the final documentation in the appendix or in the footnotes because I'm like, hey, if this gets past like three degrees away from me, which again that's a win if your work gets passed that far along.

Moe Kiss: I see this happen all the time with experiment results where like a data scientist has pulled something together and then someone like summarizes it and someone might summarizes it for a slack message and like these steps further and further away, which look, do I want stakeholders to be able to interpret experiments and communicate it? Absolutely lutely. But I think sometimes that like just the, the, the understanding of like the core trade offs kind of gets watered down or there's like different incentives and it just gets really tricky. Uh, Eric, I know like you've done a lot of thinking about this about like once the analysis is out, the experiment result is out and you kind of lose control of the narrative, so to speak. Like how do you approach that?

Eric Friedman: Yeah, so it definitely happens like sometimes you, you get a uh, chart or something that gets copy pasted and then someone puts it on a slide deck and it's completely out of context and you lost that. And one thing that I try is again if you have a documentation of your analysis, you should definitely have a link to that. So if it's copy pasted at least the link to that document is still there and there's the footnote in the chart where you can highlight the main details. So it's definitely a question of a balance. You don't want to overload your visuals with all the assumptions and all the details. It's distracting. But at least have this kind of selective context or pointer to where the information is. So it kind of travels together with the visuals and sometimes you just lose control and you have nothing to do about that. But at least you can take some mitigating steps so the evidence is there. If anyone really will want to find this information, there is a way for them to get there. Ah, so that's the least we can do to at least control that.

Tim Wilson: Yeah, I mean I think like pointing to like having the footnote of like what was the, what was the interlang with the source. I mean you're well stated that it's like you don't want to overload it with all the assumptions but you want to provide the breadcrumb to say and the more you can put that proximate to the chart. So they're not likely. I mean that's pretty gross misbehavior. If somebody says oh, there are footnotes on this slide but I'm going to pull just the chart and drop it in an email. There's a little bit of that's on them. Um, if it gets its own legs, if you're giving it like hey, this is probably important reference information that should go along with it. Um, I think that's good. But this I feel like we could go on for uh, multiple hours but unfortunately we need to head to wrap or we will lose trust with our audience by having a two hour episode which um, when you're called the analytics power hour, that would be a disconnect between data sources. No, that's a stretch. But before we leave, the last thing we like to do on the show always is go around the horn and share a last Call something that might be of interest to our users. And Arc, you're our first guest. Do you have a last call or maybe a couple last calls you'd like to share?

Eric Friedman: The first one is I guess the classic the islr, like the Introduction to Statistical Learning book, uh, which is actually a free resource. And today there's also a version for Python, Introduction to Statistical Learning in Python. And at times when AI sucks, all that tension, I think that actually going back to the basics, to the foundations is just as important as ever, if not more so. And I know that at least from my experience, even just going over the first chapters of the books, Linear Regression, it's like a very practical oriented book. So yeah, that's a recommendation that I usually provide. Like, it landed for me, they have

Tim Wilson: an R version and a Python version with like examples in it. Is that right?

Eric Friedman: Uh, yeah. So there are two versions of the book. The original one was with R, ISO,

Tim Wilson: R and ISL introduction. Okay, gotcha.

Eric Friedman: And they're both available at www.statlearning.com and I know that at least for me it landed a lot of concepts that I didn't really get before. So I really recommend that. And if I'm allowed a second, um, call, which is maybe more AI related. So I saw recently an article from Hamil Hussain called Revenge of the Data Scientist and it's actually both available as a YouTube talk. It's a PI AI talk from, uh, March, and he also posted it as a Twitter article and he actually talks about. I think it's specifically about mindsets. We have the mindset that you go with in this Agent first world. And I think that actually his point is that the data scientist approach, their mindset, their core skills like exploratory data analysis, metric design, model evaluation, all these things are as critical as ever and translate really well to this world. So I definitely recommend giving this a, uh, read. And also Hamil Hussain and Sureshankar had a terrific episode about AI evils in Lenny's podcast. So that's a great resource as well.

Tim Wilson: Awesome. So that was three really sorta. But those all sound amazing. I feel like I've read the first three chapters of like. More. Those are the books I'm most likely to abandon, but still get a lot of value because it's like chapter four where I start to. I'm like, what? We're at the area under the curve. I'm like, oh, oh boy, oh boy. The equations are getting pretty in depth here. Uh, so I, I kinda want to check those out. Julie, what's uh, your last call?

Julie Hoyer: Well, my last call is accurate and precise. Um, and I'm feeling like maybe I have called this one out before. But you know what, it's such a good one that I'm just gonna say it again. Um, it's called the Monarch app and I've actually been using it now for quite a while, um, for my own family budgeting, um, and financial like tool for the family spentures. Um, and it's really nice because you can connect everything directly into it. So on top of having all your different credit cards, bank accounts, um, you get really nice cash flow visuals. You can re categorize like any of your expenses that come through and it will notify you like, hey, this is, is a recurring one or hey, this is one that's not categorized. You want to come in and quickly do it. Um, but budgeting is not easy to do on your own. And I feel like this is the first app and different thing I've tried that I can actually keep up with it. I can quickly get to it. Um, you know, there's no delay. Like I spent. It shows there. Um, my paycheck hits. It shows there. So it's been awesome. You can also set a lot of your budgets and goals in the app as well. So like for all your different categories, you can customize categories or not. Um, and then you can say, you know, I'm looking to spend X amount each month in each category and it just, I feel like, makes it a lot easier to actually live out the financial plan and budget that we're going for. So check it out. If you're needing a tool, have you

Tim Wilson: figured out a way to turn off the, Turn off the net worth words? Uh, because there are periods where I want to not look at that when the. Uh.

Julie Hoyer: Yeah, thank you.

Tim Wilson: I'm a Monarch user so I'm a fan.

Eric Friedman: Oh, you are?

Speaker F: Yeah.

Julie Hoyer: See you guys. It must be good if Tim's into it, if he approves of the visuals, the tools.

Eric Friedman: Yeah.

Tim Wilson: Judge, judge your trust in the source. Um, um, um, Mo, what's your last call?

Moe Kiss: Uh, uh, I know we talk about her a lot, but Cassie, Cassie Kazarkov has a really great new. I mean it's just her regular newsletter, but she's doing a couple of back to back newsletters on why vibe coding will bite you. And here is exactly where. And she's talked through a few like text scenarios of where things have gone really wrong like uh, prod systems being completely wiped, things like that. So definitely go have a Listen to that. But I think the thing, like, the real takeaway is that the stories are all, all about the same thing, which is just like misplaced trust and the speed at which it computes. And so it's not like the nobody got hacked AI didn't go rogue. It's just like people let their guard down. And I think the thing that's been on my mind the most, um, which she's just so damn articulate, but it's like, expertise won't save you. Guardrails might. And I think guardrails is the topic that just keeps looping on my brain at the moment. And. And so, yeah, I really enjoyed that newsletter. And she's got a couple more coming out on the same topic, so check it out.

Tim Wilson: I think that series motivated me to dive into some new Vibe coding project specifically. Um, so far I'm safe. I haven't crashed the podcast because that's where most of them happen. Starting to wonder if Riverside, our podcast recording app, might actually be leaning a little too much on Vibe coding as it's having, uh, some of the joys it's been bringing us of late. But so. So, uh, I'm going to pander to MO here a little bit, um, because, uh, with the new season of Choiceology that Katie Milkman came out with, um, one of the things she has is this checklist which is this mapping of. Now she said she had a couple of undergraduate students do it and that means it's like in a PDF for some insane reason. But she's basically gone through and looked at like, what are the different topics like attribution bias or Dunning Kruger or left digit bias, like the stuff that her episodes cover. This kind of reverses it. And it's a guide to. Broken down by. These are all the topics of kind of cognitive biases. Um, then which are the episodes you can actually listen to. So if you're not a, um, Katie Milkman choiceology completionist, like I am, but you like, I wonder if she's ever had an episode about, um, mean reversion neglect, then this little guide will pop you to is comically in a PDF, which I'm like, this is great. You guys really work to format this thing to one page. And she's about to start a new season, which means this, of all things that should be a, uh, Vibe coded website where it gets updated and maintained. This is it. But you know what? The undergrads, they'll learn that's how it was scoped. Academia, they're going to do Their little thing. So, um, with that, um, Eric, thanks so much for coming on. I feel like this is a case where we actually have, like, the show prep documents that have, like, even more gold in it that we were not able to get to. So we will have a lot of fun with that content our own, ourselves. We may figure out how to bring you back for more of that. So thanks so much for coming on.

Eric Friedman: Thank you very much.

Julie Hoyer: Awesome.

Tim Wilson: So if you, uh, listeners, um, we love to hear from you. So if you'd love to have you leave a review or rating on whatever platform you, um, listen to us on. If you'd like a free sticker of the podcast, you, um, can go to AnalyticsHour IO and request one. I will. If you've gotten this far in the episode and you think, wow, that was a smooth conversation and these guys are professional, I will just call out. Now that we have dealt with a tornado, uh, warning that led to a power outage and two young children and a dog sheltering in place with one co host. We've dealt with a busted Internet, um, that has been busted for the entire episode. But of course, the repair team showed up for that during the episode. Um, and we've dealt with various cases of people dropping off and returning and not even realizing that we were still recording the show. So I encourage you to stick around for the outtakes because there might be some real doozies in those. I, um, would like to. Tony, please leave this in. Thank you so much. Because if you pulled this thing together, you're, uh, good on your mate. Um, so it's been fun. It's been a fun discussion. Uh, uh, we've been at this for four hours to get this one hour pulled together. No, it hadn't been quite that bad. But we would love to hear from you. We'd love to hear if you thought that the edit was pretty smooth. If you've got your own thoughts on how to build, maintain, recover trust, uh, you can reach out to us on LinkedIn. You can reach out to us on the measure Slack. You can just send us an email at. Contact analyticshour IO. So, for Julie, for Mo, for all of the conspiring Mother Nature and construction projects that tried to not allow us to record this show about trust and accuracy and precision. Keep analyzing.

Narrator: Thanks for listening. Let's go. Keep the conversation going with your comments, suggestions and questions on Twitter at analyticshour, on the web at analyticshour IO, our LinkedIn group and the MeasuredChat Slack group. Music for the podcast by Josh Crowhurst.

Tim Wilson: Those smart guys wanted to fit in so they made up a term called analytics. Analytics don't work do.

Narrator: The analytics say go for it no matter who's going for it. So if you and I were on the field, the analytics say go for it. It's the stupidest, laziest, lamest thing I've ever heard. For reasoning in competition.

Moe Kiss: Fuck my life. So the work that's being done out the front of my house knocked the NBN cable out.

Tim Wilson: Okay, I'm wrapped now.

Moe Kiss: Oh shit. Were we still recording?

Tim Wilson: Oh my life. I had some but I was, I was trying to make so many notes that I uh, on other aspects that

Julie Hoyer: I, There was a lot going on today.

Moe Kiss: Oh my God.

Eric Friedman: Oh you came in.

Moe Kiss: So I just, there was no way.

Julie Hoyer: Nobody had the heart to tell you my life.

Tim Wilson: I'm like what?

Julie Hoyer: Like and then the second one one is like so can I rap?

Moe Kiss: Oh guys. Eric.

Eric Friedman: I definitely want to thank Tony in advance because like yeah like that's probably some work to do here.

Narrator: Yeah,

Julie Hoyer: my hot mic was the least of our concerns. He was would have thought um.

Moe Kiss: Oh Jesus.

Tim Wilson: Yeah but I mean I think it's, I think it's going to come together well and my mid show recap was like me organizing my thoughts cuz I was like we're actually hitting on some

Moe Kiss: I, I, I actually feel like Eric, we need like another two hours with you cuz there's so much stuff here. It's such gold.

Julie Hoyer: Seriously. Yeah. You had so much in the show prep doc that I was like oh I want to talk about that.

Moe Kiss: Oh this is why I was like I knew that the two of you would really like Eric cuz like you're the, the same with the very good at prep and organization and all those things.

Julie Hoyer: I literally bought a new microphone and

Moe Kiss: it's still, why have I still got this shitty one that doesn't even have a proper stand and it still works great and everyone keeps buying all these fancy ones that suck.

Julie Hoyer: I, I gotta return my hundred dollar one I guess and try a twenty dollar one. Maybe I need it to be less sensitive.

Tim Wilson: I think the show might need to buy you an audio interface. I think it's. I, well don't, don't, don't make any moves. Don't do anything drastic.

Moe Kiss: Yeah the audio interfaces and changing microphones.

Tim Wilson: It is because it's moving where the preamp is. It moves it into it separate it moves it from a little thing, crappy thing in the microphone. It then takes it out and puts it in a dedicated box.

Moe Kiss: Okay?

Tim Wilson: No.

Julie Hoyer: This is so above my head. I just want to buy a microphone that works.

Tim Wilson: M rock Flag. And accuracy versus precision.