Episode 270: How Experimentation Becomes Culture

Speaker A: Creating great products isn't just about features or roadmaps. It's about how organizations think, decide and operate around products. Product Thinking explores the systems, leadership and culture behind successful product organizations. We're bringing together insights from multiple product leaders pulled from past conversations to explore one shared topic offering different perspectives and lessons from real world experience. I'm Melissa Perry and you're listening to the Product Thinking podcast podcast by Product Institute. Today we're exploring what it actually takes for experimentation to stick inside a product organization. Not the theory of it, but the specific behaviors, portfolio decisions and mindsets that make the difference between experimentation that's real and experimentation that becomes theater. We'll start with David Bland, author of Testing Business Ideas. He shares why even the most well funded companies get experimentation wrong and what the leaders who get it right actually do differently. After that, we'll hear From Monica Lewis, VP of Product at LinkedIn, on the leadership habits that make her team feel safe enough to test and learn, and how she thinks about placing bets across a portfolio, from sure bets to venture bets, depending on how disruptive the moment is. And we'll close with Mario Rodriguez, Chief product officer at, uh, GitHub, who takes us inside how Copilot was built through failures, high iteration, velocity and a product outcome that surprised even him. Let's start with David. What do you find is the right setup and the right conditions for a company that really does want to learn experimentation, not just train a bunch of people in it, but like wants to be experimental. What do they need to do to make sure that that sticks?

Speaker B: It's a big challenge. You know, I think I won't name it, I, uh, won't call out companies, but the companies I've worked with that were very high profile billion dollar companies in the past couple years, uh, where it didn't stick was they took this check the box mentality to it. So I ran experiments, check. What else do I need to do to just launch the thing I already want to launch? And if you take that, if that's your mindset, then no amount of training is really going to help you. Um, you really need to talk about how are we de risking things, how do we changing our mindset on how we approach things? But if it's just another box they have to check to get to the thing they want to build, then it's not going to stick. It's going to be theater. And so I've been really trying to, uh, one, work on real ideas. So I don't talk about this publicly a lot, but all My work behind the scenes, if you don't hear me, you know, you don't hear from me for a while, it's because I'm behind the scenes working on real businesses and real ideas. And so what I've tried to do over the course of, I'd say the last five to seven years has been every time I do a workshop, it's on real opportunities, the team to try to figure out, figure out. And I think that, uh, has been. That's worked out really well for me because I can take something that they're kind of worried about that has high uncertainty, and I can take the people that know about that thing and are working on it, and I can say, look, here are some new tools for you to think about this a different way, and we're going to practice them on your real opportunity. So the way I've designed all my. All my stuff has been. It's really been a learning experience for me. But I'll introduce a concept, I'll introduce a fun case study that's really short. And then I'm like, okay, now we're using it on your real stuff. And I feel like dual track of, yeah, I'm learning a new tool, but I'm also learning it on something. It's a real opportunity has really helped that a bit as far as just making it stick. Now you still have to have leadership, uh, evangelize it and say that it matters and actually show action that it matters. And so that takes time. There are some really prominent companies I've worked with in Silicon Valley who were like the poster child of doing this way of working. And they stopped talking about it, and everybody in the company stopped doing it. And they were really surprised. The quote they said to me, which I'll never forget, was, it was in our bloodstream, but it wasn't in our DNA, which meant it was something that was happening. Right. But it wasn't really a part of how they worked. And as soon as the leadership stopped talking about it, people just stopped doing it and they went back to working the way they did before, and they had to restart that entire program again. So I feel like for those of you listening, you know, if you feel like you're being too repetitive or, oh, well, people got this, now I'm just going to move on. Don't, uh, keep repeating it. It's part of your job as leaders to keep repeating this and showing it and enabling it and creating a culture and an environment where people can work this way. It never stops. So if you stop the teams will stop working this way and they'll go back to what really comfortable to them. So that's something that I've learned over time as I watch this ebb and flow of companies over the last 10 years or so in my career, that when they stop talking about this, they just revert back to how they were doing it before.

Speaker C: Yeah.

Speaker A: Oh my God, I love that quote. It was in our bloodstream, but not our DNA. That's amazing. And it's true. It's like it's not embedded. I also tell leaders too, when it comes to strategy and stuff like that, I'm like, if you don't feel like you repeated it 40,000 times, it did not sink into your team.

Speaker D: Right.

Speaker A: Like, you can't just be like, oh, here it is, and then walk away. And I think you have to be so repetitive as a leader to make sure it's really sinking in and people are doing that. What David put his finger on is something a lot of organizations don't want to admit. If experimentation is just another step on the path to shipping the thing you already decided to build, it isn't experimentation, it's theater. And the bloodstream, but not DNA problem he described is one of the most common ways organizations quietly lose this capability. The moment leadership stops modeling it, teams stop doing it. Which raises the real question, what does a leader actually have to do to make it stick? Here's Monica. I think a lot of leaders struggle with creating the environment that you're describing right now. And where I see them get tripped up is like, we're locking people into super hard roadmaps or like lockdown roadmaps. We're like ask, uh, we're not giving people time for discovery. How did you cultivate this environment? What are the factors that you think lead to your team to say, hey, I'm comfortable enough just like testing this out and then I'll bring it back to Monica and we'll see what happens.

Speaker D: To be very self aware, I still have a lot of learning to do, but I think some of the things that I try to do and my partners also, it's. It's about close collaboration with my ENG partner, my design partner and so forth. Is own mistakes, own when we're wrong. Got uh, it from the rooftop. So I've been in front of the company all hands and told that story that like, hey, I was wrong and it was a great thing that the team drove that. So I think they've seen me personally embrace those failure stories. I also try, although again, still in the journey, I try to share thinking early and really open it up to the team for feedback. And we're constantly iterating our processes. So even right now, for example, as we're thinking about planning what's next, my previous mindset was like, I need to create clarity. I can't share this, these early scratch notes with the team until I have a little bit more clarity. I need to circulate it with a few other senior people and we're trying something new that's like, here's the skeleton doc. What do you all think? How does this change thing that also, that gives them not just time to react and say, hey, why does that make sense and share their unique perspective. It also gives them more time to do some of that discovery research versus the just in time. Here's the guidance on the roadmap. Now you have two weeks to come back with this plan and then lock it down. So those are some of the things that I do that I hope helps influence the culture and the team and gives everyone a sense of, hey, my voice matters. I know how I can make an input and impact.

Speaker A: When you're leading your team and looking across your portfolio and figuring out where do I place like my big bets and how much should I invest and like where do we test, what's your methodology for kind of thinking about that and how much effort, let's say, or how much you want to put into these different tests or these different concepts and giving people room to go explore them.

Speaker D: Yeah, I've seen it may depend on the context of the organization.

Speaker A: Right.

Speaker D: If you're on a zero to one, then that's it. Let's swing for the fences and really lean into taking risks. If you're running a team that's like reasonably mature, the company is counting on us to deliver some engagement, to deliver some revenue, etc. Uh, we try to think about the different buckets, the different that these investments fall into. Here's the sure bets. We know they're going to pay off. Let's fund them, let's figure out the most expedient way to get these wins in. And then there's a next set of things that are either like strategic, we're pretty sure about, there's a couple unknowns and then there's the swing for the fence's venture bets. And I try to keep in mind, okay, roughly we want whatever, 60, 30, 10 or 70, 2010 across those categories. Or maybe it's a more disruptive time and I feel like there's more risk to our business. And I want to swing a lot more for the venture bets. I have that resource allocation in mind, but really the resourcing comes down to here's our goals and objectives across these buckets and maybe, actually maybe the sure bets, I can get these with less resourcing. And I don't need to see exactly 70% of all the people in the team working on this thing. It's uh, just a sense check that can be helpful. But just understanding do I need to deliver to the customers or to the company, that can then help you think across those, like shorter time horizon and then the like bigger swing bets in your portfolio.

Speaker A: Monica showed what the answer looks like in practice. Owning mistakes publicly, sharing unfinished thinking early, giving teams room to do discovery before the plan is locked. These aren't abstract culture principles. They're specific behaviors. And she described exactly how she changed her own habits to model them. The 702010 portfolio framework is how she applies that same rigor to resource allocation, adjusting the mix depending on how much disruption the moment calls for. For our final perspective, here's Mario on what happens when that kind of culture is put to the test. Building something genuinely new. When you launch Copilot and you started watching people use it, what surprised you? What kind of, uh, things did you observe in trends and how developers are interacting with it?

Speaker C: You know, when you have a product, it's funny, even though you're very proud of it, you're also very critical of it because, you know all of the flaws you have, it's like. And we knew, look, we developed For a living, GitHub is a developer company, right? So we knew the places that it was not doing well whenever we were using it on a daily basis. We also knew it was magical, something new. And when you get innovation out there, what really made it for me is how many people used it and how many people absolutely loved it. And um, even though it maybe when we first released it, acceptance rates were like in the 20%, 30%. So just think about this. 70% of the time that we suggested something to you, you did not accept it. So you would think that makes a horrible product, but no, because of all of the value, when you did, people just absolutely loved it. And that was a learning even for me. I do not come from MLOps or AI. So if you were to ask me, Mario, you design a product, you get it out there, and 70% of the people do not actually accept the suggestion, oh, I'm telling you that's not a good product. But in this case, it was completely different. So that surprised me is, oh my God, only a 30% acceptance or 20% acceptance can create something that changes the world on this. And I could see it then. The second thing was this one didn't surprise me. But uh, there were so many other cool things that people did with it. And how many people ended up programming just by giving comments to Copilot instead of having to write the entirety of that code? And how many people did that across many programming languages and natural language? So I could be programming in Spanish an example, or at least learning how to program in Spanish as another example. So I think that was one of the things that also surprised me is can the ability to use, use that model in multiple languages, not only programming languages, but also the way we speak.

Speaker A: That's really cool.

Speaker C: It's interesting because I still remember to a lot of what we do at GitHub. We operate on, um, this set of principles, right? And we're very much, and you know this very well because of uh, the build trap, we're very much outcome driven instead of output driven on these things. And because of that we also value the learning loop or your iteration velocity through that. So if there is a week and we could do three experiments, we learned three times. If in a week we can only do one experiment and we only learn once, so we value learning fast. So the way that we got there was through a bunch of failures, uh, believe it or not is you try this thing and you're like, nope, that did not work from a UX modality. You tried this other one that did not work. You tried this other one, you're like, this is starting to feel like it. So you play a lot with the product, you experiment a lot and then just learn and learn. And if you do that and you have some people that have good taste also have some expertise in the field because you do need expertise in developer tools. Then you start getting to a place where you're like, okay, now I could actually iterate towards something that is a good product. So those are the things through a lot of iteration and, and then a significant amount of engineering work then to figure out, okay, how to make it faster, how to do this thing called fill in the middle. Which was very important to us because a lot of development doesn't just happen on a new line at times. So you do want to fill it in the middle as well. So then you start marrying the technology with the product. But it was through failures, which is something like, I don't think the product discipline talks a lot about. We like to celebrate a lot of the successes of it, but I actually think you end up creating a better product through failures than through successes the majority of the time.

Speaker A: And that's a wrap on today's episode. There's a big gap between teams that talk about experimentation and teams that have actually made it part of how they work. And I think David, Monica, and Mario each showed a different piece of what closing that gap looks like. If you want to hear the full conversations, check out episodes 44, 227, and 223 to build stronger product management skills and learn practical approaches you can use day to day. Head over to Product Institute to learn more. Thank you so much for listening to the Product Thinking podcast. We'll be back with another episode bringing you practical perspectives from across the product community. We'll see you then.