From MVPs to FAFO - Iterations, courage, and what it really means to learn fast in Product.

Speaker A: Welcome to this episode of Practical Product Management. We've never had two guests at once, so this is exciting. These are guests that have been with us before, which we've never done before, so that's exciting. Um, and this episode I'm loosely titling around and find out what fafo and the reason why we're calling this that, uh, is we are going to talk about iterations, developing products with iterations. So we brought together a, uh, power trio, but there's two product managers because together we make one and then a, uh, TPM leader and an engineering leader. So we ought to be able to figure out how to do iterations. I'm just thinking between the four of us, we ought to be able to figure this out.

Speaker B: So I can predict we're going to get it wrong.

Speaker A: Oh, that's very sneak peek into iteration.

Speaker C: It's not going to work out.

Speaker D: Right.

Speaker A: Well, since you're talking, why don't you introduce yourself and then we'll go,

Speaker B: uh, hi everyone. I was, I was, I'm so excited to be back. I had so much fun the first time. I won't lie. I was nervous and at the end I was like, oh, I can just keep going. Like, let's make it a three hour conversation. Um, but so thank you for having me back. Back again. And for those who haven't, uh, seen me, I'm Alicia Crony. So I'm director of Software engineering at Moderna. I lead their global e commerce. Um, and yeah, iteration. Iteration's a iteration sort of everything. I feel like that's how honestly I live life.

Speaker C: I up, I learned some stuff, I try again.

Speaker B: So, so here we are. Right on iteration. I don't know.

Speaker C: Right.

Speaker D: I think that's.

Speaker A: This is one of this episod. Right? Right. All right, Gino, tell us who you are.

Speaker D: Fu. Uh, I heard F U F on there up and find out

Speaker B: new iterations.

Speaker A: You know.

Speaker D: Well, hello everyone.

Speaker A: Right, right.

Speaker D: Uh, hello everyone. My name is Gino White. I am a, um, what am I? I'm a. I'm a problem solver. That's how I introduce myself most often. Um, I've led product development teams in lots of different places over two plus decades. Currently, um, operating as a consultant, helping companies to solve problems, um, as relates to software, to help them get software out the door right when they, when they struggle to do that. Um, I am my second time on a podcast. I am super excited. I was so nervous on the first one. Like, my God, I was nervous and I don't know why. Like Alicia Said at the end of the hour, I was like, these are. It's the best time of my life. But Marilyn and Leah are two of my favorite people in the history of the world. And, uh, so we've never had a bad time together. Except that one time.

Speaker A: Except that one.

Speaker C: We don't talk about that one.

Speaker D: But we don't talk about that.

Speaker A: Yeah.

Speaker D: So anyway, thanks for having me back.

Speaker C: Awesome. Well, I'm m super excited to be here with you all, um, because you are my favorite humans. Um, and I'm really excited to talk about iteration, or fafo, um, which I'm now going to have to tattoo on my knuckles. So I'll imagine. Yeah. Um, but, uh, what do you, like, what do you mean? Let's start with the definition because, like, my life these days has been like, trying to get people on the same page with the same words. Um, because we all mean different things when we say the same words. So, Alicia, what do you mean when you talk about iteration, iterative development?

Speaker B: Yeah, I think of, uh,

Speaker A: I mean,

Speaker B: for me it's like a collection actually. So I think of it more as a process and a way of framing, but learning, growth, not being too tied to perfection, not starting with this is where we're gonna end up, but kind of stealing from what Gino even said about. He thinks of himself as a problem solver. Starting with the problem or even to maybe even challenge it. Like, how do you know you even have the right problem? You might need to iterate on the problem and figure out what, what really fits. Right. Where do you want to go in on? So it's, it's, it's almost not to kind of use a grand. It's almost like a philosophy in a way, right. Of like, how do you approach your work, um, and work and collaborate with the people around you? That's m. My take. I'm curious. Add to it.

Speaker D: Um, yes, yes, and yes to all of that. When I hear iterations and iterative development, which, by the way, I don't know if we're using those terms in all the same way. I think we're going to figure it out over the next few, uh, minutes here. But when I think about that, I do think about. I go back to when I learned Agile development practices at Amazon a long time ago. Um, and I really embrace the spirit of agile now. You can do it in all different sorts of ways. It's done in all different sorts of ways. Whether you do it differently or not, it's going to be done in all different sorts of Ways depending on where you go. Um, and. But the spirit of it still, uh, rings true to me no matter how it's being implemented. And one big part of that is the, the notion of this, uh, incremental iterative improvement. Right? You give yourself the opportunity to um, get better quickly over time in these small chunks and that's. That there's a lot of, that's very powerful, right, when you just sort of embrace that. Like I, I don't have to be perfect right now in this sprint or this iteration or whatever, but I am working towards something much greater than what we, what we have.

Speaker C: Yeah, I like that. I like the notion of even the problem is iteration. Um, and I think that iteration is changing now. Right. So I, um, know like these gray hairs are evidence that I've been around for a little while. Um, and I remember when iteration was waterfall, uh, right. And big documents were written and iteration might have been like a year. Certainly, um, was more than quarters. And then we like magically went to agile and we were able to push to the cloud. And then iteration was like every two weeks. I mean we tried it for a little while as a group is every one week. Um, and I don't know if you remember how chaotic that was. It was probably a little too much. Um, um, so two weeks was kind of agile. But I don't like, I'm really interested to see where iteration goes as we have tooling that enables us to move faster to identify problems, um, and explore problem spaces and solutions in a more rapid fashion. So I think I love the notion and um, now I'm going to throw it at Leah because you always have a really interesting perspective. I love the notion of being able to iterate on the problem space itself as well. Um, Leah, how do you approach iteration? What does it mean to you?

Speaker A: Yeah, I mean I so like you. I've been around a while and when I first started we were writing like PRDs and things were taking a long time and I've turned down a job once because they only did four releases a year. And I was like, that's not fast enough because I was, I started doing, you know, scrum at a company that. That was the first time I'd ever done that and this was before Amazon. And so I think you're absolutely spot on that it is iteration about the problem. Uh, sometimes it is iteration about what we think we know, quite frankly because we start out from a place where if we're doing it well, we gathered some information, we've talked to our users we have a sense of what problem. We're trying to solve the problem. We may get quite, you know, be quite, uh, a little off here or there and need to go back to the well. But also, as we start to solve the problem, we may go, oh, yeah, that's not gonna solve the problem because we start from a place of what we know, and there's always a whole lot we don't know, and everybody's uncomfortable with that, but that's the whole point of what we do. Like, if we're. If we're supposed to know all the things, just hang it up and go home. Because you're not going to know, right? Like, there's just no way to know the market. I mean, I've worked in travel a number of times now, and travel is so volatile. So volatile. And it's been one of the things that's really. I mean, I thought E commerce was volatile, but travel is insane. A little skirmish breaks out over here, a little war breaks out over there. A little, you know, the drop in the, you know, the NASDAQ happens, this person gets elected, this happen. And it's just travels just like, ding, ding, ding, ding, ding, ding. And you just never know. So if you can't iterate on the fly, you're going to build the wrong thing every damn time. And so for me, it's really being, you know, to your point some. And that's getting faster and faster to the point that there's a lot of tools out there that now just can predict intent based on what's happening and then go, yeah, you should stop doing that and do something else. Right? So there's a lot of things happening quickly. Um, and yet we're still real slow in a lot of places. And I'm like, why are we so slow still? Right? And it's just, we don't. The iteration is not. We're still learning how to iterate even with all these tools.

Speaker D: Are we are. That's interesting. Are we slow to produce or are we slow to learn? Because really, the whole iteration thing is it's. You're trying to learn faster. Like, we, we talk about rapid results and things like that, but you're really trying to just learn faster. The faster you learn, the more you learn, the more you learn, the better job you can do. So I'm curiously. Do you think we're. Are we slow to learn or are we slow to produce tangible results? Or are those the same?

Speaker A: Yeah, I think, I think we are slow to, uh. I think that learning is there often like, what we need to learn, the information is there so we can learn. But deciding to pivot, deciding to switch, deciding to do something different, we're still sort of paralyzed and hunker down. And then we have to. If we have to go back out to the committee to make the decision, if you don't trust the team, if you don't trust the people who know how to do the thing that they're trying to do, you're going to be slow.

Speaker C: I was just going to say, do we even know how to learn? Like, you know, we've been talking about outcomes and learnings for a long time and being able to take information and adapt against an outcome. But to be honest, like, even today, I've had people tell me, we've already done that. You don't need to know anything about it. And it's like, maybe I do. If you actually want it out in the universe, maybe you do need to have me know something about it.

Speaker A: Like, uh.

Speaker C: Like, are we even good at learning? Sorry, Alicia, I interrupted you. What were you gonna say?

Speaker A: I told you, Alicia, you're gonna have to push.

Speaker B: I know. Just gonna say, Lee, I actually think you. You nailed it on the head in the beginning. You. I think you said, uh, people feel discomfort right, when they. To be able to say, oh, I don't really know this thing, or, okay, I know this, but I know that I don't know all these other pieces. And I feel like if you can't be comfortable with the discomfort, you're never gonna even get to that stage of trusting your teammates. Because you're right. We have to all accept that. We're all like, okay, maybe, you know, Gino might not know this. That's okay. I don't know this either. But I can trust Gino to work with me on let's figure it out together and we sure learn together.

Speaker A: Question. Yeah, we know that he's going to ask 7,7000 questions.

Speaker D: I'm asked one or two. One or two.

Speaker C: I see why mostly it's going to

Speaker A: start, but, yeah, well. And I think, Alicia, what you just said is really important to me because I have been in boardrooms and in situations where people have said, what's, uh, literally what's going to happen if we do this? What's going. Hell if I know. I don't know what's going to happen. Like, I can only do the thing and then follow what happens and do the next thing. And I have said. I can think of. I mean, I said this three weeks ago. We have to Be willing to be uncomfortable and hold. Hold our spine. You have to have a spine to do the work we do.

Speaker C: Yeah.

Speaker A: And you have to be willing to let it ride just a little longer to see what's going to happen. Right. And often we're not.

Speaker B: And it has. Yeah. Uh, and it has to be okay to say, we can't predict a future. Don't exactly know. Here's what we're doing to find out. Here's what I, like, you know, I'm currently expecting based on this information and just let go. Uh, let go and move forward.

Speaker C: Yeah, I actually. Interesting. Um, so, uh, just, like, here's what I'm expecting. I actually, I hope you guys are. Are like, all right, let me put my thoughts together. Um, I am gonna assert some things. I think if you already. And, like, you think you know the outcome of a test. Like, if you're doing an A B test. If you already think you already know the outcome of a test, don't do the test. Like, you've already, like, biased things. It's, like, stupid. Um, I actually. I remember when we were doing optimizely tests a really long time ago at StubHub, like, we were doing six and seven variations at a time because the first two were trash. The team already had a bias. They already knew what they wanted to see. And you really don't get interesting, um, information until you sort of, like, start getting way out on the edge of how to drive an outcome. And that's where I think a lot of innovation happens. And I think a lot of companies don't necessarily want to do that, like, from the top down. They already know what they want. They already have, like, architectures and screens, like, already on a thing that they've showed to the board. And they don't actually really care about innovating in the space. They just want that thing that they can point to and be like, well, this thing is going to give me access into this opportunity. Um, and from my perspective, like, that is legacy thinking, I think. Um, and it really robs you of the opportunity of innovating in a space and focusing on a customer problem and freeing yourself. Gino, to what you said about, um, learning, like, you're not learning at all. You're just outputting stuff.

Speaker B: You're, like, hamstringing yourself because you're using iteration and testing as your confirmation bias. Yeah. Rather than, like, you said, Maryland, like, using it as a. Let's actually go into the unknown.

Speaker C: Yeah.

Speaker B: Learn things and innovate.

Speaker C: Like, helping people get really comfortable with that discomfort of not knowing. I know there's a space I want to be in. I know there's an overall problem to solve. I don't actually know how I'm going to solve it. Um, but I don't want to do it in the way everybody else has done it because then you're just another me too. Pick me, girl. Pick me. Product.

Speaker D: Okay. Yeah.

Speaker B: I don't.

Speaker C: What is a pic Me.

Speaker A: I like it. I'm doing the right thing. I know what to do. What do you think about Marilyn? What do you think about the idea of if you're running a B tests that maybe are a bit trash and the minute you see something that you don't can't explain, digging into that and saying, okay, now let's, let's figure out, let's, let's investigate why customers are doing that.

Speaker C: That's my favorite part, chasing the hell

Speaker A: out of that, whatever that mystery is.

Speaker C: I think that the most interesting innovations come from when you get a response like what the hell is that? Or that's super curious. I don't understand that at all. Um, and I do feel like sometimes leaders particularly want to be seen as all knowing, competent, have the plan, and so they don't allow themselves the space to even kind of go, uh, like that seems super weird. We should go and investigate that thing over there. You can sort of explain away things in a way that eventually decays your product market fit because you're no longer curious about what your customer's doing. You're just so convicted about the thing you're building. I think you rob yourself of it, of opportunity when you do that.

Speaker B: Do, do you think that lies in the incentives or. Mhm. Maybe it's like, is it a gap in. I don't know how we talk about it. Do you know, like, what do you think drives that?

Speaker C: I have, I have opinions. Did you see my face?

Speaker A: I have one too. All right, everybody gets to have an opinion. What's yours, Marilyn?

Speaker C: I think the answer to your question is yes, Alicia. I, uh, think that there are corporations that have corporate scorecards. And those corporate scorecards are. They're made in advance with targets in advance that leaders pretty much know how to get to. So they're predictable. And anything that allows for deviation from that metric, which is probably not tied to overall business success, it's not really a very well constructed okr, um, is too risky because that impacts the executive's bonus. And I've actually seen this in corporations in the past. And so there is no space for Rationalization, reducing tech debt, reducing overhead, innovating on behalf of the customer. Because all of that introduces risk. And that does not.

Speaker A: That.

Speaker C: That is. That is at odds with the corporate scorecard. Um, which means that you're not going to invest in it.

Speaker B: Right.

Speaker C: Or if you do, you do it in such a marginal way that it's not impactful. I also, again, think that personally, people that are on a leadership track need to be seen as authoritative and right. And know everything. And so to be able to be like, I don't freaking know, but I'll find out. Such a huge risk for them personally. Um, that's my opinion. All right, Leah or Gina, who's next?

Speaker A: You go.

Speaker D: This is the last point. Like, I. Marilyn, that is, man, biggest red flag in the world is when I talk to a leader or an executive somewhere, and they already have it all figured out.

Speaker C: Yep.

Speaker D: Um, and they're just closed off to other things. And it. This iteration is an iterative thing, though. You can. You can be confident that you know something, but you still got to be open to learn, willing to be like, surprise me. Show me some new data that proves or disproves this hypothesis that I've been sitting on for a while. Um, and. And if you're. If you're. Even if you're unwilling to say that out loud, if you're willing to entertain this, let's do things in iterations, it iteratively, let's learn that way. Then you're gonna. You're gonna discover some stuff.

Speaker A: So, yeah.

Speaker D: Yep.

Speaker A: I think. Yeah, I agree with that. I think. I think there's a. Definitely a courage gap.

Speaker B: Right.

Speaker A: I think along with an incentive, you know, incentive bias, I think there's a courage gap. And I think if. If the number we have in our head, that is, this is the number we have to meet this year, and this is the way we've always done that. So we're just going to keep doing that same nonsense, and we're not going to take a risk, even if we see that the risk is moving us in a new direction and telling us something new. Sometimes that's exactly the reason to be like, let's not. Let's stop. Let's not do that, and let's go backwards. And it's just like, well, then why are we. Do I. I. You. You guys know me well enough to know that I'm the. I'm the one that's like, I'm not your girl. That's what you want to do. I'm not your girl.

Speaker C: Yeah.

Speaker A: Right. Like, if you want to just keep going, doing the. You can do the same damn thing you did last year without me. You don't need me.

Speaker C: Right.

Speaker A: I think that there are likely people who do what we all do that are suited for that. Just give me an order and off we go. And I'll do the thing you've asked me to do and I'll collect my bonus and off we go. But I don't think anybody in this room is that. And it sure as hell not as much fun.

Speaker C: No.

Speaker A: Right. So it's. I mean, it's not fun to me to be like, make this number keep going up and to the right. Okay.

Speaker D: Hm.

Speaker C: Cool.

Speaker A: Buy more traffic, off we go. Tell the marketing team to do that. You don't need me for that. Like, just pay Google some more money and off and we'll do. We'll make that happen. I don't care. It's like, it doesn't matter to me if that's what you want to do. You don't need me. Right. It's not fun.

Speaker C: M. I think that the counter message to that, Leah, though. And don't you. Don't you agree that like, just because you're here to like, push the boundaries and understand customer problems and really dig into sort of innovation creative solutions does not mean that you're approaching the problem without thought.

Speaker A: Right.

Speaker C: Without structure, without being able to measure progress. And I think there's this like, weird dichotomy. And um, I. What did you say? Courage. You said there's a courage gap.

Speaker A: Yeah.

Speaker C: Um. And I, I love that term and I'm now going to steal it. Um, just because you are down to experiment and iterate does not mean that you're approaching this with chaos.

Speaker A: Right. Usually, uh, it means the opposite because you've got to have all the metrics in place, monitor to see what's happening. You've been down in the weeds in places that no one else has been in the weeds because you know you're going to have to account for what you're learning.

Speaker C: Yes.

Speaker A: It usually means there's actually less chaos. It's more tightly constructed. But to everyone else it's like, this is madness. Okay. It's not.

Speaker D: Right.

Speaker A: Yeah.

Speaker B: I've often been in rooms where I can tell that approach feels like chaos.

Speaker C: Ah.

Speaker B: And I'm sort of self reflecting and I do think in some, in some degree that's due to how we talk about it. Sometimes say more like, so something Marilyn, like you were saying, like, where you, when you started iteration, looked so Differently. Right. Like the time cycles with different sort of the approach was different. And so I'm realizing like sort of as I, as I'm sitting here, what if it's just like we went along this journey and we haven't taken everybody along. And so there is just like this deep seated. Right. Whether it's in the incentives and in the structure and like how do you get promoted? Uh, not often by saying I did 100 experiments. I haven't cracked it yet. But you know, maybe.

Speaker A: Right.

Speaker B: Like it's all of these diff. Like all of these pieces. I feel like just push us to your poilet to not have courage. Yeah.

Speaker A: Well, don't, don't accuse Marilyn and I are either one of ever not taken everybody on the journey. We've neither one ever gotten that feedback.

Speaker C: I get that you over communicate. Um, you give us too much, give us less.

Speaker A: That's why we have.

Speaker C: You know, I wonder. Alicia. And this is another, like I seem to have soapboxes every time we get together. Leah. So like I'm just gonna hop right back up on my soapbox here. Um, I wonder for such a long time there were so many businesses that didn't have tech embedded or tech enabled like there were. There were businesses that were purely manual. And a lot of those businesses, like maybe financial services and, or others, um, when they did start to use technology because they have to, they see it as again an operational activity. It's not core to the business. It's not, it's not where the IP starts to come from. Companies that are digital first obviously don't work like that. And I feel like those are the companies where the executives understand technology, they understand iterations. It's when you get to some of the um, what I'm going to call legacy thinking, um, where that fear starts to creep in. I think we see that in a lot of industries that are trying to bolt on tech and become technology companies and be all A.I. first. Um, that don't think that way in reality. Like it's a marketing ploy more than a mindset. Um, and I think that fear is there, there.

Speaker B: Sorry, no, no, I actually think you, you kind of described the problem. It's isn't even saying I want to use technology because I want to use technology is not iterative. Right to your point of. Even if you. More like, uh, okay, we're trying to solve these problems. Oh, wow, if we apply this like we can get to these places, I mean hypothesized then, then you're building it naturally and it becomes part of your innovation driver rather than this, like, necessary evil. Yeah,

Speaker C: yeah. And maybe your experiments are fully manual. Like, you guys all know that I'm like a huge fan of BJ Fogg, and when we get into snap testing, a lot of those things are super manual to see you want to test the psychology of an idea in the day or two days before you bring in the engineering team to build everything because it's exciting, expensive, not, you know, it's not a, it's a way a business operates. It's not the way the tech team operates. Um, innovation, experimentation, iteration should be the way a business approaches problems. And I wonder if there is a larger disconnect between, you know, those that have to report to the street and have their numbers and have a level of certainty and then this notion of experimentation and this idea that it feels like chaos.

Speaker A: Yeah, but I mean, that you're. What you're poking at to me is also the problem of venture capital money and private equity money. They don't understand any of this shit. And yeah, I'm talking to you guys, if you hear me and you're mad. Yes. If you would like to have it explained, one of us would be happy to. But they don't understand. They don't understand how, how it works. But so when they put money into something, they're like, oh, they run these numbers and say, okay, when it gets to this point, we need get our return on our investment. Okay, good luck. Uh, good luck. But that's not how, that's generally not how startups work. It's generally not how businesses work that are acquired by private equity firms to turn them around.

Speaker B: Right.

Speaker A: If you acquire an old school company and want to make it digital first, it's not going to be that on month 18, you get your return on your investment. Your model is the fact that they don't know that after all this time is crazy. So. But I, but I think that goes to the bigger problem of we are operating in one fashion and often the business is operating in another fashion because they are the business.

Speaker C: Yeah.

Speaker A: And, uh, you know, I have a problem with product not being the business. Right. Like, it drives me crazy when the business needs to talk to product. I'm all, I'm sorry, was I not the business? Is that not me? Right. So hold on. Right.

Speaker C: Who am I?

Speaker B: Yeah.

Speaker C: Or even engineering. Like, that's, I mean, I think so. I've seen both Alicia and Gino do this. Like, uh, they're like the separation between people that are labeled the, uh, Business and the people that actually make the thing work. Like, that gap is killer. Don't you think that air gap is, like, deadly?

Speaker A: Yeah.

Speaker B: Yes.

Speaker D: Yep.

Speaker B: Yes.

Speaker D: And speaking to that, like, it's frustrating.

Speaker A: Go ahead.

Speaker D: Go ahead, Lisa.

Speaker B: Oh, it's frustrating. The number of rooms I've been in where it's like, well, you're not responsible for this dollar figure, right? Like, is it like. Like, why are you opining on, like, how to make more money or lower cost or any of. And if there's goals, it's like, you just built the thing.

Speaker A: Just build it.

Speaker B: I don't think so.

Speaker D: Actually,

Speaker B: to borrow from Leah, I'm not there.

Speaker A: That's not me.

Speaker B: That's not what I'm going to provide. Yeah.

Speaker A: What were you going to say?

Speaker D: Going back to this iterative approach and iterations thing and along these lines of, um, business, working with the product development team and the separation between the two, one of the things for me. So, um, as a TPM or leaders of tpm, or as a person held accountable for delivering the thing to market, um, oftentimes you're not exactly sure if this engineering team can do what you're asking them to do. And so I use iterations to learn how they operate. I also use the iterations to help stakeholders and business partners and others also learn how we operate so that they can have a more realistic view of how things get done. So I think it's very important to be, um, to be inclusive in that process. Right? Bringing those stakeholders, those business people, whoever they are, marketing, sales, bringing them into the fold so that they can see this happen, so we can learn together. Right? Because one of the things we're learning is just how, how this whole thing operates. And oftentimes they're surprised because it, um, they start to understand that, oh, we have to do things differently too. They being the. The business side, like brothers, marketing, sales or operations, whatever, they have to do things differently too. And they learn that when they're a part of this iterative process.

Speaker A: I was just gonna say one of the places that I learned that the most was when I worked at Providence. And we work. We're a digital, um, innovation lab in the hospital system. And so we were building things that had patients and doctors and nurses and like, all these different pieces. And so we would do our Sprint reviews and invite the neurologists and nurses and all these people to. And they were like, oh, it was so different for them to watch us, like, demonstrate what we've built in the last two weeks. Uh, you know, and it made it more real for them. And they're in. Their requirements got better.

Speaker B: Right.

Speaker A: Because they started to understand, like, we need this kind of information so that we can get precise on what you're asking us for. We don't know, you know, you're the user.

Speaker B: Right.

Speaker A: And that was a really, it was a really helpful learning experiment for a lot of people. But for me, because there was like, it was one of those places where there was multiple users on a, on a telemedicine cart, for instance.

Speaker B: Right.

Speaker A: Um, and it was important. So.

Speaker C: And you're not asking questions just to be a jerk. You're asking questions because you really need the answer. You're interested. Um, I think one of the most interesting, uh, approaches of, um, this is sometimes you don't know how. Not only do you not know if the team is capable of building the thing, you actually don't know how you're going to solve the problem. For me, I think if you can get a raw rich customer problem and bring that to the team, like, uh, I don't know what to build, I don't know how to solve the problem. And this is where your, you know, your engineers. I've seen, uh, I will, I will give one example for MasterCard. I don't usually talk about anybody and I never name names, but there is, there is an. There was a problem we had in MasterCard we didn't know how to solve. And it was about how to understand the customer's context when they were on a screen so that, you know, someone in the world could help them with what they had going on. And there was lots of ideas. There were lots of ideas. And people were like, let me write the requirements. And I'm like, no, no, no, no. What if we just, like, what if we just soaked in the problem for a little longer and then you guys can ideate, you can brainstorm, we can think about the problem as a group. And the solution was genius. And it came from an IC engineer who, like, really deeply thought about what was happening. And to Leah, to your point, the people in the ecosystem, like the customer that was having the problem somewhere in the world, the customer service person who is going to try to help them either synchronously or asynchronously. And then an engineer of the, you know, 70, 50, like 100 teams, an engineer on one of those random teams that also had to understand what was going on so they could potentially say if it was a feature or a bug or they need to fix it.

Speaker A: Right.

Speaker C: He went home and thought about that problem and came back with a, uh, with a prototype like over a weekend. That was brilliant. Yeah, yeah, was brilliant. Right? And so like that, that's the magic right there. Not, not someone being like, I got a requirements doc.

Speaker A: I mean, listen, we, I mean we can name products that were built like that. Square was designed by the guys at Twitter so they could run a, uh, they could actually manage a transaction at a conference. They did it over the weekend and put a, something on a phone so that they could run it. That was a whole industry opened up. Yeah, because one of those engineers working for Jack Dorsey was like, oh, let me fix that for you, Jack. Right. Like I got an idea. Right. I mean that stuff happens. Um, it's brilliant when it happens. But yeah, I agree with you 100%. Like though, that's like, how do you sit in the problem long enough and say use some creativity folks. Figure it out. Let's be great.

Speaker D: Yeah, Problem solvers.

Speaker C: Um, so now does that mean we all get to adopt your title? Do you know? Are we all now problem solvers?

Speaker D: I think we're all problem solvers.

Speaker C: I'm taking it.

Speaker A: I like it. I like it. Just because you're the chief problem solver doesn't mean we can't all be problem solvers.

Speaker D: No, you can be problem solvers too.

Speaker A: Assistant to the chief problem solvers.

Speaker B: There's space. There's space. I was mulling on because that magic, right? Like, I mean, I know we've all had those moments, those projects, but I'm sort of mulling on, like

Speaker D: do we think.

Speaker B: I think, you know, it's kind of a controversial question. So we like Agile Scrum, which is, I feel like an attempt to make up this magic into a repeatable process. Right. Uh, do you think it gets in the way? I'm, um, actually curious, like in your guys experience, is it like you start there and then you almost iterate on top of that, right? To get to your own sort of version of what that process is for your teams that works.

Speaker A: I mean the answer is like it always is. It depends, right? Like uh, my opinion is that those are all just languages, communication models. I don't think of them as actual frameworks that always work in every circumstance. It's all just how do we communicate? And that can be Scrum, it can be a document, it can be we sit in a room and build it together. I think it depends. Yeah, right. I don't know, Marilyn.

Speaker C: And that's why I was grinning like a lunatic, because Alicia, you just like came Back to the whole purpose of the podcast, which is like, it, uh, like you can't follow the checklist and then be like, look how magical I am. Like, it really depends. It is. And, and the goal is to move faster, to solve problems faster, so you can learn faster. I think that Gino and I were just talking about this earlier today. Like, as you start to apply some of these AI emerging gen AI tools to the job families in the act of scrum, I, um, think it would. It's incumbent or it's. It's important for people to keep the why in mind. Like, what are the larger whys that you're doing this? It's not because some moron on a podcast said you should use this tool because it's now the cool thing to do. It's just not. And if you don't have that why in front of you, you're going to apply a tool. Um, you're gonna get rid of half of your company and then six months later you're gonna look like a dummy because you're gonna have to hire everybody back because you didn't really ground yourself on the problem you were solving. You just applied a framework or a tool.

Speaker B: Yeah, yeah. Like you said, you were a tech company instead of becoming a tech company. You're like, where AI instead of. Okay, naturally growing. Yeah.

Speaker A: Oh, your agents aren't actually here.

Speaker D: Right?

Speaker A: Go ahead.

Speaker D: Well, speaking of which, like, um, this might be a little more salacious, controversial. I don't know.

Speaker C: We're all in.

Speaker D: I'm going to make this statement on the tail of what Marilyn just said. I don't, um. As an engineering leader, I don't know if I need to hire more developers now or just more product managers. Like, I think. So what is, what is the normal ratio of. What is an ideal or a good ratio of product manager to engineer? Marilyn and Leah, Like, I know it depends. I mean, look, I worked in environments where it was like two developers for every one product manager. And that was like too many product managers. Um, and it's too many thinkers. So right is they're smoking up stuff that the Devs can't build.

Speaker A: 4, 4 to 1, 5 to 1, 10 to 1.

Speaker D: Somewhere. Okay, so let's say somewhere between 4 to 1 and 10 to 1.

Speaker A: Depends on what you're doing. Right. Somewhere in that room.

Speaker D: What if we does the, um, does the rise of AI code generation tools means that that ratio will flip or should flip, because now, given the promise of. Given the prom or given the, um, I don't know if it's a promise. Look man m how people talk about these things. You could just talk it. You write code and you build code.

Speaker A: Yeah.

Speaker D: The product managers understand what needs to be built. They understand the problems they're trying to solve. So why isn't it four product managers now just talking into these things, generating code and you got one developer that's just kind of making it actually work in an enterprise fashion.

Speaker C: Like I love this conversation. This is my favorite conversation. So I'm going to go back to it depends. Um, and I think so. I think you can do more, I think you can do more with fewer people. It's no longer going to take a whole team of people to do a thing. Now do lines start to blur? Um, absolutely. I think the maturity of our tools is going to dictate which direction they blur. I'm going to uh, elaborate on that just a little bit. I don't know if it's a really strong front end engineer that has a great grasp of the customer that starts to bleed down into product management or if it's a really strong product manager that can speak words into a tool. But I will tell you that the maturity of the tools today, um, is cool for like putting out a simple website, but you're not getting enterprise level products out of these tools right now. And if you think you are like again, I don't know what you've been smoking, but please share. Um, someone still gotta check for applicability and maturity and security and all those things. Right.

Speaker A: I think what it's doing. Alicia probably has a strong opinion about this, but I would just say one of the things that I've seen so I mean shout out to the FIGMA people for FIGMA make, they're crushing it. But I think what that does is it allows designers and product managers to get buy in faster.

Speaker C: Mhm.

Speaker A: Than hand it actually over to people who know how to make a thing.

Speaker C: Yep.

Speaker A: Right. Because if you can produce a thing and everybody goes oh my God, it's so beautiful, when can we have it? Then I, all of a sudden we've, oh, we've tested this, we've used this to do some testing. We've done, you know, we've taken out into the market, we've done whatever, then you actually you're, you speed that process up from engineers having to build something so that everybody can look at it and then go through that cycle. So that's, that's, that's one thought. But I don't know Alicia, as the engineer in the room, what do you think the pressure.

Speaker B: No, I, I love this time, to be honest. When I'm digging into where do I want to use AI, how to use it. Like, sometimes I'm like, oh, should be an ic, because this is just, like, this is just pure joy. But, uh, I honestly think it depends. And so I think it depends on where do you want to innovate, what sort of direction are you trying to go into? So, yeah, uh, and I think in all honesty, you. Some of the same questions that, ah, as an industry we've always asked ourselves still apply. Like, what I mean is, do you build it in house or proprietary? Or do you rely on something out of the box? How far can that out of the box get you? What corner might you paint yourself in if that's the direction you go? But do you even have the expertise to hook up multiple different models, right? Decide how to fine tune those models? Understand exactly. Like, how do you deploy system like that to production, manage it, observe it, because it's different. We're all fairly well grounded now in common metrics for a production system. Latency load. I think those are words, everybody. It's like, okay, you kind of understand what they are. Uh, but when it's AI or machine learning, uh, systems in production, are you measuring like your hallucination rate? Like, does everyone even understand, like, ooh, what that means? Or like how many times you run into your guardrails, right? Like, how many times did you have to escalate to a human in the loop? Like, all of these types of metrics are new. And I do think, like, how you think about those. Just both, like, engineer is probably TPM and like, delivering a system like that and a product manager, like, we're all still learning and the knowledge is changing very rapidly.

Speaker C: And what's the ROI of the thing? I think that's like, that's the piece that I don't see discussed enough. If we build this thing that is supposed to achieve this business outcome, do you get a return on this? Is it 5x, is it 10x? What happens when the cost of compute or the cost of your model goes through the roof?

Speaker A: Yeah.

Speaker C: Do you have to turn it off?

Speaker A: Uh, yeah, that's a good point. Listen, we are out of time. How did that happen? So here is your, here's your single lightning round question. You get, you get to vote and then decide you can elaborate a little. MVPs. Multiple viable products. Love them, hate them. They're. They're real. What do you say to an mvp? What does it mean to You. Who wants to go first?

Speaker D: I think the word is trash. Now, um, because what is. It's, it's, it's never, it's. They're never actually minimal or most often they're actually not viable. Right. And I wouldn't even call it a product. Most of the time, it's not a product. So, look, look, here's a baseline of what we want to mvp. Uh, in my mind is replaced with. Here's a baseline of what we're trying to learn in this given amount of time. And we just have time enough to do this stuff, and then we'll learn from it and then go from there.

Speaker B: Yeah, I love the you came out the gate trash word.

Speaker C: I love it.

Speaker B: No, I, I, I, I actually agree because, um, just too often I feel like it's like people just go, oh, it's just the mvp. Like, don't ask too many questions. Don't poke too deep. Just put it together, put it out there.

Speaker C: I'm like, that so missing the point, right?

Speaker B: Of what this is for. So I, um, I'm more a fan now of, uh, prototypes lately. Like, it's a more recent one for me than Images M. Because you all

Speaker C: know that once it's in production, it's never coming out, right?

Speaker D: It lives forever.

Speaker A: I mean, hell, someone's gonna go sell it as is and ask you to stop doing that. Something else.

Speaker C: They've already sold it.

Speaker A: They've already sold it. And now it's. You're already like, oh, God, you want me to build something else? We're not done with this. It's working.

Speaker C: Um, can we add minimum lovable products

Speaker A: to the trash turn?

Speaker D: Oh, I was just about to say I didn't. Maybe that term has been around for a while. I just heard it for the first time, I think, two years ago. I'm like, what?

Speaker A: What's the difference?

Speaker C: Dumb. Um, it means your MVP sucks and no one loves it.

Speaker D: Right? And then I'm like, but I've never met an MLP that I actually love left.

Speaker C: No, they're all trash, too. They're all trash.

Speaker A: So what we're, what we've, where we've come to is that instead of worrying about MVPs, we want the team to around and find out.

Speaker B: Yes.

Speaker D: Around and find out. What can we learn?

Speaker B: I vote for that.

Speaker D: Boom.

Speaker A: That's. That's the model. There's the book title.

Speaker B: I mean, honestly.

Speaker A: Yeah, but it's written. This is good. You guys want to write a book?

Speaker D: Yeah. From MB from MVP to fafo. That's what it is.

Speaker C: Let's do it.

Speaker B: There we go.

Speaker D: Paradigm Shift.

Speaker B: Discomfort of Learning.

Speaker A: That's the title of the podcast. There it is.

Speaker C: I actually think we should rename ourselves and keep this conversation going.

Speaker D: Love it, you guys.

Speaker A: Well, this was fun. Uh, thank you both for coming back. We will, uh, maybe we'll just make a habit of having you on every season a couple times just because everybody, you know, why not? We have plenty to talk about.

Speaker B: Thank you, guys. This is. This is the. This is a fun forum. Yeah, it's great.

Speaker A: It is fun. It's fun to just talk. So thanks for coming, and we will see everyone in the next episode. Sam.