#194 Brian Donohue: Intercom threw their playbook out the window when AI got good - A case study on questioning your mental models.

Speaker A: I'm so glad I'm talking to the VP of Finn. I want to hear the behind the scenes, like I want the documentary.

Speaker B: I think what we did early is the stuff that folks, everyone eventually did is what's like the easy stuff you can do, summarize, reword this sort of stuff that we could put in our inbox. So it's not like one big decision. There wasn't one big fork in the road. It was this continual challenging. Are we thinking big enough here? This incremental big swings incrementally added on adds up to a really huge swing. So Fin is our AI agent for customer support and it's been in the wild since like June 2023. So over two years later we finally shipped kind of what seemed like small UX improvements in the messenger. It was crazy, bad ux. It was the wrong question to ask at the wrong time. We've got this graph we show anyone who will listen of look where this gone from like 25% when we launched, it's now looking at today it's now like 65%. So that's across all our paying customers that average rate. And it's just this beautiful incremental improve, improvement in the products value that we're delivering. But I definitely feel like hey, we're in a time of builders. You need to be more happy to be here. I'm Brian Donahue, I'm a VP of product at uh, Intercom focused on building Fin, our AI agent. I've been at Intercom 11 years so been through the ups and downs and then really the last two and a half years has really gone progressively more and more all in on AI. So excited to be here, share what we've learned and experienced and have a chat.

Speaker A: I've been listening to a lot of different companies that I've been following talk about the behind the scenes story of doubling down on AI. One of them is the browser company, Arc Browser. One of my favorite. I love Arc Browser. They're not going to support it in the long term. They've completely rebuilt another browser called dia, uh, which I love and they take a lot of the great things from arc, but they've built it like started over, built from the ground up with AI as the focal point of the design process and the fact that they're like yeah, we can't bolt this on to our existing product and um, very famously like your CPO just talked about. No, this is like a ah, existential threat. We can't like iterate into AI, we can't just put a chat box into the. So we're just going to start over. Can you walk me through those conversations of when you saw the potential of AI and what did that look like from an architectural perspective, like a product thinking perspective where you had to reinvent like you're offering?

Speaker B: Yeah. So for us it started, it really was. So first of all we had an ML group prior to Jet GPT's launch. We had folks who were already deep in at the time were like this doesn't feel like AI. This is ML, right? ML felt like the right word to use back then. And for example like the summer prior to ChatGPT's launch. So was that summer of 2022 one of the guys went up and shared a at a show and tell of like hey, I'm trying to get summarization to work well. And he, he went long and in depth and it was like it just still doesn't work that well. And this was like the leap. And chatgpt Fergal head of AI is like oh wow. Because they had been working with the models. But that that was still opened everyone's eyes as to the power of it, seeing it in that chat framing of like exposing their technology there. So summary, like the summer we're like this just doing something like summarizing a ah, support conversation. Technology was not good enough and literally that opened the door. It immediately was good enough and that's just like a small thing like summarization. So we kind of had well our now AI group folks are like oh yeah wait, this seems big. I think we were the. I think what we did early is the stuff that folks of everyone eventually did is what's like the easy stuff you can do, summarize, reword this sort of stuff that we could put in our inbox. And it was like we thought hey, these are somewhat useful things. We're just going to learn fast and try to see if we can build product faster and really just get out of the gates fast. And so we did that like by January and But really what was interesting was Mario, one of the guys in Fergal's group. So it's basically he's like I think I can control the hallucinations which was the huge risk. And so from early on once he got the sense and essentially this was before it was called Rag Rag. It was a system to build to kind of constrain hey only answer from within this set of content open the door to like the first swing. So we just started like what's value we can get and learn and build and chip and they're like, oh wait, this actually feels like the big prize we can go for. Like we had old school bots where there's a little bit of ML where you're trying to detect some parts of the conversation. You know, there's a little bit of smarts in there, but it's mostly deterministic stuff going on to like threw all that away and restart with that system. I think one thing that's so we kind of. And then from there just building out and that's like uh. I won't go on through the longer story there. So two other thoughts that I'll pull out is I think I feel like it's almost every three months we've like stopped and assessed are we going big enough? Are we going bold enough? And it's been this. So it's not like one big decision. There wasn't one big fork in the road. It was this continual challenging. Are we thinking big enough here? And I think that's really helped us get to the point now. So whether it's hiring, we gotta swing more to hey, this is gonna take time. We gotta build up our AI group with the folks with the right capabilities and just this in a way like this incremental big swings incrementally added on adds up to a really huge swing. But it wasn't like there was one point. This is the huge swing. Here it goes if that makes sense. And so I think that's one thing looking back over the two and a half years, just like continually challenging ourselves. Are we thinking big enough and broad enough? And you know, you guys are we. Are we too gung ho on the opportunity? Uh, is this going to, you know, are we too far ahead trying to get that timing right? And then obviously all the product strategy which part of it in. But I do think that's one part of it. A continuous manual reassessment is part one, part two. Another thing on like this, you know, the ripping up the. By the way, I'm also big fan of the browser. I didn't know I haven't tried the new one by the way. So I'm looking forward to try that, see how uh, that goes. But um, the. What's interesting is even though so we built up our core techers literally from scratch around the LLM around and we still had inherited product and you almost are blind to it. So Fin is our AI agent for customer support and it's been in the wild since like June 2023. So over two years later, we finally shipped kind of what seemed like small UX improvements in the messenger and what these were to not even realize you have it. So in, uh, our FIN product we put all this effort, but the UX that we did initially was button based. And so FIN would give an answer. And this was in the early stage of where people willing to believe and trust this and we list out the sources earlier on and then we say, did FIN answer the question? Then you'd have two buttons that would be like know or talk to the team. And this is anchored into our whole system because it's hard resolutions, how we calculate resolutions and how we charge. Like it's a fundamental UX of part of the system. And it was crazy bad ux. It was the wrong question to ask at the wrong time. The way we frame the question, did this answer question no or talk to the team or sorry, yes. Or talk to the team. You couldn't say no and continue. It was like, uh, no one would ever ask this. It was inherited product. We were blind to. And we've only just shipped improvements to actually make it more natural conversations to more naturally encourage people to talk more to the bot. So it was like a good example of, oh, we built FIN from scratch. But no, we didn't. We still had parts of the system we were anchoring on. And two years later we only realized, oh my God, I can't believe we've still inherited that design and hadn't challenged it. And it's because it's slow, it's deep in this process system and so everyone's reluctant to change it. And then it had. We wanted to change it because it felt like crappy UX and then the huge upside and we wanted to change it and make sure it didn't degrade performance. And it turned out it actually significantly helped resolution because customers were now engaging in a more of back and forth with fin. So we had like, we almost felt like we were competing with one hand tied behind our back with this UX that we just got out the door fast. You know, you got to move fast, get stuff out the door and then you forget what to go back and reassess of. Like, oh, yeah, we didn't think about that. So anyway, uh, that's an example where we didn't even realize we had some inherited product.

Speaker A: I'm so glad I'm talking to the VP of Finn rather than the cpo, because he's, he's a fantastic storyteller. Um, and it's a lot more buttoned up and polished and I think for good reason. But this I want to hear the behind the scenes like the. I want the documentary, I don't want the blockbuster. And this is fantastic because yeah the it really just comes down to the ux. But like human computer interaction paradigms are completely shifting and it's not turn based ivr decision making trees where you have to do all this thought and decision making of like oh well, what happens when this happens? Okay, now do this. My first question is what happened to the ML team? Do they just get absorbed into the AI team? Is ML still a function at uh, Intercom or.

Speaker B: Yeah, that's all absorbed within the AI group. So we call that the AI group. So yeah, massively expanded. The ML folks are all rebranded.

Speaker A: Yeah. Do you still have features where uses more traditional ML or is the push to be like. The reason we use traditional ML is because we didn't have the technology we have today. And it's like why would we not.

Speaker B: We only used ML because of that. But in our recent Insights product that we shipped, uh, one of this was topic categorizations. You got all your support conversations and the thing we've tried to build five years ago and it was just fine. There's improve technology to automatically detect your categories, your topics and your subtopics. It's really the granularity and accuracy of this where the value is. A lot of that was actually ML techers not using LLMs. So we kind of used LLMs on top of the core ML techers. So and I can't actually speak the detail of what's advanced from five years ago on that side to it. And then we use the LLM on top of that to actually uh, decide on some of the categories, name the category, summarize the categories, ext. Make those human usable almost from the core ML side. So yeah, that's like one example where we, you'd expect LLMs did that work, but we actually relied on ML technology

Speaker A: in maybe using AI LLMs as an interface to the traditional ML models.

Speaker B: Probably would, but that's like I think the exception to be clear.

Speaker A: But hey, that still happens with uh, Sana. It's getting really good at reasoning and it's impressive. I got an invite to the Comet browser. My longtime uh, college friend is there as a designer and he gave me an invite and I've been using it, uh, playing around with its in browser agentic flows and it's getting really good. And just reading the reasoning like stream of consciousness from Claude is the model that they're obviously using it. It's getting fantastic. I think starting. You're a really good case study of being at a company for so long and then this massive reinvention of the concept of how you interact with Intercoms products. Coming up with a new interface and a new interaction design style to interface with these new technologies. Um, like what lessons did you have to unlearn coming from a more I'll say button based transition into a very like leading agentic product.

Speaker B: There's kind of like multiple levels of it if we look on how you work. We really quite deliberately ripped up our um, decade of what we've learned from building. We basically said our principles hold true, but everything after that is like we have no more strong opinions on these. So to folks, it was kind of like uh, almost carte blanche for Hotarch, even like our six week cycle. The thing I thought would be like, I don't think this is ever going away. This just feels like the sweet spot. And I remember way back when when Independently Basecamp said they used this six week thing, we both independently came to this. This is just the sweet spot. And they're like, we've thrown even that out the window. The way we work was. It's not that AI working for AI, uh, mandated that. It was just in the spirit of trying to deliberately throw out. It's hard to know which preconceived notions are right or wrong. So throw most of them out and then rebuild them up as we're going. We threw out how teams were structured where all the assumptions are off. You don't have your triad necessarily. You don't have build things through teams. So like our operating model for R and D that we had this mantra of just like follow the work. What's the work we want to do, organize ourselves around the work. And this was all optimizing for speed and focusing on the most important thing. So that's like in the how we work and then like in terms of what you build. I, I think there's multiple things. One is the something that the AI group has pushed like we, we still have, I'll just call it core R and D in the AI group. And we kind of merge together across work streams there. One of the things folks from the AI group, many of whom came from the core R and D but I've um, pushed on is like the trying to up level the rigorous of validating the product you're building. Almost bringing in like a consumer a B testing mindset that you never brought into SaaS type thing particularly we've got this, we're super proud of like our resolution. Right. Finn's resolution. Right. We've got this graph we show anyone who, who will listen of look where this gone from like 25% when we launched, it's now looking at today it's now like 65%. So that's across all our paying customers at average rate. And it's just this beautiful incremental improvement in the product's value that we're delivering in order to get that is the result of like hundreds of a B testing and being really disciplined about ensuring what you're shipping isn't degrading that with those UX improvements that we just wanted to make sure they didn't make things worse because we felt the UX was better which was harder to measure and then it turned out they did. So I think that's been one shift that we're still undergoing is the whole rigor of the testing methodology and like hey, you know, back testing and how we're going, that's like step one. And then if you can do it with a B testing which depends on what you're building, you still can't do that with everything because you know, if it's something that's going right to the end user, you have scale, you can do it. I think there's a, a rigor uh, as you'd expect from you know, the people call themselves AI scientists and researchers. There's a real rigor to most of these. A lot of these folks are basically academics or actually academic folks who are optimized towards building products. Optimize in their heads are like I can't stand academics way too so I need to build product. Right. But they're coming from that background. So I think that's a huge part is like the rigor and we're still trying to. That's still, that's a work in progress.

Speaker A: Yeah. So from like the outside looking in we're seeing designers posting PRs like uh, intercom. Um, I'm going to focus in on okay through it up. We're not necessarily structured in traditional ways like with the triad anymore. How do you like that also just means like the portfolio management aspect of being a VP of product where usually you have okay I got my scrums like uh, the squad kind of sounds like the button based UX like where it's okay, it's a button based squad team topology of like very well scoped experiences that you could divide and conquer as a team. Now that like, you're not doing that. What does the management of the product portfolio and the teams work? Because I know that a lot of PMs I would know would be like, destabilized by the fact that they can't clear ownership for designers. Like, how do you do that?

Speaker B: It is destabilizing. Like, there's a real cost to this. And, um, the cost is like, we know how we operate. You know how you get things done. Everyone has a sense of, I broadly know what I'm be working on the next three months. I know what kind of comes next. We build our own roadmap. That sort of stability of I know what my job is and how I do it and how I'm going to go about doing it. We did remove a big chunk of that and there's a big cost to that. So now the upside to it is we basically said, here's our priority work streams. Assemble the people you need for that work stream. Maybe you need five designers, maybe you need none. Maybe three would be the max.

Speaker A: You're talking to the product lead or the PM in this case.

Speaker B: Uh, well, this is like four R&D across AI group and core R&D. We've got eight main work streams. This is the most important work to get done. We need to staff this work. Let's staff it according to what each project needs. One project might not need anyone from the AI group. One might need to be half from the AI group, depending on that. None of this maps cleanly to teams. And so we optimize for the most important work to get done. She went all the way up to Owen, to our CEO, to say, are you aligned to this centralized roadmap list? Follow that work. And then the group remits were kind of to the continued on if they still had capacity, which they wouldn't know how long they would have it for. So it's really destabilizing. It's hard for folks. And so if you're on a work stream on there, then you kind of know what your focus is and you get this huge upside of focus for people on the work stream. You don't have to sweat the other stuff. You're just focus on this, building the best product the fastest you can. But there's a lot of destabilizing forces, uh, outside of it. That is hard for folks. It's hard for everyone. So there's a real cost to this. And I definitely don't think we figured it out. We haven't figured out what the new operating model is but we just evaluate and we're like, this seems to work, it's sufficiently good. We only fix things that clearly need to be fixed. So you can just focus on what matters. That kind of ruthless focus, what I described to you doesn't really make coherent sense and I can't really articulate how it all works. But that's all right. If everything is generally not, you know, that's good enough to make sure the most important work is getting done. That's been the spirit of how we've worked the last eight or nine months and it's been effective, but it has cost. We'll have to keep reevaluating, keep readjusting. But yeah, it is a big shift. It's a big shift for folks.

Speaker A: I'll tie that into the other item that you brought up, which is that it's increased rigor. It's also just the irony of like the democratization of software is the fact that you have to be more technical to take advantage of this democratization. If like the terminal is like natural language and you just need to talk to it, you need to know how to articulate, how to transform data, how to navigate the code and everything like that. But your operating model just sounds like more human, like why did we have Agile in the first place? One, software is really expensive to produce. Like you need to have these rituals in place to protect your expensive engineering resources. And then the whole point of the product operating model, I mean like Marty Kagan and stuff, is the fact that there's a lot of coordination in project management. Most of the project management software is built around a full time job of work or documenting everything. What I've been noticing in my personal practice is that it's not that hard to make glue documentation like Dasses, like decision making documents, summaries of meetings anymore. If anything it's increased the standard of like rigor of thinking. And if you're being very rigorous in your thinking, you're going to be questioning the team topology like every two weeks because you're just like ah, ah, do we even need to have a dedicated team for this? It just doesn't seem like it justifies consuming this many resources. We could retask things. I've done this, a similar model like this in my current gig where it's like centralized roadmap. We have to like basically pitch every time, like for engineering resources every time. Pre AI it was super painful because you have to do all these traditional project management documentation things. But in the world of AI where it's not that hard to have the meeting transcript, create a summary of the meeting, post it in the wiki for future prosperity, you can move on with your day and start making like better decisions. And like it was so hard that it was like, the centralized model's not working. We gotta figure out how to get back into product teams, trios and clear ownership. But now hearing this, it's like, oh, but the technology moves so fast that all those constraints that the whole trio model was built around to solve the complete problem brief has changed on project management and doing product work. And I'm um, now kind of being like, ah, um, maybe we're onto something. It's just the technology wasn't there to stay on top of the work enough.

Speaker B: Yeah, it's interesting because uh, like, and you're drawing a direct line between AI enabling this and I have in my head it's kind of like a dotted line. But it's harder to like fully justify the changes we've made. It's almost like the macro industry trends rather than literally AI is enabling this. I just feel like, you know, we're at this, you know, we're at the start of this S curve. Not right at the start, but just on like day three of it or actually in year three of it really. And it's the time of builders and it's the. This is like, this is so I think the, the irony is, is it's like this AI brings a democratization of technology, but requires for at least for those of us building it, an uh, up leveling of our technical skills. I think for PMs, that's definitely one of the things is like you need to be more technical now to understand the product. You didn't really need to know the actual mechanics of how it was delivering the product you were trying to build. With AI, that's not the case. You need to understand how this is working, the architecture of the system, what the constraints are, what the risks are. And with the huge unknown of uh, will the have it's like, but will it actually be good? The whole uncertainty is it, yeah, it all sounds good, but will it actually be good? And how are we going to actually validate that good in terms of the. Because you don't know what the output will be or how valuable it will be to your customer. So like there's the uncertainty that it brings is like there's a technology uplift that's required and you're talking about, you know, if you, if English is the coding language or natural language is the coding language. Which seems like therefore everything's within reach, right? And you know, people can make marketing websites and do it without any technology. So for sure part of that is true. But then the flip side is also like what we found in building tasks. Basically like the most agentic part of this, how do you get someone to uh, an agent to do a refund process? Those sort of things that are always deceptively complex. It's the hurdle is not a technical one. You need to be able to plug into data. So there's literally some technical hurdle that you might actually require code to get to that small part there and then as you're saying, maybe to transform the data. The working with data part is the most technical part, but the other part is the block. There's not a barrier of code, but there's a barrier of working with technology and a barrier of thinking logically and extracting rule sets and testing rule sets and iterating this and who's naturally good at this is, you know, engineers. This is what engineers do. This is how they think already. So this has been the lesson for us. And like it's natural language and then you can pull WYSIWYG stuff, but it's still not easy to do or it's harder to do to effectively prompt engineering is really what we're talking about there. So I think there's like one of the ironies that I've kind of grappled my head around is God, it seems like it should be available to all, but it's still not. But the barrier is almost a manner of thinking, an approach of thinking. I remember I had, we were talking about like this small thing of like in our Insights product and just how we framed like summarized by a, uh, conversation got a certain CX score and we're just like, hey, how should we say. But wait, what if it is an old school but just boring product? Shit, if you're just trying to be consistent and use your language correctly. And we had as a engineer, uh, and me and I think a PM talking through details of it. Oh, wait, what should we do here? And, and then you have the chat and then you see the engineer comes out of it. It's like, okay, wait, before we go. So to be clear, and he basically articulated rule one, when this happens, then we're going to do this. Rule two, when this. I was like, oh, this is how engineers work. They're like, I need. You can't leave it up to the program to do it. You still have to program. Exactly. You just don't use the programming language of before. That's exactly the right thing. You still have to program to get this to work.

Speaker A: Yeah. So if you don't know how to say a natural language, how like the math should work and like some business logic, then you're leaving it up to the LLM, which is leaving up chance, which is increasing risk and the output being bad. I think like one way that I've discussed it with other guests on the podcast was um, second language, like language learning. So like I speak Spanish. Right. Um, and one of my favorite things to do is go to Spain or Mexico, cover this blonde mop like that I have with a ball cap. Um, my, my beard is a lot darker than my hair. It's weird, it looks like I bleach my hair sometimes. But I always try to opt for the Spanish speaking country because it's just so fun to travel there. Because you know how to interface with natural language, articulate ideas and say jokes and you understand like the ditos.

Speaker B: How do you say that?

Speaker A: Yeah, there's sayings like there, um, idioms. Exactly. Like you understand the idioms of the language. So like you're able to connect better. You actually understand like the emotional resonance, like tone, timber, you know, volume. And there's a reason why like in sales it's a lot of the training has to do with like how you use your voice like it's an instrument and like that, that you still have to. And that's sort of the reason why some people are more persuasive than others. That's like a engineer is going to be way more persuasive programming in natural language than a non technical pm. And I, uh, yeah, I totally resonate with that. And do you know how to ask for what you want? Because the company's not paying you to leave it up to chance.

Speaker B: Exactly.

Speaker A: From Jackie bt Like you're, you're master of the content. You're just having it organized it in a way that like makes sense.

Speaker B: Yeah. If you're building product for other people to use, you need to reduce the risk of chance. Whereas if you're just building it for yourself, you could wait till you get what you want. You can't do that when you're building product, so. Yeah, exactly. I think that's been really interesting learning that side to it. But I definitely feel like, hey, we're in a time of builders. You, uh, need to be more technical. Like I've had several chats about that now. I don't know exactly what it means yet, but it's surely a thing and it's the same as like hey now because there's for designers shipping code. That's a idea that's been around for a while. Right. But now it's now easier to do it, but really is within reach and obviously there's a bunch of things that now are within reach that previously weren't. So what was your phrase? I can't remember the phrase you used earlier.

Speaker A: You have to understand the idioms.

Speaker B: You had a nice phrase there.

Speaker A: Yeah, it sounds like it. Ah, uh, well just to add on that like it's really clicking for me because like designers always been thinking in like CSS and like HTML. It's just not. Hasn't never been efficient. I learned, I come from a design background. I learned how to do react and I ah, learned enough to know I'm never gonna be able to spend enough time on this to get good at it and fast enough sense I'd rather get really good at the designs. Uh, but it like that transition makes a lot more sense. The one where I'm like a little skeptical is PMs doing design because it's okay, you don't know like vocab. You don't have any vocabulary around type or composition or like hierarchy and stuff like that so that you don't know what to ask for. And so whatever the output is, you don't know if it's good or not, but you're hoping that it is. And that would mean that like PMs would have to level up like their design vocabulary as well. Oh, increase the padding by this much or something like that.

Speaker B: I do think the often used phrase of product taste has come to the forefront in the age of AI. Um, but I do believe PMs we used to come from a place where all the PMs we hired kind of had a design background. I had an X background if you want to count that.

Speaker A: I mean your, your CPO is a ux.

Speaker B: Uh, yeah. And Des and Owen. So we're unusual having a design heavy exec team. I do think PMs should have a good taste for like decent UX, but also agree with you. So I think the delta in my head of PM's doing design is actually it's really all about prototyping and it's like designing, building to think, think to build, build to think. You write to think and then you need to build to think and basically prototype to think. So I think that's the part that's valuable for PMs to mess around with an idea and not have to wait for someone else to build a Figma prototype or something like that, or go through it and just get the idea out there to mess around with it. And it's basically like speeding up the conversation, speeding up the getting to confidence or not. Is this a good idea? Is this worth doing? Rather than the PM actually goes all the way to like, hey, here's like, the final design, which to me is like, yeah. Oh, yeah, that definitely doesn't. Well, uh, maybe there's a world where it's like, small changes that you're making. You can use your design system in the same way an engineer could do. I'm sure that that's coherent, but. Yeah, no, broadly with you, it's not PMs doing design, but PMs. I think I really feel like it's that build the thing part, the prototyping.

Speaker A: Yeah. It's like an intellectual wireframe or like a literal wireframe, but so cheap to, like, make a tangible thing with interactive elements that, uh, you could share internally or whip it up in front of a customer. And then the designers just cool. Like, we validated an actual problem. The customer's eyes lit up when we showed this, like, PM prototype. Like, okay, where can we take this next? Really, like, make it ready for production. And I would definitely encourage that to, like, any PMs. But I think it's like the PMs being like, okay, how do I hook up Supabase and stuff like that? And I could just bypass the designer. I don't know. It's just. It's like me saying I could bypass a backend engineer. I don't have the vocabulary.

Speaker B: Yeah.

Speaker A: Uh, to, like, program business logic.

Speaker B: Like, we're all still so new for everyone figuring out what's the opportunity. And we don't know what makes us, what speeds us up and what slows us down. And that's part of what's hard with this is, am I gonna waste two hours trying to do something here? There's no default best practice for anything. At the moment. We're operating in a wild west. The macro industry landscape and the micro, how you work, which is fun and exciting, but also like, she. It's, you know, it's a, uh, hard

Speaker A: being a people manager because people like, what's my job? And you're like, well, you know, just use your judgment. And it's like, what. I don't know, like, what I'm focusing on. And that really, like, sifts out what kind of talent you need for very bleeding Edge technology is self managing. A little bit of entrepreneurialism. Like those people are going to do very well, which historically they really haven't. In traditional big tech company scenarios, the people that are entrepreneurial don't like being put in a box like me. It's been very exciting because like everyone's so destabilized that no one's going to like hold me to a standard where I'm just like hey, I'm just going prototype. Yeah, I know I'm a pm but I got, I have a design background. Like I know I could take this to like a pretty decent like valid like concept and I don't know, it's fun, scary for a lot of people. Fun for me so far now like talk that's like how to work, how to price is like another thing I'm very interested in. One thing we talked about before the interview is like usage based pricing, results based pricing, traditional SaaS pricing and like how destabilizing that is to like how do we monetize this idea? How do we go to market?

Speaker B: I think we're pretty early with the whole outcome based pricing. Outcome based pricing means we charge uh, rather than charging based on usage which maps to like RLM M costs and then you know, we put a cost on top for the software that we're building and that's usage based pricing every, every time, every question that Fin answers. For example. Instead we only charge when Fin answers a question, resolves it and doesn't need to go to your team. So that's outcome based pricing. So we charge per resolution. There's a huge amount of uncertainty when we initially took the swing because you don't know, it depends on how successful you know. There's just a huge amount of variability in it. Dara, our CTO actually I think ultimately made the call on it of like, no, we really believe this is the future and we want to try it because it's pricing aligned to value. And you know Intercoms has a pretty bumpy history with pricing. We've got a lot of things wrong with pricing and we finally gotten pricing right prior to this with just our regular help desk. And then this was trying to go in that vein of what do we think the right pricing is. So anyway, one thing that's been interesting is when you have outcome based pricing, the most beautiful things and this was very deliberate but is like the incentives are aligned. So what your customers want out of the product is what your product team wants out of the product is what your finance team wants out of the product. Right. Everyone is happy when resolutions go up. This is very unusual in SaaS. Products say if it's like seats based pricing where people adding more seats doesn't necessarily mean much, right? Well, it doesn't mean they're happier as customers. It doesn't mean they're getting any better outcome. The seat space means, okay, your finance team is happier, your sales team is happier because they're paying more money. But that doesn't mean they're getting more value out of the product or they're happier there. So the product people are the customer or not. So outcome based pricing takes the circles and like the puts concentric circles over these three. So it's and one of the um, unexpected side. So that was quite deliberate to like align incentives. But what was unexpected was like your metrics, suddenly you know the whole thing about outcome metrics, which has always been this long hard slog of like what are the right outcome metrics? Do you really optimize around it? And there's still a pro. It's always a proxy. You're measuring a proxy and is this the right thing? And suddenly so much of that conversation, like we had built a customer charity framework of trying to identify where the attributes in the product, how much do they need to use of these parts. And if you use, you know, three of these five parts sufficiently, well then you're a mature customer. And that's our success metric for trying to get this percent of our customers to maturity based on all these product usage proxies which we think equate, uh, to them getting value, which we hope equate to them, you know, doesn't even necessarily equate to them being satisfied with the product. Whereas so much of that washes away with outcome based pricing and where you've just got like for us, uh, the customer, you know, to a health check of the customer is what is their involvement rate? Which is how many of their inbound conversations are they letting Fin try to answer what is Fin's resolution rate? And then what is the score? What's the quality of it, what's the performance of it? And are they applying it broadly? So with three metrics, which are the three metrics we show to our customers on their performance page. And one of those metrics is how we make money. So like you've just got phenomenal alignment of incentives that really simplifies a lot of product. Now you still have proxies. Do we know if what we built helped actually move the resolution? Right. So it's not like all the complexity goes away. But just from that core alignment of those three things, the customer, the business and the product. It's, that's been a like a wonderful thing that we've gotten from outcome based pricing that we never had before.

Speaker A: Yeah. And in outcome based pricing is not necessarily like a new concept. It's just really hard to do a SaaS because like how do you prove it? Right. It reminds me of a tire company where they put pricing based off miles. Right. It's just like how many miles these tires, that's what you pay for. And that uh, like increased conversions very highly because it's like cool. Like this will get you this many miles if you pay this much. Right. If you buy a more expensive tire, it gets you more miles. And outbound sales companies, it's like hey, you only pay when we like get you a lead or something like that. Are you saying that like they pay $0? It's only when like Fin says there's no like base cost to help maintain the server.

Speaker B: So if you get a resolution it's M $0.99. If you get a thousand of them, it's a thousand bucks, you know, 999M bucks. So it is literally that straightforward. Here's like how we priced before Resolution Bot was our old techers, bot based deterministic stuff with some ML in there and we charge based on seats. And so what's the result is that we had a lot of customers paying a lot of money who had not properly set it up and were getting like no value was built in the contract and then they're done and they didn't go back to it. The payment was divorced from the outcome and we had other customers who had put huge effort in and like we're getting way more return on it. Like getting. They were paying not nearly enough. So we had to like both ends of the spectrum of people who are paying for product they weren't using and getting no value and customers who were, we were not monetized anywhere near the value they were getting out of it. There's just so the seat based approach which is, you know, standard set, it just like was not fit for purpose. But I don't think we, I can't remember the pricing at the time, how much we consider to your point, Outcome Baby's pricing isn't brand new, but it's just anyway. So uh, it's been a huge unlock just in that building product and knowing that your customers are getting value. If a Customer's got a 90% involvement rate, a 60% resolution rate and they're seeing scores in the 80s%. Like I'm pretty sure they're going to be happy with fin, you know, because that's amazing value they're getting out of that. So that's been wonderful.

Speaker A: Another thing that we talked about before the interview is a really great segue. It's not a new concept to have outcome based pricing, but usually those companies have four deployed engineers to stack the deck in their favor. That means like custom development service based businesses where it's like, hey, I need to like make sure that this is implemented and integrated into the client's system, basically guarantee that we hit those outcome numbers because if we just say, okay, cool, just open up the web app, upload your customer data, link all your phone chats into FIN and then cross your fingers like FIN is going to resolve things like that. That's highly dependent on like the customer, how proactive they are to feed the right data into FIN that impacts resolution rate. M what has I guess the implementation process looked like?

Speaker B: Yeah, so we found we have leaned in the whole concept of this Ford Deployed Engineer, which I think came from Palantir or maybe they just publicized it going back years, which felt like to us a different operating model. We just call it R and D Services. It's kind of like also the pro serve. Right. But this is R and D folks, uh, leaning heavily into this and we have found that this is really important and it's exactly for what you said. And I think the delta here is it's way easier to realize when people aren't getting value and it's like, hey, they need help here. Whereas previously that would have been more opaque and so you could have had someone coming in and it's just be harder to detect that they're not close to getting that value. Whereas here, just wait, if someone's at a, uh, 5% involvement rate and a 30% resolution rate, well, something's going wrong here. So we got way more visibility of this going wrong. And then people are making the wrong conclusion. Oh, this isn't actually that great of a product. And it's like, no, there's huge value. You just need more help at getting the setup right, getting your content right. Usually a lot of it is in the content, but sometimes it was how they had set up and configured and they didn't quite realize that they had not given FIN enough of a chance. There's a bunch of reasons and a lot of it is also internal changes they need to go through. The R and D services has been a critical new muscle for us to build. It seems like I've seen this with multiple different AI companies. It's like it's almost to that. God, this is ironic. You would have expected, if you had asked me two years ago to predict what would happen, I would say, oh yeah, plg, this is going to be AI will just reinforce the power of PLG and that will be amplified. And that is not what's happened. AI products at least now maybe this will just come in a couple of years, maybe in five years this will change. But right now I think most AI first products generally have a good bit of handholding at the outset. And it's both like success using the product. There's also usually transformation behind the scenes. Like transformation is such a terrible word to use, right. It's like, oh my God, it's like digital transformation, the worst phrase ever. It's like someone's paying consultants God knows how much money for some God awful project. Oh my God, get away from the work. But unfortunately it's like a transformation I think is the right word. It's painful. It is to say but it's like a big shift for companies and they need support on that side of it as well. So yeah, the that concept and it's like. And you're like hey, do we need PMs for this? Do you need PMs? And it's like turns out PMs are good at this and they need engineers as well. But it's just like it naturally maps well. So yeah, we're like maybe it was just for the first three months but it's like no, this seems pretty durable the PMs. And I think a bunch of other people have come to the same conclusion that whole space. And it's also a space that's very fun for PMs to work because you're so close with the customer. And uh, one other thing that related to this that I feel like has changed. You realize when we're building product and particularly when your product's bigger and broader and so you're always focused on a small part. So if you know outside of the first startup land where you've got the whole view but everyone building has buy necessarily has. Has blinkers on for the product area which is probably quite narrow that they're focused in on. Right. And they're like I just want to build this. And when I talk to, I talk to customers all the time but I'm really like ignoring not everything except for this. Uh-huh. Uh, I'm Just thinking of this. And when you do R and D services, you have like, there's no blinkers. You're like, I got to get you to success. And I have the whole gamut of their experience, what they care about, what matters. And you feel the pain of your own product in a way that never happens otherwise. This is when everyone's gone and done services. Like, oh God, that part I can't believe. You know, like, they feel the pain of the product in a way that you don't do when you're in your standard product building. So I think like the, it's just like cast this huge light on your product that you. It's so hard to see when you're in a necessarily siloed thinking of, I'm, um, trying to build a feature, improve something X. So it's been almost like a slap in the face for kind of exposing how narrow you can be building product.

Speaker A: Yeah. And then that blows up the whole like product trio thinking. Because then it's, oh, who talks to customers, which customers, which segments? And then that customer interview is totally wasted because it's through the lens of my feature.

Speaker B: Exactly.

Speaker A: Even though the customer has said several things that is holistic across the whole thesis of what the business is and how we're servicing them. So it's cool. I'm going to come up with these specific problem features to solve 5% of this problem when so many things have to orchestrate together to even succeed in the first place.

Speaker B: Yeah.

Speaker A: Very excited. So essentially you're just like, okay, yeah, we still have PMs. It's just letting them loose like on a broader scope.

Speaker B: And so we have folks who are clearly working there. We need more deployed engineers effectively. So we have a group and that's their focus and job. And then we move some people in and out. So it's not fully fluid, but.

Speaker A: Well, that's an important detail. Right. Because the solution can't be like a custom solution. That's like the downside of four deployed engineers historically is that, uh, it's almost too custom and then it makes a Franken product. And so like the product thinking that's go into it is, okay, how do we take this custom implementation and then learn from it and then upgrade the product to scale this onboarding workflow. Right. Repeatable stuff with.

Speaker B: But I think another thing that's challenged is like the in product, it's like the human tendency with everything is to slow down and add more bureaucracy. Right. So the product that is like, oh, we'll take that feedback on and then we'll aggregate it up and then we'll figure out some genius way of coming up with our idea of what the right priorities based on the all. Ah, the feedback that we're hearing is like that process is slow. And ironically you end up removing judgment and just trying to wait up. Hey, here's all the signals and you know, put some sort of, hey, just how many times was asked what's the value of those customers or what's the blah, blah, blah. And pushing more to like the services thing does also unlock of like, hey, does it make sense what they're asking for? Just build that thing and then we'll figure out how to productize it. We actually haven't built too much of just stuff behind feature flags. But, uh, we've tried to come. I think this is. Does what the customer asks for make sense? Is it small? Just build it in. Don't sweat it, just do it. Like adapt, adapt faster. You know, like trying to get a big R and D Org to act more like a startup who just wouldn't sweat these things. And this makes sense what the customer is asking for. So just build it, you know, and then there's bigger ones that you do have to sweat and you got to think about and like, whoa, whoa, whoa. Okay, that, that does require more deliberation. But I think R and D teams have almost like hindered themselves of not just like making quick, fast, small product decisions based on one customer or a couple customers heard it. It's like a couple customers heard it. What they're saying makes sense. Okay, let's just go build this small thing. Let's add this thing. Just do it. Just do it.

Speaker A: Yeah. A lot of product teams have frameworked themselves into a corner where they can't do anything unless they go through the steps in the framework. And, uh, sometimes things just make sense. The thing that the sales guy said, we need to close this deal makes sense. We shouldn't say no to sales every time they come up with a good idea or even if it's a bad idea, dig down deeper. Okay, why are you asking that? And then it ends up being a good idea. Is there anything that you feel like, Brian, that hasn't been said, that shouldn't be said? We covered a lot of material, but do you feel like there's any, like, loose threads that you want to close before you sign off?

Speaker B: And then the world we're in now, like, all the uncertainty, it's at all the levels, right? And it's like AI is this convergent Force that just is knocking down our notions of how you build product, what R and D teams are, as well as what your product is in the first place. Right? And so it's kind of this amazing, like, I don't know if juggernaut is the right word, but it's like this blob that just spreads in all directions and knocks down all your preconceptions, preconceived ideas, and that's even a strategy, you know, your strategy idea. And I just knocks walls down. It just knocks walls down. And so it's fun and it's exciting and it's uncertain. But I think that's, uh, rare to get to work in a time like that where so much how you build, you know, so much of what we said is how you price, how you build products, what your product is, what your role is. All. So many things are, uh, up in the air. We all get to figure it out now and define it now. It's uncertain and it, uh, uh, certainly causes anxiety for me, figuring out what's my role, what am I doing here. But it also means we get to write the next rule set or the next best way of working. So that's like the massive upside here. All the winners are yet to be figured out. The next wave of here's how you build an AI, it's all to be played for. So it's fun times.

Speaker A: I've always been very jealous of people that start, like, we're in the prime of their career during the Internet, because I'm like, man, it's so disruptive. Like, there weren't any rules and now it's cool. Here's our app.

Speaker B: Uh, exactly.

Speaker A: And I'm like, yes, I get my Internet, like, macro trend that I get to cut my teeth in my career.

Speaker B: Yeah.

Speaker A: Brian, how can people reach you? And, um, God, it's weird.

Speaker B: You know, I'm not on. I'm. I'm on Twitter, but I don't write anything on Twitter anymore because, uh, almost nothing. So I guess LinkedIn is where we actually share most of our stuff. It's weird to say that out loud, but it's actually true. So God knows what my LinkedIn handle is, but Brian Donahue intercom. You find it. And yeah, that's what we share. Uh, by the way, we actually share a lot more building. You know, this is also what's fun. People share what they're building. We share way earlier engineers talking about stuff. So there's a lot more transparency as well across folks. So that's also fun. Just learning faster from everyone. So, yeah, love it.

Speaker A: Well, Brian, thank you for coming on the show. You have a good one.

Speaker B: All right, Take care. Thanks.

Speaker A: Hey, listener, thanks again for listening to the Way of Product as we power down today's session. Um, on the fusion of creativity and technology, let's solidify the energy you've gotten from our dialogue today. If you've gleaned insights or strategies that could turbocharge your projects, I have a small but powerful ask. Swing by the platform where you're hearing this and leave a review. Think of it as your digital high five. To us, it's a fuel that propels this content to more creative technologists like you, amplifying the impact of the shared knowledge. So until our next interview, keep pioneering and pushing the envelope. Thanks again for lending your ears and your imagination to our community. Catch you on the next download.