Lessons from Building LiftLab Media Mix Modeling and Experimentation Platform | John Wallace

Speaker A: The inputs that we want to provide are things like marginal roas. You could just derive from that roas, uh, the profitability. I like to say that LiftLab does not own the P and L. And what we've observed is across a lot of marketers that uh, quite often you may start the fiscal period out focused on profitability, but everyone throws that out the window if you're towards the end of the period and you haven't hit the growth number. And so the ability to tool teams up to walk into a conference room or boardroom to say, I can deliver X if you give me Y, I can deliver Z more if you give me Y. That is what we're excited about helping marketers to do.

Speaker B: John, welcome to the podcast.

Speaker A: Hey Paul, thanks for having me. You ready to talk about mix models? Because I wake up every day and

Speaker B: think about them because it's something that is completely unrelated to media mix modeling and Lift Lab, but I think you can maybe find a way to tie it in. When I was looking at your LinkedIn profile doing research for this podcast, I was excited to see that you took a two year sabbatical, mini retirement break from work before starting Lift Lab. And I wonder if you could just share a little bit about what you did with, with that time, how you spent it and how that ultimately led to building Lift Lab.

Speaker A: Thanks for the curveball. I could the very first words out of my mouth to talk about that time is that I was a stay at home dad. And so I spent a couple of years, you know, with not babies, they're you know, teenage sons, really spending a lot of time with them. Uh, we set up some projects that we did. We actually restored a 1952 Ford pickup truck from my grandfather's farm. So that was a total nostalgia, a bonding kind of moment. And in my case it was on the heels of a long run as an entrepreneur. And those entrepreneur days, my former company was called Data Song. You know, really added up in terms of my presence, right my, even my, you know, my physical shape. I did spend a lot of time exercising, working on projects with my son. Uh, we're going to talk about remodeling during this call. And I remodeled a home. So it was, these were all, you know, ways that I wanted to spend some time and a reset. I did pursue some non marketing data science passion projects. I worked a little bit on, uh, trying to accumulate data sets of agricultural yields. See, could we find ways to increase yields in the field? Could we feed more people with the same resources? That was a Passion project. I did a little other work that some of my network knows about. I hit people up when I was playing with could we train better models for a taste recommender. So most recommendation engines are about say your music tastes right? And that problem's pretty elegantly solved by the algorithms and Spotify and Apple Music. And this was about the taste for your mouth. So Spotify for the mouth, not the ears. Uh, and we were using data sets on literally the chemistry and gas chromatography, you know, of what's going in your mouth and deciding would that help us predict what else you would like to eat from a menu and. But this all led me back to what I know really well are the trials and tribulations of getting the most out of an ad budget. The data sets that were going into the multi touch Attribution that was part of a data song and the company that I sold those data sets were essentially disintegrating slowly over time. And I felt like, what if we were to kind of just clean the decks and think about this same problem of allocating paid media budgets efficiently using a completely new data set. And that was the nexus for Lift Lab.

Speaker B: Uh, that's an awesome story and it sounds like you were experimenting, tinkering with a number of intellectual passion projects to kind of feed that and then committing to one with Lift Lab. What was that thought process like? And the reason why I ask is you go from running a startup, you know, working all the time, intense focus on one thing to focusing on a number of different things and then I'm assuming you did well financially, you're in a good position. Why go all in into Lift Lab right? Where you're like, okay, now I have to commit this, you know, you've done it before. I, this is not something I commit to for three months, six months. I'm starting a new company, big commitment.

Speaker A: It's a personal decision. One thing that you, uh, that when you have kind of pursued your career at building companies is that there's a label that gets placed on you and that is that you're not hirable. So you know, how well would a person who's forcing themselves to make a whole certain type of risk reward decision on a daily basis sit inside of a large organization, right? So part of that's a little bit self fulfilling that you know, that the decisions that I've made that kind of, you know, help shape where I thrive and you know, where I succeed kind of beget where some of my next decisions will come from. Uh, so I did A little bit of interviewing in corporate America, and I kind of came to the similar conclusion that I'm sharing that maybe I'm not that hireable. As far as the decision to work again on paid media allocation, uh, a lot of that came from the fact that some of my former team had already kind of started on the problem, and I was advising them from the sideline. And the more that I looked at what they were doing and the more that we started ideate, the more excited I got about the opportunity. And it didn't take long to throw my hat back in the ring.

Speaker B: That's awesome. I have been working with and using Lift Flap for almost a year now and went through a pretty comprehensive evaluation process looking at a number of different vendors. One of the things that sold me on Lift Lab was the hybrid approach of media mix modeling and experimentation. Can you speak to why that was such an important part of the way that you communicated the value at Lift Lab? And I asked that also, interestingly, because I think that some other companies are starting to mimic that approach.

Speaker A: Well, thanks for recognizing that. You know, I think what we like to say is, uh, imitation is the ultimate form of flattery. And when we went this direction, there was a lot of posturing and defensiveness in, you know, in the vendor world of, you know, why, if you came from the, you know, the, uh, experiments background, you try to shoot down the mixed model as not necessary. And if you came from the mixed model background, you'd try to shoot down the experiments as not necessary or problematic or not, you know, end all, be all. And I think that the vision that we had over five years ago to combine these two analytic methods in a very fundamentally sound way has come back now to become industry standard. Uh, everyone has fallen on their sword and kind of, you know, mimicked or copycatted some of the innovation. And I think it's the form of innovation that welcomed that. And, you know, what we'd like to think is that with that many years of Head Start, we've been able to look at what works well and what doesn't. Uh, and we really do have what we call our. It really is a trust engine. This, these decisions that marketers are making with models require an awful lot of trust. And, uh, we can. If you want to take the questions in this direction, I can talk a little bit about the unified econometric framework that does this. Right. What I find is frustrating sometimes for, uh, marketers or buyers. I can point at lots of medium posts about how you should triangulate from the, you know, the experiments and the, either the multi touch attribution or the last click attribution or what you're seeing in your mix model. And no one gives a recipe for how to do that or if they do, it's kind of spreadsheet math or something like that. And it turns out they're not sharing that because it's a hard problem and there's quite a bit of IP wrapped up in that. And uh, that's the part that I feel like we really do have nailed. And you know, everyone will aspire and look, there's a lot of innovators out there. You know, given enough time, I'm sure they can put something together that's solid.

Speaker B: Yeah. I want to get into how we've started to operationalize media mix modeling in conjunction with experimentation. But I also want to hit on another important, I think, thesis or kind of perspective that you have that is a little bit different than other folks. When you are a software company, when you are an agency, you're trying to get clients, it's very easy to go into. We, uh, will personalize for you. We will make an approach that works for your business. You want to run into this KPI? Sure, we can do that. We'll make it work. Just sign on the, on the dotted line. Lift Lab has been very specific about 1 KPI is the primary one to be looking at and that is marginal roas. And that was something that sold me as well after talking to a number of vendors. Just the idea fundamentally that the most important thing is to think about maximizing that next dollar in your mix. I wonder if you could speak a little bit about how you came to that kind of perspective.

Speaker A: You know, it really, I'd love to say that this is just something I was walking, you know, down the block, but my nature was to find as many brilliant, smart people as I could. If we're going to take another, you know, swing at a problem that we've that we know pretty intimately. And this advice came from one of our advisors, it was Dr. Dominic Hanson from the Business School Anderson School at UCLA. And we had a number of conversations about, you know, what, what, how should we rally, what would be our North Star, you know, our mantra, if you will, and you know, the literature covers this pretty clearly. And. But what. Where we innovated and where he got excited about Lift Lab as, you know, as a platform or a nascent platform at the time was how do we bake that into our experiments? How do we rethink experimentation for the purposes of paid media and that I hadn't seen happen to this date. Actually I've seen that most people have taken off the shelf experiment designs and applied them to paid media. It's pretty convenient. It's like, okay, well I can use an a B design and A is going to be, get it getting the ads and B is going to be go dark and I'm going to go, you know, calculate into the lift and that type of thing. And those tests are valid science. I'm not saying that where I like to redirect the thinking on this is, but the question that I want to answer is how saturated is the media? And to your question, the KPI to do that is marginal roas and I'm not getting marginal roas out of an a B test. So, so that, that really was the first grownup moment was to say, look, brands can run a B test for free in meta. Uh, right. They were called conversions lift tests. Right. They're not used as frequently now because of changes in mobile tracking and things like that. But that was in the, you know, that was our competitor, if you will, was free. So we felt like we needed to design an experiment approach that was made for paid media, that thought solely about paid media. And that turns out has paid off handsomely.

Speaker B: I want to uh, challenge you a little bit on it because one of the things that we've struggled with, and I think many businesses struggle with, is that a change in spend, to equate that to a, uh, change in revenue, you need to make predictions around what that customer will be worth over the long time. And what we've had to do is incorporate predicted LTV models for our different product categories in order to back into a, uh, marginal roas which inherently change and adjust over time. So part of the reason that we're doing a remodel, for example, is, is to update our predicted lifetime value models to be more accurate, to more accurately reflect. This person got a credit card, we think that they're worth X over the next two years. And I wonder when you're making a model on top of a model, if there are challenges around that and how you can kind of think about that. You know, what we're looking at is, hey, this marginal roas says it's five, but it might be more like seven because our PLTV model doesn't have these new assumptions baked into it. Right?

Speaker A: Yeah, yeah, no, well, by the way, just as a data scientist, anytime we have a model with a model, right. You compound the errors. It is the world we Live in. And then, yeah, the in businesses that are. And to me, I think a lot about the two. I actually divide the marketing problem into two types of businesses that marketing is trying to support some of these businesses M and there might be industry names that are better than the labels I give here. So apologies, but I think of them as transactional marketers or lifetime value marketers. And absolutely, uh, they actually have quite different economics inside of their business in theory, for, uh, a lifetime value market. I don't think I'm stating anything controversial. Something you don't know. You're modeling those two events, acquisition and churn. And. And then you may, you know, done a lot of churn modeling in my career and done it with what are called survival models. Actually in your case, you could get multiple products. So you have something even more complex, which is called an intensity function where you're trying to predict the coming and going of products. It just lets your mind kind of bend a little bit when you think about it versus the transactional marketers where we just never know if they're going to come back or not. And you really are. You're modeling the first transaction and the Lift Lab framework separately from the subsequent transactions because the incrementality is not equal between them. Just a parking lot. Maybe we'll come back to it for the transactional marketers. Look, m M the reason I distinguish between them is I see marketers conflate them quite often. And so it's. I've seen this. Sorry to, uh, simplify, but someone worked at a kind of subscription business and they walked around talking about CAC and they walked around talking about ltv and they bring that over to a transactional marketer when they get a new job. Right. Or, you know, and then they use those metrics. But those metrics actually can hurt a transactional marketer. They if you're really saying, well, you know, it's really hard for a transactional marketer. Quite a lot of them have a large one and done problem. All right, so to say my CAC is $80. And then I'm just going to kind of say, and then on average I'm going to get, you know, whatever that is, $200 over a lifetime. What they're missing in that assumption is it's going to require paid media to get them from 80 to 200. Right. There's no reason I have to buy another sweater from you ever, because I can buy it from almost an infinite number of sources. And so paid media is going to help that over in Your case where you are more of a lifetime marketer, it is appropriate to calculate your predicted lifetime value. And now let's bring this all the way back to your question. You can by the way, change your predicted lifetime value, at least if it's mildly and not say I've changed the consumer response functions that are part of my model. So I would kind of say specific to your question and in the weeds, but maybe there's someone listening that gets, you know, into this level of weeds as well. Refreshing your revenue data and rescoring the data is kind of to me an independent function of I need to re actually revisit the consumer response functions. So maybe it's not quite as much overhead there to at the first blush answer to what you're saying wanting to recognize is that if you start to land markedly different consumer behaviors that will have an impact on the consumer response auctions. I think that's kind of the nature of your question. And then, uh, we have a fairly active and aggressive uh, cadence of remodeling that some of our other remodeling that happens is automated. Ah, the ones where we're looking at consumer response functions is pretty frequent. And uh, there's in a business like yours, we look at the business case of doing it even more frequently. I'll be excited to see what you see an actual change here in the coming days and let's go look at the data and let it help dictate.

Speaker B: I'm um, excited about it as well and I'm excited about the shift that we've made from a CAC focused organization to a ROI revenue focused organization. It hasn't come without challenges, of course and there's still a lot of education required even at the C suite level to talk about that. Especially as goals get set around CAC but then move towards roi. It is a little bit challenging. And one of the things that I would love to get your, your thoughts on as well. There is we and many advertisers probably that work with the flab, get the results from your model and then there are things that they like and that there's things that they don't like. And there are. And what I mean by that is, you know what, we have a conviction that Facebook is probably driving this marginal roas and yeah, we want to increase it, but you're telling me that TV's driving this marginal roas and you know what? We just don't buy it. And if you were acting like you were a compute completely objective decision maker and you got the Results from LIFT Lab. You would optimize your mix in one way. You would do what the model told you. But then there is the human opinion, perspective, philosophy, whatever. That then goes in another direction. And one of the things that I've been challenged by is still believing that, like brand spend, for example, and certain executions need to be done, even though the model does not indicate that they should be done. Um, and I'm just wondering how you, how you think about that balance, right? Like that, that constant fight that we're playing between. Here's what the data is telling me to do and here's kind of what I believe to be right.

Speaker A: Wow. So there's a couple, there's a couple recurring themes, I think, in the question. Let's get the tactical one out of the way. You can have low, low lift on something like branded search or affiliate or something like that at a high price point for your product. And the answer can come back by those clicks every time you can get a chance to. So, you know, the, the model may revise the lift down considerably, but it may still be in your economic interest to, to be a buyer of all that traffic. So this I just wanted you to know, like, the shape of these answers is pretty dependent on, you know, the economics of the, you know, of the particular advertiser. The. I used to tell the same story. You're asking about traditional mixed models, right, where you've done all the work, you, you know, tamed the data to the extent that you can. You've kind of looked under every rock and you're taking a, uh, marketing team through the results for the first time and they're like, channel A, that looks cool. Channel B, that's a little higher than I thought. Channel C. Whoa, wait a minute. Uh, I would call it, I can't get there. Right. Like, what's wrong with your model? And if channel C is wrong, maybe I want to revisit my opinion about channel A and channel B. Right. So I've seen that dynamic play out in boardrooms and, uh, before. And so it's always been our, uh, mantra to actually lead with. These are the areas where the model struggled with the data that you provided to train the model. So let's have a really strong dose of honesty. Let's raise our hands and say we could have told the model, start with a row as of two. And with the data set that you provided, it would always come back with the answer 2. And if I told it start with a 3, it'll come back with a 3. There's no signal in this data. Right. And being honest and transparent about that has helped lift flap kind of uh, gain like we're beating the team to the concern by airing it ourselves. Kind of like the little kid that, you know, you have a lamp behind you, you know, knock the lamp down. It's a lot better to come and say I knocked the lamp down. That punishment's a lot different than trying to hide it and then flagging those channels as the ones to prioritize where we're going to run our first incrementality and diminishing returns experiments. Right. So if the evidence isn't in the media plan data, then we need to go generate the evidence. And that's the synonym that we use for experimentation. It's really setting up marketers to go get testimony, if you want to call that evidence. Right. Expert testimony from, from their meta account or from their Google account or TikTok account. Right. That's really what they're doing is graduating from. Here's uh, the data I got. How far can you take it to? And if we answer we couldn't get, we, we tortured the data. This is actually a data. I didn't coin this. This is a data science term. Our expression, I've tortured the data all that I can and I've made it speak, but I don't know if I believe what it said. Right. And so being honest about those conditions and saying we're actually going to help you remediate the data by running experiments has really been the sweet spot between these two pieces of analytics.

Speaker B: Um, let me ask you another question on the measurement side that I'm thinking about now. Obviously, medium mix models, generally they like signal spend, impressions, clicks, volatility, variability over a long time period. And so, you know, if you are an advertiser running a multichannel mix and you are trying to think about how to diversify your upper funnel, for example, and I'm talking about the upper funnel because that one's a little bit more challenging. There might be a CMO listening to this that is thinking about their, you know, 2026 budget and they're, they're saying, you know, we're getting a lot more search interest from AI and I think we need to invest more in organic content on Reddit. Hey, Lyft, how can we actually kind of model that? Right. And that's going to be an investment that you're going to spend the dollars to maybe hire a content writer, hire an agency. Right. They're going to produce content over, over a time period. And then that's going to show up. And that trade off might be made between that thing which has a dollar amount but maybe unclear impressions, unclear translation to revenue and something like tv, which is this is how much we invested, this is how many impressions we got, clearly able to model it, right. And I feel like more marketers and brands are coming to that conclusion of hey, maybe we need to invest on some of these non paid channels. But they're still looking to a media mix model to try to answer that question. And I wonder how you think about that. And that's very timely as of today, you know.

Speaker A: Well, I do think, just like I said, there's limitations to the data going presented to the mixed model. It doesn't matter who estimated the model, right? There's limitations to the data. There's also limitations to the actual model itself, its assumption, the data that it's built on. And so I, I, I want to be really transparent and say there are strategic questions that you shouldn't answer with your mixed model. You're going to have to do it through your intuition, your gut, uh, best practices, word of mouth that doesn't go away. There's not a mixed model that's replacing, you know, smart marketers. I, I, I really don't think of it that way. If anything were, you know, the mix model should be viewed as an assistant to a smart marketer, like make better economic fundamental decisions, you know, in your media planning. But there's to me there's like your example of uh, you know, should we invest in content to try to increase traffic from a new source? You know, you don't even have any history to train that model on. So we kind of, kind of can rule the mixed model out, right? Or maybe the, this one's even more fun in your example because the data is very thin, it's highly unstationary, right. It's ramping like crazy and doubling and tripling the amount of traffic that's coming in. Uh, so yeah, I don't think you look to a mixed model to answer like it's great that you gave me a kind of like juicy question like that because the answer is a little more clear. But as you work through a continuum of questions, you know, should I double my spend on creative to try to get much better creative than I have today, right? Should I spend a lot of money on audiences because the ones that I have are stale or the audiences I rent from Meta or Google Google are kind of saturated? You know, these are questions that just traditionally don't lend themselves that well to looking at, you know, a time series of media plan data and some of them can lend themselves to be answered through experimentation. Uh, but your hard question that you answered, I'm actually going to say you get to answer that with the best AI you know, on the planet. You know, it's, it's really the marketer spinning that up and, and, and looking at it as a strategic decision that, that the, that, that the model is a lot more of, you know, we'll call it an economic model than a, than a statistical or data science model.

Speaker B: I love that my favorite parts of my job is having this kind of lens, uh, looking at my work, looking at life as an experiment. And I've taken that approach not only to my work but also to my personal life. I think that a lot of marketers, you're even you yourself taking the sabbatical, you want to experiment with different things. Do I like this? Am I interested in this? What's this problem? And it's, and it's fun and it's engaging. You know, when I, when we've adopted LIFT Lab and we've started to kind of incorporate it into our media mix optimization process, to your point, we have an experimentation framework where we have data from the Mediamix model on our existing channels. We can make bets about what to do with that. We can measure that in LIFT Lab. We've already spent enough on TV to know this is something we want to test. We're spending enough on TV to run an experiment and so that is something we can execute. We can see the data in Mediamix model, we can run a geolift experiment and that's fine. And then there's these smaller experiments for new channels, new ideas, new things that need to be encompassing our larger experimentation kind of process. But maybe potentially they're not in the medium mix model yet. So we have medium large bets that we're making. We're looking to the medium mix model for. But then if we're going to test a new channel like Reddit, well, we're not going to spend a million dollars on Reddit out the gate. We're not going to measure it with an mmm. So let's launch it, let's use deterministic and let's go from there.

Speaker A: What I like to say when you're m going to test a new channel is that it's not appropriate to count on a mixed model. I don't think the mixed model is the right place to think about measuring a new channel because that talk track goes like what's the minimum number of days of data that you want? And then no. For me the checklist for new channels is go understand the channel, go run a bunch of stuff, get your creative figured out. You should fall in love with the in platform metrics first. If those are not working for you, then there's probably something wrong and you need to actually no one wants the diminishing return curve and the economics of a poorly optimized channel. Right? So just learn how to play on that field. And you don't need a mixed model to do that. You don't even need an incrementality test to do that. Once you are in love with the end platform metrics, then I would say, uh, do this as an incrementality test and it becomes your decision whether you go dark in a small portion of the country and these are nationwide campaigns that are already kind of in your. Or do you flip that and say I'm actually only going to light this media up in a handful of markets and see what it does. So those are complements of the same test design. And then what you learn from the incrementality test can in our case be placed directly into the model. So now I can say what I learned from the experiment, how does that play out in a whole multi channel media plan where I'm trying to make bets? And so that's a great example of where these two pieces of analytics really do scratch one another's back. What you learn from that experiment on the new channel, those answers are going to vary a month in the future if the price of your product changes or are going to vary three months into the future if you're also running that media during cyber week. Right. So the model handles that really well, whereas the experiment, that was a point in time, you know, during summer, just isn't really conducive to understanding some of the uh, follow on questions that are going to be required of, you know, the follow up questions on how do I spend on this new channel? So for us the answer is definitely an ant.

Speaker B: Um, you talk to obviously a lot of marketers, founders, CMOs. It probably varies by, it definitely varies by industry, by company stage. But I wonder, do you think that generally folks are too trigger happy to test a new channel and don't focus enough on optimizing existing or that they focus a little bit too much on existing and don't spend enough time testing the new thing?

Speaker A: Hot take. Well, in my perfect world they could first answer the question how saturated are all of my existing channels, because if they are all saturated or have marginal ROAS is less than 1, then you better be looking for somewhere else to run your media. It should have already started looking, actually. Right. And so then if you do run that experiment right, on the new channel, you get a chance to see not only should I stay in this channel, how far can I scale it? Net of that. So that's my ideal world. Net of that. Most of these decisions are made on gut, right. You know, or herd mentality, right. Like everyone's ramping on TikTok. We, we need to have a presence. Right. And so it's okay if in this case, not everything gets, I don't know, laid out neatly by economic decisions like I'm laying out. It's fine for folks to go get their sea legs on, on TikTok, maybe get some quick wins, maybe lick some wounds, uh, and then turn their focus to what are the economics of this? It. It. But your general question, brands typically do this without a model. They, they're. They grew up on Instagram, let's say, or maybe they're very bottom of funnel, they're very transactional and they grew up on Google, right. At some point they feel the saturation of that channel, so they go to the next channel and then they see another round of growth and that starts to saturate and they go to the next channel. And then when they've gotten to four or five channels, they start to feel that there's a lot of overlap between these channels. And that's where the question of how do I do, how do I treat this now as a portfolio? How do I balance the spend and mark to market, if you want to call it my portfolio, so that I, I can rebalance this with all the new information that I have. And so that's the crawl, walk, run, you know, that we see. And once people are in five, six or more channels and something new comes along, I think it's I salute people. You know, unless they've dramatically underspent on the media plan they already have, I salute them going and tinkering and then putting it under the scrutiny of an, of a, of an incrementality and diminishing returns experiment.

Speaker B: You know, there, there's an element of what we do, there's the human element of what we do. And like you said, a lot of these decisions come down to there's a smart marketer and there's a tool that we leverage. There's also a lot of these thought processes and decision making, which is fundamental, repeatable, Right. And now with AI becoming so much more prevalent in our space and companies now focusing on it, I have been thinking a lot more about what are the elements of the tasks and things that I do that can be automated without me thinking about it. And the budget making decisions are definitely an aspect of that. Where there it falls the it toes the line of you need to have some conviction, there needs to be the human there. But there's also some like if this, then that very simple rule based things. I wonder how you're thinking about the absolute craziness that is happening in our industry right now, this pivotal point that we're in and how Lift Lab is leveraging it too. Right. Because I know that other MMMs are starting to incorporate LLMs into their, into their kind of software, into their tooling and I'm really curious what your perspective is there.

Speaker A: Thanks uh, for asking me. You know, for me I've never felt that the, or the experiments or the combination of the two replaces a marketer. I think that they give sound as sound as evidence that you can buy about the economics of your partners. Right. The meta stick to Googles of the world and that those become an input into a decision making process. But ultimately the inputs that we want to provide, we talked about it earlier on the call, are things like marginal roas. You can uh, just derive from that roas, uh, the profitability. And I like to say that Lift Lab does not own the P and L. And what we've observed is across a lot of marketers that uh, quite often you may start the fiscal period out focused on profitability but everyone throws that out the window. If you're towards the end of the period and you haven't hit the growth number and so the ability to tool teams up, to walk into a conference room or boardroom to say I can deliver X if you give me Y, I can deliver Z more if you give me Y. That is what we're excited about helping marketers to do. The role of AI in this is to further automate any of the decisions that are happening about the economics of the media. And I think of that as an AI assistant for media planning. That's our manifestation of it. And so any way we can make a marketer's job easier by having agents, AI agents that are running on your behalf that are taking advantage of more volatile signals. Uh, so you could say in a use case I had a forecast but the price of, I don't know, non brand search clicks just changed. How would that impact my forecast? And should I reallocate my budget. And here's the suggested reallocation. That to me is absolutely in everyone's interest. The vendors should be pushing themselves to create that kind of software. Marketers should be opening to take it, putting it into their workflow and depending on it. But I don't know, let's contrast that with a much more strategic variable, like we're being conquested by Amazon on these keywords. How do we want to react to that? The, Is there some way, how did you get that information? Maybe an agent, founder, maybe not. But like that variable is not in your model, uh, at least in the example that I'm making up. And we are still going to need the best AI on the planet to go react to that. So to me it's a hybrid. I don't, in the, I mean I've been uh, in whatever generation of it, I've been working with AI all my career, we've had lots of discussions about the man versus machine. I'm as excited as anybody about this round of AI, the generative AI round of, and its impacts are definitely accelerated. So it, for me it's the uh, design of how do we put that in and augment a person, make them superhuman, but still not let them let their guard down to making the kind of tough calls that an LLM can't be trained on. Right. That hasn't seen the data for that type of thing.

Speaker B: Sure, yeah, I'm realizing that, you know, it's going to be easier to answer questions, but marketers are going to need to get better and better at asking them. And so a big part of what we're doing is developing a very comprehensive prompt library to be able to identify those questions and also have those questions documented in a way where people across our team can leverage them. Because, you know, the most senior marketer might know how to ask a question about paid search competition, but maybe the most junior marketer doesn't. And so just having that backlog is super helpful. I have a few rapid fire, if you would not mind indulging me. And this has been a great conversation, by the way. John. Um, first one, favorite book all marketers should read.

Speaker A: Uh, I'd say Market Response Models by Dominic Hansen. So I, I, he is an advisor of ours. He wrote the textbook on media mixed models. It is a textbook and it's a little thick, so I'm saying this with a smile on my face. But if you want to know where all of the theory came from for mixed models, that's the book that does it, uh, so I, I, I'd have to say, will I make a lot of friends with my answer? There's probably some people that start that book and put it down and there's other people that will read it cover to cover, but uh, it should be on your bookshelf.

Speaker B: Okay. Definitely not Friday evening reading, but something to dig into. Awesome. Second one, biggest mmm myth you'd like

Speaker A: to debunk, uh, that we can debunk. Download all of our spend data out of ad platforms, stick it in a regression model and magic will happen. You know the number of models that actually you asked that question earlier, you know, that says, hey, I'm walking through a series of results and one of these just doesn't pass the sniff test. The most naive models, well it, but it doesn't take much to kind of debunk one. Right. If you only spent on TV in your lowest part of the year and your model makes the conclusion that I can increase the accuracy of the model by saying TV made sales go down, that model should never be shared. Right. Uh, so to me it's that you can do everything with data science or AI and not, not have a fundamental understanding of behavioral marketing and behavioral economics. And so when we say the word model, we, we mean an umbrella over all of that. And, and it's, it's kind of fun because as long as I've been in this field of, of of data science in AI, we've always said that, uh, the very last mile is estimating the model. Right. Presenting the data to the model and estimating the model. And so much of this comes from, but I think I like to refer to as problem formulation. And so yeah, for me, debunking mmms are people who think they can skip problem formulation because they're going to be sorely surprised.

Speaker B: Awesome. Last question. And this is probably a deep one as well. But the best career advice that you've personally received, do you have an idea of, can you think of some of the best career advice that you've received? Uh, and if you want to share from who that was, that would be interesting as well.

Speaker A: Well, it's going to be hard for me to remember who said it because I think it was whispered to me and just different shapes and passions. But I'm going to answer this. I think it applies to anybody's career, but most definitely as an entrepreneur and that is your reputation is your only asset. You need to be just treating everyone the way that you would want to be treated or the way that they want to be. Treated and really deliver, deliver, deliver, deliver, deliver. Uh, and if, if you do that, there's a compounding effect that comes along with it. And if you don't do that, there's a negative compounding effect that comes along with it. So I, I, I, I, I, I don't think you can get far from that advice to say your reputation is your only asset.

Speaker B: Cool. John, thank you so much for being on the show.

Speaker A: All right. Hey, Paul, thanks for all the questions.