Agentic MMM with Dr. Luca Fiaschi (PyMC Labs)

Speaker A: Measure up, measure up what's the day? Measure up, measure up Listen to what we will say Measure up, measure up, measure up.

Speaker B: Hey Simon.

Speaker A: Jim, how are you? I didn't have my microphone on. How are you?

Speaker B: I am pretty good, Pretty good. Uh, you know, we're going to be talking about something that is, uh, really exciting today. Um, and I was, you know, I was trying to think about a good intro or joke about agents for today's episode. I was thinking of things like James Bond, you know, famous 007 agent. Uh, but truth be told, I just couldn't come up with anything funny. So, um, and you know, my, my open Call agent also couldn't come up with anything funny. So, uh, we're just going to jump right into it today. Okay.

Speaker A: I was going to say LLM. Like that is one of those interesting things about LLMs is that they are not very funny. Uh, and I don't know why that's such a humor, such an innate human, uh, function, I guess. But, uh, yeah, well, let's dive on in. Let's uh, I don't know, agents of change, something like that.

Speaker B: Yes, Agents of change, yes. You know, and I don't think our guest today is a secret agent, but, uh, he does know a lot about agents. Um, so today we're joined by Dr. Luca Fiaschi. Luca is currently a partner at PMC Labs where he leads the Gen AI vertical, helping Fortune 500 companies deploy LLM powered systems and drives research and development initiatives on LLM uncertainty, synthetic consumers and agent based simulations. Before PYMC Labs, he was a Chief Data and AI Officer at Mistplay, VP of Data Science at Stitch Fix, VP CP of Data and Machine Learning at HelloFresh, just to name a few. Uh, he has his PhD in AI Computer Vision and Machine learning from Heidelberg University. And when he's not behind the keyboard telling his agents how Beijing mmm works, he has been known to go skydiving and I believe, race motorcycles. Welcome to the show, Luca.

Speaker C: Thank you very much, Tim. And thank you for inviting me. It's great to be here.

Speaker B: Yeah. And did I get that right? Do you actually race motorcycles or do you just.

Speaker C: I do actually. This is like, it's me single motorbike. So this was one of the first, uh, actually LLM's techniques were invented if you remember a few years ago for uh, neural style transfer where you could take something like uh, a, uh, Picasso style and then apply to a picture. And so, uh, this actual like photo of me on a motorbike and then my wife used that technique to enhance it and um, become this uh, like art.

Speaker B: So all right, this is, this is a first for the Measure of podcast. Our first time having a motorcycle uh, racer, uh, on the show. So that's good. What's more exciting, zipping uh, down the highway at like 150 miles an hour on two wheels or uh, programming agentic marketing mix modeling.

Speaker C: Well they are both very exciting. I would say they have a different threshold of risk inherently. But uh, they're both like very exciting uh, things to do actually. Um, yeah, I think because I spent so much time um, sitting and thinking about problems and uh, engineering and strategies and technical stuff, uh, I need to disconnect. Then I need to do some uh, interesting sports, uh, to rebalance. And so that's how my brain works I guess.

Speaker B: Yeah, that will definitely do it.

Speaker A: Yeah, yeah, that is uh, well, high risk. But also I guess uh, you're talking about risk thresholds there. And as we think about um, look, I think there's always been this idea in the broader advertising world that um, there's almost like a Turing test associated with this idea of look, if an agent can actually build and interpret an mmm, that is this moment of uh, we'll call it agentic measurement. Um, and maybe it's a good place to start just to be what is agentic mmm? What does that mean to you? Um, and how are others defining it if it's any different from the way you're thinking about it?

Speaker C: Yeah, very good. As you know like measurement has a long history, um, like in marketing and um, mmms has become one of the primary um, measurement frameworks adopted by many companies for many, many different reasons including like cookie deprecations and so on and so for forth. Um, and so this idea is that um, they were always very hard to build as models because of all the problems we have with marketing data. And they are very hard to put in practice. And so although they are like I read that these surveys from emarketeers, very recent, a couple of months ago came out and says that 62% of all advertisers are using mmms, um, and I believe 90% are working on it, um, still companies find it very hard to use them in their day to day practice. And uh, our core belief at PIMC Labs is um, real problem is the language, uh, barrier between the people who built these MMMs and the people who actually using. And if you can solve that language barrier and LLMs are a universal translator so they can solve actually the problem you could actually use way More sophisticated analytics in your day to day decision making, capturing a lot of value uh, for every company. And so agentic data science, to come back to your question, is our solution to combine advanced analytics and to empower stakeholders with advanced analytics and agentic mmm uh, it's a way to connect very sophisticated models with an interface that make it very, very easy to use for everyone.

Speaker A: Okay, so it's really focused on the model piece. So one of the things I think about a lot is how much time, effort and energy goes into uh, the precursors with mmm as we think about uh, data cleaning or even you're saying the data's so messy and maybe places. Is that like, I guess maybe walk us through how you're thinking about the various components of developing MMM and how you're seeing the agentic uh, functions applied uh, to each of them.

Speaker C: Yes, um, very good point. Um, so I do think that there are two uh, stakeholders in the mix of building and using an mmm and these two stakeholders are the data scientists that creates them and then these stakeholders that use uh, them for like the budget allocation process and they have different needs and agentic AI can play in both of UH areas. And when I think about uh, an end to end MMM m solution from the build stage to the actual user stage, um, there are different components that need to line up to make these real and possible. And agentic AI can play a role in each one of them. So from the data consolidation process, for example, you can have agents that smartly uh, merge the data and do the data validation steps. Two agents that um, intelligently build the model and incorporate priors and business knowledge into the way these models are built. And then on the second step, once you deploy these models, once they are used, retrained on an actual UH pipeline, um, you want to have agents that can help you translate um, the uh, outcome of the models to the stakeholders in a way that can understand and allow the stakeholders to ask follow up questions. And the follow up questions are often what really um is one of the biggest challenge to solve in business settings. Because um, it may happen to the stakeholders, um, by the virtue of the interactive process of developing a business, they may want to ask 3, 4, 5 different scenarios and then they come up and say hey, but what happens if you change this variable again? And nowadays this is taken care from data scientists, but um, really you want an agent to be able to do that. So across the entire chains this does two things. One, it elevates the work of data scientists to be a little bit higher level so the data scientist, not anymore the person who codes the model but it's the person who lines up the right agents in the correct way so that they can produce a uh, model that then addresses the problem of the stakeholder. It is not the person that addresses the questions uh, of the stakeholders anymore but it's the person who uh, builds the model and deploy the model uh in a way that can be done effectively. And then of course it has again obvious advantages in terms of speed for the stakeholder downstream.

Speaker B: So this may be getting ahead of where uh we're at right now. But uh, the question that just pops into my mind and I can't quiet it down is at what point do we no longer need a data scientist? Are the agents going to be good enough where the cmo, the head of marketing is just working with the agents directly or do you see there's always going to be a place for a uh, knowledgeable data scientist with domain knowledge of the business to be working with the models and the agents themselves. And then kind of being that in between uh, layer.

Speaker C: Yeah that's a very good question. And uh, it goes I think really like at the core essence of what the transformations we're going to see in the next year they're going to play out. Um, I do think that um, generally um, I do think that there are going to be only two things that distinguish us humans from AI ah in the future. And these two things are intent and taste. And so intent to tell the agents um, what they should be working on. That's a human uh judgment like capability. So we are going to be able to line them up to solve concrete problems and telling them what concrete problems even at high levels, what general objectives they should think about rather than you know, the fine grain downstream like this is the task component of the problem you need to solve. But that's a human thing. And then the other one is taste and judgment. So is this really correct? Is the agent uh missing something that's fundamentally important or is the agent output just slope? Uh so that's the two things that data scientists are going to be doing. And then generally um, I do think that to deploy these systems with that translating practice that's going to be still data scientists needed ah across organizations if you want they can play out in a way that's going to hopefully, and that's my belief, increase the demand for data scientists because businesses that nowadays they um, can't afford to hire uh, data scientists uh anymore they may be going to become more efficient and data scientists will be able to operate not just across the marketing domains but across marketing and finance and supply chain operations being able to uh, focus themselves on capturing more and more uh value across, across the organization. So one of the things that we used to say um two to three years ago, maybe a little bit longer was that the data scientist was the sexist job of the 21st century. Well it doesn't maintain its promise yet and um, there is a series of reasons why that's the case. But uh, the reality is that data science techniques are not yet used by other organizations in the way they should be and it's really a hard problem still and maybe AI has a shot at being that solutions and maintain that problems in the longer term. So that's my core belief.

Speaker A: It's interesting. Sorry Jim, just the one note there when you're talking about um, taste because I've been thinking a lot about taste and sort of um, I don't know the opinionated nature of mmm um at times and I was reading a paper recently about being um, able to uh, ah for agents to be able to uh, uh generate uh taste or to be able to um, develop taste ah in the scientific scenes and they were talking about reinforcement learning through community feedback. Is that, and when we talk about these roles I guess I'm wondering does that mean though that we would be, you know the data scientists that perhaps that are working at pymc, they'll be the ones who are providing the core community that can all be calibrated by what they're hearing from essentially just do you have confidence to invest these dollars or not? Um, do you have confidence in the output of these models versus having in house data scientists in that regard because to Jim's point it sounds like we're maybe close to that chasm or crossing the chasm of the vibe measurement moment where it would just be look the model is going to be calibrated and the systems and the agency but at the end of the day you don't need a high degree of data scientific rigor in order to actually um, um use a model or apply it to your business. Or am I off base?

Speaker C: Um you're not. I do think it's going to be very subtle here. Uh the difference of course we are embedding a lot of knowledge uh into these agents in the first place and you know ways of we're doing that is by codifying this knowledge into skills and uh, these skills are opinionated ways of using specific techniques and libraries that in our experience at PMC labs Um allows you to get the uh, best outcome for your business. But uh, um, and so one way of arguing is that well maybe um, the ability to deploy these agents in the enterprise is going to be democratized and a lot of people, they're going to be able to do that. Um, on the other hand though, um, like there is, there will be I think still the need for like a validation of the end to end agents process once it's there. And the agents is going to build checks and balances internally. But there is so many, let's say, nuances to a specific business that at least once you set it up for the very first time and for a few times there is a uh, very high degree of customizations and judgments and review that needs to happen in these uh, agents pipelines to make sure that it is correct and effective and so on. And you can think about um, these also being a factor of the increasing level of sophistications and demand that um, you know, it's gonna happen as a result of being able to deploy these agents um, for building MMMs at a larger scale. So what's gonna happen I believe is that probably we're gonna solve these simplest cases of mmms. So if you are in E commerce, all digital channels and you um, wanna, you know, without having to implement too many um, gift tests, you wanna deploy like it's kind of what we call a vanilla MMM. Probably the agents will bring you 95% there already. But then there is a component of, hey, you want to start incorporating um, like lift tests results. And so you want to build an experimentation program around the results of the MMM and that program. Well the agents can help you design it and can even help you implement it. But you need to put some judgment into the way it's structured. And then the other one will be oh, maybe you want to measure like more sophisticated effects and so referrals, uh, for example on top of the standard MMMs. And so you're going to keep asking more and more sophisticated questions that the current agents may not be able to tackle. And then you're going to need to build more complex agents pipeline on top of what you actually have and somebody will need to validate those. And so I think it's going to be a shifting target and there's going to be always the need for data scientists to be in that field as humans that have judgments, intent and taste and can really assess uh, what's needed, why is it needed and is this ultimately correct for the business?

Speaker B: Yeah, the thing that um, I Agree with all of that. I keep going back and forth and thinking like, oh, you know, agents and LLMs will never be able to do all of what we do. We'll still be needed. And maybe that's just the human of me trying to justify my own existence. But um, I also have another side of me that thinks, yeah, eventually they'll just be able to do everything that I'm doing right now and a thousand times better. But coming back to the um, maybe some of the more like specifics of the agents themselves. Um, are we talking about just like, you know, we have a model and then we're chatting with ChatGPT or you mentioned skills. I know, um, maybe let's hone in on that because like that's maybe a big part of the agent is like we have to give them skills that tell them knowledge about our business, knowledge about how mmms, uh, work and how to build them. You know, docs about how PyMC marketing works and things like that. So I know PyMC labs had done some research recently around just using like Claude code, plain vanilla cloud code by itself and trying to build mmms with that versus cloud code skills. Um, maybe talk a little bit about the benefit that you get from there once you start going beyond just I'm chatting with a cloud code versus okay, I'm giving IT skills, I'm giving IT knowledge, I'm giving it context. What were the findings that you got from that?

Speaker C: Yeah, for sure. And I think maybe even before you go into that question, I think it's helpful to kind of like uh, explain um, like very high level, what's a standard LLM, what's an agent and what capabilities uh, really enable an agent. Um, I do think that there are many different definition and effect. It's a running joke that nobody knows what an agent is and everything is an agent. But I think at the core of it is the agent is an LLM with the ability to interact with the environment and um, remember things. So reasonable things and remember things. And so if you accept this definition, a standard API calls an LLM. It's not an agent to have an agency. You need to give the ability to do something with that LLM call. So the LLM needs to be able to call some tools and LLM needs to remember what uh, he has done. So collect the results of these actions from the environment and remember what he has done. So clone code is an agent because it can write code, run code, I can get feedback of running code, um, and incorporate these to modify it remembers it and modify its behavior. And so cloud code is a very powerful agent. In fact if you ask um, cloud code to build um, uh, code for like um, uh a lot of different coding tasks, including data science task building models, it's gonna give you code that perfectly runs, maybe not at the first try. So the first try is gonna find a bug here and there. But what's gonna happen is it's gonna report back very quickly, uh, it's gonna learn from a mistake, look at the error iterate and gives you back something that's good, that's running. And so in this case I say that codes where you have a verifiable um, domain and output like coding tasks, they can be solved by agents today very effectively and by coding agents very effectively. Now you get to data science, you are in a domain that's less verifiable. And the reason why, especially for causal inferences solutions, the reason is because uh, you have problems which have uh, degenerate solutions. Um, let's say an mmm, you can have uh, 10 models, they fit perfectly your data but they have different, they correspond to different uh, ways of understanding the world and reality depending on their configurations. And one of that is maybe the one you want to prefer or the others for many different reasons including business logic. And on the other side, uh, there is an exponential number of choices you need to make during a typical data science process. And so people call it the ah, ah, three of 14 paths. You decide to pre process your data in a certain way that has a cascade on the way you build the models later on that are on top of the data. On the modeling side you have different ways of structuring your model. And so it's an infinite amount of possibility. And because there is no strong guiding principle on this possibility, a priori, when you start the modeling process, unless you have some best practices knowledge you want to inject, uh, you can end up anywhere. In fact cloud code has probably read any possible book in statistics, uh, and it's going to try and land in some solutions of the space that is actually perfectly reasonable. In a way, is it the best one, uh, according to best practices that you know they need to be true? Probably not. And skills, it's a way of helping uh, cloud code to select out of these solutions the ones that you think uh, is going to be the best. And I can give you a concrete examples um, of something like this. So we had measured um, one way of measuring this is to give uh, the LLMs, um, the agent um, coding the task of Coding specific well known m, um Bayesian models, uh, for example for stochastic um, price volatility. And what happens is that in uh, almost let's say 100% of the cases, cold code is going to produce code that actually runs uh, and delivers any inference. Now when you go and analyze that inference you will find out that uh, um, the model has a fit, but the fit is incorrect because some of the parameters are actually misunderstimating because of really numerical subtleties. And the reason why it does that is because he has used ah, a version of the library to write that models that uh, actually was the old version of the library or it has used a way of parameterizing the model that's not, not uh, the most effective way of reparameterizing the model. So the model that comes out is complete uh, garbage. In fact when you start evaluating the correctness of the models downstream as a criteria, you will find out that with and without or with persons without skills, the pass rate tremendously increase from something that's close to 0% to something that's close to 70% uh, on something of the hardest problems. And the reason why it does that is that the skill doesn't really teach the agent how to reason, but it tells the agents hey, if there are five different ways of writing these models, well the most appropriate one um, for these classes of problems is the number fourth and this is the reason why. And so it biases the models to follow one of staff specifically in this like forking world of choices.

Speaker B: So in essence you're basically giving it skills files that say here's how, here's all of our knowledge about how to build models from years and years and years of experience. And this is the preferential way to do it. If you come across this situation, try this parameter setting or if you, you know, if you're going to build it with this model, use this version, things like that. Do you find that um, it always adheres to those skills or does it sometimes ignore them?

Speaker C: It does ignore it a lot. And then you come back to another problem. The skills is not enough and you need to hide ARC checks. And so because the agents um, so they're becoming way way better at following instructions. And in fact they can hold way more context and you can add longer scale files and you know, people is already working on models that are 100 millions context windows. So you will be able to hold like a uh, lot of instructions into the agent. But it's still a probabilistic model. So sometimes may choose not to follow the prescription of the skill. And so how you get around that is this ability of the agents to interact with the environment, um, not to um, just give you the output of a prompt. And so in the environments you can do anything you want. You can set hard checks on the output of the models, uh, to uh, ensure that certain constraints are actually respected. A simple example from Bayesian modeling is to say, hey, check, uh, a quantity which is called the hour hats. They are a quantity that it's derived from agent models and it's statistical quantity that tells you whether the model has a good enough fit or not. And instead of telling the agents to verify this property, you can just add it to a callback to every model that it creates by adding it as a deterministic script on top of the code base. And so this is a, uh, check that needs to always happen. Don't tell the agent to do it. You tell deterministic script to do it as a hook and a callback. And so that ensures that you set guardrails for the agents to always follow certain things. And the feedback you give from these deterministic script and guardrails is taken into account from the agents, but then says, hey, wait a minute, uh, uh, here I'm following the best practice, but, um, the result doesn't look good according to this feedback I get from the environment. Let me change my behavior and pivot in a different direction. So that's what really distinguish an agent from a simple LLM call.

Speaker B: So this is interesting. This is just like kind of blew my mind a little bit and opened it up and the sense that like, I'm thinking of like, okay, I have an agent and I'm instructing it and I'm saying, okay, here's, here's your skills file. Now build this. Mmm, here's the data. Uh, you know, here's some context about the business. Now go build the mmn. And it comes back and I look at it and like, and then I respond to it. I'm the person like at the keyboard responding to it saying, no, no, the r hat's way too high. Try again. But what you're saying is like, no, no, that's just all a program that you've written a script.

Speaker C: That's correct.

Speaker B: Given back. Okay, that's amazing.

Speaker C: And that's what I meant exactly. You bring it up again. I meant exactly what the rule of the data scientist. The rule of the data scientists are kind of designing this dance between the model and the environment so that these dance end up in the right spot rather than doing the implementation and dancing together with the model.

Speaker B: Yeah.

Speaker A: Mhm. There's so many interesting pieces of this. I guess I'm still trying to reckon with the realities of uh, maybe the time to value piece here. Do you see that massively diminishing. And I'm just trying to, I don't know, from a business stakeholder point of view, they look at this and go, cool, you've added more, frankly maybe even more cost into this moment in time because we're using all these tokens to all these things. Is ultimately, is it going to provide me with a better, more confident output? Is it going to be faster? Like what, what are the real value props that you see associated with using agents throughout this journey, knowing that you are probably increasing the degree of risk that you're uh, introducing into these systems?

Speaker C: Yeah, it's a very good question. Especially nowadays where you know like um, anthropics and probably was going to follow up, other companies are trying to rise the price of the tokens. And you used to say that tokens used to be a cheap commodity. Uh, we have seen that for a long time, actually for a couple of years because yeah, it's one of my

Speaker A: greatest fears is that we all build all these incredible solutions and they use mass amounts of tokens and then the price of tokens quadruples overnight. And you go, okay. And it's like uh, you invent the car and then gas is now a thousand dollars a gallon. You're like, why do I have a car? What am I going to do now? Yeah.

Speaker B: Drug dealer is like here, first one.

Speaker A: Yeah, exactly. And the system is heavily subsidized. We know it is. That's a real challenge.

Speaker C: Yeah, um, it is also like uh, um, let's say a consideration of the company needs to take into account uh, in the longer term very much which hasn't been primary until very recently. I do think that a possible solution to that now that's the ultimate solution is think about in terms of like uh, sovereign AI and the ability to deploy your own LLMs internally rather than only rely on vendors and have absolute lock in. Um, generally, um, it is very complicated. So the question of whether you build or you buy something and you expose yourself to increasing price later on. It's always a very tricky question for companies and there is no right and wrong. There is a lot of considerations. Um, now what I do think is that generally speaking if you free up one of your data scientists that was only dedicated to build. Mmm. Say you are companies with uh um, let's say like hundreds of business lines like Colgate for Mole, uh dozens and dozens of countries, uh complicating pricing structures or interactions across um the different uh product lines. You have very complicated media, uh mix models to be able to estimate that you have a team that's dedicated to these. If you find that you can leverage that team and uh, being able with kind of the same headcount to handle more problems then you get more value out of it from the business sense. And I do believe that the demand for problems and analytics is going in the longer term it's going to be way way higher than our ability to build these models even with agents. And so what I was pointing out before is that because I think that the demand is going to increase exponentially faster than our ability to deploy these models on these specific business problems. There is going to be still a role for humans and what's going to happen is that people will do more uh, with the tools that they have at hand. So.

Speaker A: Mhm. Okay. So overall uh, I guess it is the velocity but it's also the time reallocation scope here.

Speaker C: Yes. That's what unlocks the value.

Speaker A: Right.

Speaker C: So you're going to be able with same headcount to unlock much more value because the data scientist that was not really specialized in marketing can now do. Wait supply chain optimization can make me do some kind of like very sophisticated financial models for your uh investment team to decide whether you want to buy an extra company. And before you needed to ask a McKinsey to do something like this for you.

Speaker A: M. Forgive me just namesake. And it's almost halfway through the discussion and where just um bring this up. What is the PMC position at this moment in time as to what you're actually planning to do in the future? And I say this as well with the mind on uh, when we think about PMC as far as I'm aware being the most rigorously uh developed open source model, uh open source uh platform right now for MMM development. When you compare it to say Meridian and um, Robin and so forth. I don't know if I'm hearing anything about them doing agentic mmm m solution solution design. But I am hearing about it from some of the um, closed door providers out there and I guess I'm just wondering like where do you see the future of this going and is this maybe more of like PMC is going to have to take um on more of a managed services capacity as you think about um, you know the use of the platform across the board or just what does that look like? Are we now, is there a major divide between those two camps? Are all MMs going to go with Gentech? I'd love to know what you're thinking there.

Speaker C: Yeah, I think you don't understand PMC Labs if you don't think about a little bit about the DNA of the company and where it came from. And uh, Thomas Vicki, who's the founder, um, fundamentally came from the open source community in the first place. And um, I think we are very much rooted into this. Of course we are a business, we need to make money at some point. But we also curious scientists who just want to implement new stuff and also be precursor when the future is, is going. And so from our side we uh, understand and we have this thesis that the role of data scientists is going to change in the future and it's going to happen no matter what, whether we are involved or not. And so we want to have a stake in just shipping it into certain directions and directions that we think is effective for the community is going to empower data scientists in the longest term rather than really get them um, into a side role. And so the way we focus on this is because we really believe it's going to solve a series of problems of data science at large, even beyond mmms that um, we want to tackle. And so our position is in longer term is that they're going to open source as much as we can of our stack. And also we think that in a world where um, you know, building software is extremely cheap, um, I don't know how closed source MMM solutions uh, are going to be able to prevail. And uh, meaning if you have all the components out, uh, and you have somebody that uh, can put them together relatively quickly and all the components are open source, well how can you justify closed source platforms? And I, I personally don't see it. But what you can justify is the expertise of the data scientists both to act as advisor to tell you, hey, do you need this solution? That's the roi, uh, can generate for your company and then hey, let's put it together so that you have this proprietary ap, customize it to your business and that allows you to capture the value you need to capture. And so our position is a consulting advisory strategy in this area rather than deploying our um, you know, like a SaaS platform for MMMs.

Speaker A: M. It's a. Oh, sorry Jamiga.

Speaker B: Oh, ah, yeah, no, I was gonna say and that's one of the things that I absolutely love about Pymc Labs, coming from that open source community, I'm a bit of an open source junkie myself. And so like I know, uh, is it Decision Lab? Is that the first part? So we've talked a lot so far about the using LLMs and agents to help the building of the models. Right, that's Decision Lab that's been open sourced, is that right?

Speaker C: Yeah. So Decision Lab is the building of the model builder. So that's the component that has been open source. Um, so PMC Labs is the consulting company built around the PMC ecosystem. But we started to build this brand also called Decision AI. The entire field of using agentic data science, we call it Decision AI. And uh, as part of these we built several open source stack. One is called Decision Hub, which is an agent skill repo open source. Then we have Decision Lab which is the uh, Agent builder, uh, agent scientists, and Decision Lens which is this interactive agentic dashboard. And the reason we do that is also to um, uh, let's say have a clear positioning also with our community of PMC developers. So I do think the idea of Bayesian modeling, its core to the PMC Labs ecosystem and we don't want to abandon that. In fact there is people that come to us and say, look guys, but I'm not interested in this AI stuff and it's completely fine if you're not interested into this. Um, the way we're going to brand that is branding it towards this Decision AI stack to really differentiate people that are interested into combining agents with traditional models and people who are just interested into using traditional models in the way you used to do it before, which is, you know, it's a totally defensible position too.

Speaker B: Yeah. And Decision Lens, um, is something that's really interesting. I know that's not kind of fully

Speaker C: been, uh, open source yet.

Speaker B: Open source yet. Um, but uh, Simon, I was on a webinar last week with Luca, um, and they were showing the Decision Lens part of the stack, which is basically okay, you've built the model, you know, it's a good model, it's got good fit, it's validated out of sample forecasts, get all that good stuff. But then the problem is always the bottleneck of okay, you, you, you spend, you know, a week preparing some slides and recommendations and you have your 72 page slide deck that you present to the CMO and the head of marketing. And then nothing happens. Right. Because they're like, oh, that's nice. Or they, maybe they, maybe they're interested and they, they ask a bunch of follow up questions and then the data science team goes back and they like spend a week or two or three or four trying to get answers to those questions because they're very thorough and rigorous and by the time they get the answers it's too late. And so Decision Lens is basically the agent, the LLM on top of the model. So now the head of marketing, the manager of marketing, the cmo, can basically just talk to the model and say hey, I was thinking about increasing spend in this channel. What do you think about that? Or hey, why does, why are you saying spend more in Facebook when the ROI is this and not this other channel that has a better roi? And then it looks at the diminishing return curves. It's basically like yeah, it's chatting with the model. And uh, I have to say uh, Luca, the webinar, the live demo that they gave was amazing. Um, it was like peeking into the future. It was really nice.

Speaker A: So the one thing I'm thinking about here though is in uh, conjunction with this, there's also a movement within the world of uh, paid media buying toward um, agentic management concepts. And of course Mark Zuckerberg about a year ago now came out and was like, just give us your credit card. Um, I don't think it's going to be like that. I think there are still weights and biases and balance, I guess, uh, guardrails that need to be enforced and that'll be utilized. But do you see a future world where the outputs of uh, an MMM and you know, run in this capacity with an agent that can provide the context as to why and the follow ups would just be agent to agent interaction level and be a just a component of the agentic buying uh, mechanism and not necessarily observed by humans the majority of the time.

Speaker C: Yeah, 100% actually. We uh, really very much think that then the next step to unblock decision science at scale is actually action on those recommendations, uh, and insights. And uh, I do see that the last component of Energentic mmm stack in the longer term is the ability to buy and to modify the campaigns that are running live or launch new campaigns uh, in real time using agents. And um, we have actually um, a prototype of these concepts for um, some of our clients. We do have an agent, we have a discord community which is called Decision uh AI. So if you enter the discord community, what you're going to find is that you're going to be greeted by our agents called Damon. Damon is our overall orchestrator. The thing that he does it can interface with all our agents, including the, um, builder of MMM models and the uh, decision lens agents that gives insights on existing MMM models and that can do and orchestrate work downstream. So it can tell, for example, Facebook, hey, um, uh, diminish the budget on this set of campaigns. And so the bringing it all together, the insights with the actions. Having a set of agents that can coordinate each other across multiple surfaces and have at the same time the entire context of your business, I think is going to be the differentiator and as people become more and more comfortable with the entire stack and with this tool. So they're gonna want to uh, give, you know, more and more agency, uh, to these systems.

Speaker B: Yeah, in a sense. It's funny because who are we talking to, Simon? Was it someone on the podcast just recently that was basically making the analogy to like the stock exchange, like trading? Exactly. Yeah. So go back and listen to that episode. We'll put it in the show notes. But yeah, it's like the stock exchange traders on NYSE have been doing this programmatically for decades. I, I can't wait until someone's agent decides to spend like a million dollars in a day on Facebook and they're like, wait a minute. Whoops.

Speaker A: Yeah, well, I mean, I've said it tongue in cheek at a couple conferences, but this idea of agentic insurance, I think is, is a very real cottage industry that we will see rise up at this moment in time because agents are going to do things and just like Waymo, uh, you know, well, Waymo is an interesting one because they have this, this combination of this deterministic signal through the LIDAR as well as AI utility in terms of interpretability. Um, but there are considerations there as to the insurance model that is used and the, you know, the number of times a human messes up. This is a vehicle. And I look at this in a similar capacity. Yeah, like there's, there's this world where we have very little tolerance for, um, agents messing up and if they are going to mess m up at that scale. By the way, Ford is where I think the thing with the stock exchange one, I was thinking about that a lot more. I was like, yeah, it's kind of true. But the thing is that the financial system itself isn't changing the rules of how dollars are processed. Right. The idea here though is that META is changing the rules because they're changing their algorithm on an ongoing basis. So it's really hard to have that degree of confidence of just because something has worked historically, just because you've seen the same curve before. Doesn't mean that you'll see the same results if you invest up to the level that you think that you're very confident in. But no one will ever solve that. Uh, uh, it's a single dimension problem that we have of not having another dimension where we can test all these things in a different version of Facebook that will be working out one anyway too. Tinfoil hat.

Speaker C: No. And what you're saying, uh, I really believe there is going to be always misaligned incentives between advertiser and publishers. The nature of the market and of course you don't want to give the publishers even the wallet gardens the ability to do measurements and uh, you don't want to outsource that completely. Although maybe balance provide you some things.

Speaker A: It's the last bastion of control we really have is to moderate the measurement signal they get so they know how hard they should be pushing otherwise they're left to their own devices.

Speaker C: That's right. And uh, of course in the affiliate markets there is platforms that do a third party neutral that do the measurements exactly for that. It's kind of the extreme case. But in every other media um, and publishing channel there is always these misalignment incentives.

Speaker A: Yeah. Gosh. Uh, you know, so I'm going to just take us way into the future. There was a paper published, what was it Jim? Two years ago. Nnn. Do you remember?

Speaker B: Oh yeah.

Speaker A: Ah, the next generation neural nets. Uh, I don't know. Um, look at if you're familiar with that paper.

Speaker C: Very familiar.

Speaker A: Okay. Uh, is that where we're headed? Is that a reality as we move this world?

Speaker C: No, no. But I do think we are adding in slightly 10 to 20 degrees angle compared to that. So that, that paper really showed that you can use neural Networks to train MMMs. And actually what they really did, they what they showed that they have advantages on and it's really good is that um, ah first uh, you can use neural network to represent causal structure in the data and they have a way of extracting that. Actually there's a couple of follow up papers exactly on that topic. And then uh, the other one is that you can use it through your genius data sets like Creative for example, embedding them into the uh, mmm which is very hard to do today with traditional Bayesian models. And so I do think that that's gonna stay. I don't think that that classes of models from that paper is gonna be the classes of models that's gonna be in the longer term used. I do think that there is yet another direction of research, maybe 20 degrees angle which goes in the direction of foundational models for uh, mmms that's gonna start to appear uh, in the longer terms. And the way that they work is that um, today um, building uh, mmms is uh, still freaking hard. You can use agents but you have all the um, it's still like uh, a very computationally hard problem. And you need to refit your models every now and then on new data sets. While the concept of these foundational models are neural networks that are built on very very large data sets, some of these are entirely synthetic. They are not um, data sets that are of uh, advertisers. And they're trained into a way that when you have the next data sets and you want to apply it, you don't need to retrain the model, you just literally just apply it to your data and it gives you the most plausible causal structure that represents your data and gives the same results of an mmm without you having to do anything but just write a single line. And that's amazing because it's gonna really kill the barrier to entry for the entire media mix model industry and change the media mix model industry. Now these foundational models exist uh, in really a lot of industry nowadays. Like if you're looking in time series forecasting, um, you have things like Kronos from Amazon or um, chat. It's called ah, Time GPT from Nixla. So there is a series of um, very well established foundational models for time series. There is virtually no reason why somebody who's gonna figure it out a way to do it for uh, mmms. Beside the fact that uh, you know, um, somebody has to do the research.

Speaker A: So maybe an angel will do it. Uh, who does?

Speaker C: I mean we are already working on this.

Speaker A: Yeah, I guess that's the question is like will someone have to do it or will agents start to figure it out? And I hear about uh, anthropic talking about this idea of what's about two years until we see agents, uh, uh, or the ability to self iterate. Uh, and I guess you see that happening in our space.

Speaker C: We have agents actually working on this problem exactly as we speak.

Speaker B: So I need to crack the whip on my age and he's just uh, over here doing nothing. I haven't given him access to anything. I'm still nervous.

Speaker A: Yes, the agent difference chasm is a big one.

Speaker B: Uh, well, I hate to be the person to end the party, turn the lights on, turn the music off, but we uh, have to start wrapping up uh, this has been a great. We could keep talking about this for a lot. In fact we didn't even cover like half of our outline or show notes of things that we wanted to talk to. But uh, but uh, I guess we, we kind of went rose colored glasses at the end. Is there like one thing that you would tell people to be aware of? Uh, if they're getting into this whole agentic mmm world like one, one key tip of like watch out for this because we've been there and done this and it's going to be a problem.

Speaker C: Um, beware of. I don't know, I think like um, and want to scare people or be aware of. What I would like to tell them is something like just try break stuff. Um, if it doesn't work, try to understand how the system works. If you haven't used AI in your world life, um, if you want to. For example, I went through the exercise of setting up my OpenCloud recently and my assistant Harry helps me with a million things. And um, I was very scared at the beginning of giving access to all my digital life. But then I thought it through and I found a way to do it in uh, uh, at the level I feel comfortable with. So it doesn't have access to my personal email or to my work email. It has its own email set up uh, that it can use and I communicate only through certain channels to it. So find a way that you uh, can experiment with these tools in a way that feel comfortable to you and thinker with it. Because what's going to distinguish people that thrive in the world with AI with people they um, don't. It's the amount of repetitions they get with these new technologies and the ability to understand where these new technologies are going very very early. So track. That's what I would want to say to people. And by the way we have an agentic data science course if you're interested in too. You can find it on our website. That's a good way to get started if you want.

Speaker B: Absolutely. And uh, Luca, we do like to end our podcast with an incremental insight. Uh, I think you might have mentioned this earlier but if you want to go ahead and give a shout out to uh, the Discord community again if that's your.

Speaker C: Oh for sure. Sorry. There is a ah, Discord um, channel a Discord server which we call Decision AI. Um, it's fun if you want to uh, get there. There's a lot of people interested in the topic of agentic data science. They're exchanging messages topics continuously. You um, get support around our software also if you can ask questions around our software stack and open source ecosystem. So welcome everyone to join. As soon as you join you're going to meet Damon uh, which is our own version of Operco. This was a. We're going to open source it also very soon it's uh, reimplementation rethinking of OpenCloud. It has some advantages compared to uh, OpenCloud that we are going to highlight. So I guess we are going to link it um, into the podcast. Everyone is welcome.

Speaker B: Absolutely excellent links as they say.

Speaker A: Indeed. And uh, if you're into traditional forum spaces like I am, I will also know that the uh, what is it Discourse PYMC IO um has also some just really great forum discussions around some of these topics where um, not that they're unstructured in discourse but it can be a high velocity of people chiming in and so forth versus more of the uh, traditional read through like a story type of approach. And I've seen some really interesting conversations on there of late around things like uh, agentic management. But uh, you know, just as we're going through today's conversation, as I've been thinking and I know there's such an interesting divide in, in the world right now of the folks who are really leaning in and wanting to embrace AI and those who have a degree of aversion, I think I saw a study recently saying um, at Google, Google looks like John Deere apparently in terms of the employees there who are utilizing AI, uh, which I thought hey, good for John Deere. Uh, but it is this interesting moment in time where it was 20 to 25% of the organization. You invented the transformer. How do you not see greater adoption? I think there is a very real degree of aversion. If your aversion to agentic data science is a fear that it might take your job, uh, well then maybe it's time to reevaluate what is your job in the future and begin to measure up.