051: The End of Interesting: AI in Experimentation with Deborah O'Malley

Speaker A: Welcome to the Intended Consequences podcast. I'm Brian Massey, the conversion scientist. So I've been uh, had mixed emotions about today's interview. Deborah o' Malley is with us and she's going to be talking about AI in experimentation and uh, in doing a little research on her. Uh, she is one of the first voices that has started to say the sorts of things that we've been looking very closely at here at Conversion Sciences and, and that is uh, AI's involvement in experimentation or is it AI's taking over experimentation? I think we're going to have a very interesting practical and philosophical conversation on that. She is a leading voice in the field of uh, AI powered experimentation. She's the founder of Guess the Test, which is uh, an experimentation resource that allows you to go in and see case studies of A B tests and even test your ability to pick winners in that. She's been involved in conversion optimization and digital strategy ux, um, and she bridges that data and intuition kind of to help us run smarter and more human centered experiments. So it's very interesting that she is um, well so focused on AI. Currently she's looking at AI and understanding how that is reshaping the landscape. Uh, we're going to be talking about Agent first optimization and even emotionally intelligent design. And the goal is to use these tools without losing our humanity or our jobs. That is um, a bold and I think ah, difficult prediction to make. So Deborah o', Malley, I look forward to hearing what we have to talk about.

Speaker B: Thank you Brian. It's wonderful to be here and to talk with you. I'm really honored to be able to speak with the conversion scientist, that's me actually.

Speaker A: Uh, we have a number of them and they're all very good at what they do. Okay, um, I think you know, to start off with ah, just a quick history on how you ended up in conversion Optimization because as everyone I talk to who does this, including me, has some of the strangest journeys. How'd you get here?

Speaker B: Yeah, mine included. So you know, if I go way back now, this is way back when I was a kid I watched a show on TV called Muppet Babies. I don't know if you remember it, but I watched it and the episode was about finding what you could do on the palm of your hand. And at about five years old I was taken by this concept and I looked at my palm and I examined it and I, I just wished it would appear to me what I was meant to do. And I struggled with that for many years. But fast forwarding A little bit in grade five, we had to do hypothesis based science experiments for school. And while other people were pouring apple juice on diapers and seeing what was most absorbent, I cut out different colors and sizes of construction paper and I asked people what they saw first. I was fascinated by finding out what people noticed and attention to advertising and fast forward many years later in a very serious route and I actually ended up pursuing a master's of science in eye tracking technology where I officially ran eye tracking experiments to see what people saw first on digital ads. And everything came together in full circle for me when I ended up doing that master's after, uh, the master's degree, I ended up working for a local user experience firm. I live in Canada, in Ottawa, which is a government town, so running large scale usability studies for the government of Canada. And as part of my training to understand user experience, uh, one of the people I was working with said, you have to get on this site called Witch Test one. And I followed Witch Test one and I was amazed by it. I loved it. And several years after that they had a job posting for a content creator and I applied for it and I got the job. And that's what launched me into the field of A B testing. So I went from eye tracking to user experience to CRO and a B testing. And I really found my milieu within CRO and AB testing because it combines the creative and the analytical. And I've actually done a brain test And I'm basically, um, 50, 50 left brain, right brain. I think I'm 49% one side and 51% the other. But

Speaker A: what?

Speaker B: Uh, it was like an online one.

Speaker A: Oh, okay, yeah.

Speaker B: Um, and so the synthesis of the left brain, right brain, creative and analytical, just really suits me in my personality. I like to dig into the research, I like to understand things, I like to unlock keys. And I've always been fascinated by human motivation and what gets people to take action and, you know, ultimately convert. Uh, so I worked for Witch Test one for a number of years. Unfortunately, really, really sad story, they ended up dissolving the company after every Rand that went wrong. And people were coming to me as a key content creator saying, deborah, what happened? I relied on these case studies for teaching my classes or for testing inspiration and ideas. What am I going to do now? So I decided to independently create what was formerly which Test One now known as Guess the Test. And I started literally from the ground up, uh, uh, a real exercise in branding and rebuilding trust and that kind of thing. And I've grown the site over the last five years to tens of thousands of subscribers and a uh, community of engaged experimenters that I try and help through a B test case studies. So that takes us through to today. But here we are today talking about experimentation and really the viability of it as a future career path for experimenters when everything, all signs are pointing to it imploding from the inside out with AI and the potential takeover of AI and the automation of uh, what we toil and use all this brain power to try and uncover and unlock AI can do in an instant for us. And so the question that you and I are both facing is where do we stand in this brave new world and what kind of job opportunities are there going to be and how can we still position ourselves as meaningful and relevant against the AI, uh, that seems here to take us over?

Speaker A: Mhm. Well, I hate to ask this question because it will um, almost certainly be obsolete within a few months of the launch of this episode. But what is the state of the art now in terms of AI and experimentation?

Speaker B: Yeah, so it's changing so, so, so quickly. But as of right now, so let's call it May 21, 2025, AI is in its infancy. Uh, experimenters are using it, I think probably people on, you know, average. I just saw a stat, I don't want to misquote it, but something like 60% of people are using chat, GTP and agentic search with, in their normal day to day tasks. Um, and people are using it to help create content, help create LinkedIn posts, um, get ideas for things, that kind of thing. But it's just in its infancy and its capabilities are just being scratched at the surface. There is a stat that I will just pull up here so I don't get the numbers wrong. If you just give me a second. That things are, the uh, power of AI is doubling everything 3.3 months. So that means in, in a year it's going to be 16 times more powerful than it is right now.

Speaker A: Doubling every 3.3 months.

Speaker B: Every 3.3 months, I believe it. So that means after a year, in two years it's going to be 256 times more powerful. And in five years, which is not that far from now 2030, it will be a million times more powerful than it is right now. So it's on this tangent that's just exponential. And even small changes in Dublin are going to lead to massive leaps in where we are. So the world as we know it today in 2025 is going to look Radically different I expect in terms are

Speaker A: these stats for kind of the, the generic LLMs, those that are maybe on the path to um, uh, generic intelligence, artificial generic intelligence, AGI or. Yeah, probably not even considering if I take a language model which the train, the training is becoming more and more powerful using synthetic data and have it niche in so like an agent that is just good at reading analytics and pulling inferences from analytics, uh, for humans for as long as that's even relevant, we'll talk about that. Um, yeah, there's, I imagine that those systems will probably show step functions um, even higher and faster.

Speaker B: Very possible. Yeah. This step uh, specifically is by OpenAI and it's a projection on computational power. So um, it's looking specifically at you know, how powerful these agents are going to be or the GPTs are going to be. But I think we can extrapolate it and apply it to just growth in the industry in general.

Speaker A: Mhm. Well, um, you know a lot of folks listening may not really understand what the components of conversion optimization are with the, the manual steps why they pay us the big bucks today. Maybe it makes sense to very quickly kind of um, list out the components of the things that we do and um, just as a kind ah, of level setting then we can start talking about which, which pieces of those are going to um, be automated the fastest or something like that.

Speaker B: Sure, yeah, definitely. So as I say there's essentially eight main tasks that a good experimenter like conversion scientists should do and would do. And basically most experimenters if they're taking a strong data driven approach, follow this eight step framework. So we typically begin by using data, whatever it is, analytics data, heat mapping data, customer experience data, to generate ideas and recommendations. Pardon?

Speaker A: All right, eye tracking data as well.

Speaker B: Eye tracking data, there you go. Yes, to generate ideas. So you know, what are we trying to optimize here? What are some good ideas? What's going to really move the needle? And then we take those ideas and we formulate hypotheses and we hopefully if we're formulating a smart hypothesis, we're creat if then statement that has a measurable outcome. So we're not just saying it's going to move the KPI, we're saying by doing this for this audience it's going to have this effect and this impact on our audience. We can then measure m that hypothesis and then we take all these great ideas that we've generated and we start to plan and prioritize them and hopefully good experimenters are using some kind of framework or system like PI or ICE or the PXL framework to as quantitatively as possible map out the ideas and saying, these are the ones that rise to the top. These are going to give us our ah, best bang for our buck. Let's try these first. Then with those ideas in mind, planned out, probably roadmapped, we say, okay, well let's design the experiment and we maybe wireframe it and then work with designers to actually create a polished high fidelity design that we can then send to dev and actually implement on our testing platform. We then, um, run the test, we execute it on our testing platform and we hope and pray that it's going to work. And once we do so, we get data coming in and we analyze that data. Now, hopefully we're running trustworthy tests with large sample sizes and good power and we get trustworthy data that's actually meaningful and we analyze that data and we go, okay, yeah, we got a winner, or maybe we got a loser, or maybe we got insignificant data. But we figure that out and we crunch the numbers. I know conversion sciences does know rigorous job at crunching the numbers. And then hopefully most of us are implementing the findings. Not all of us do. Sometimes we get winners and they sit there. But hopefully we're implementing the findings and we're keeping an eye on things and tracking things over time and going, yeah, this is increasing our overall whatever it is, uh, revenue or you know, whatever benchmark we're looking for, overall conversion goal. And then we look at all of that holistically and we continue to optimize on the results. So there's this process of continuous optimization where we say, oh, well, we got a loser this time, but let's see if we can eek out a winner out of this idea or great, we got a winner, let's optimize on that and make that lift, uh, even stronger. And so we go through this process, this eight step process and it's very, very manual. Every step along the way is intense computational power from these things inside our heads here. A lot of brain work. Uh, I really specialize right now in sort of what I call the front end part of it, digging into the data, extracting meaningful stories out of the data, coming up with really tangible test ideas that clients can run, and then formulating hypotheses, doing the wireframing and helping implement the test so that you have winning test ideas that create large conversion lifts. These things.

Speaker A: I was just going to say, you know, some of the evidence that the human, the human interaction is really important is that if we give one of our conversion scientists too many clients, it uh, reduces the amount of time that they're able to spend, um, thinking, evaluating, analyzing and test velocities will inevitably go drop down. They will, they will drop. We're not testing, um, through ideas as quickly. So um, there's uh, uh, that's the kind of the metric that we use. How much of the traffic are we able to test on at any time?

Speaker B: Right, yeah, for sure. Yeah. So we're limited by our own human limitations, how efficient and effective we can be at each of these tasks and also by the individual and their expertise. If we don't have a person who's generating good ideas from the start because they're not data driven or they're not informed, then the whole process fails. So there's many fail points along the way and loose links in the chain that are prone to human error. So you know, as conversion scientists, we've developed a system and a framework that works over time and we have debates within the industry about this area and that area. But for the most part we've been able to eke out a system that helps, you know, ourselves or our clients get gains that are sustainable and increase revenue over time.

Speaker A: Well, and arguably the whole system is really designed to get our, our biases and um, superstitions and stereotypes out of the mix. We're not good at predicting what will win. And so this whole process is, you know, built on the scientific method which was designed to get us out of the mix and um, yeah, data and behaviors as our sole guiding star.

Speaker B: For sure. Yes. Our uh, North Star metric. And that's absolutely the way in theory we would hope that it's done. But you know, starting from the very process of generating ideas we all joke about the hippo in the room know if the hippo pushes ideas on us and we can't really fight back or the client insists, no, this is what I want to do. You kind of have to accept it. So your ideas are fallible to, you know, human, um, um, thought process and opinions just from the get go and then every step along the way prioritizing test ideas. You can be as quantitative as possible in the process following these frameworks, but it's still very susceptible to qualitative analysis of uh, oh, I think this is going to reach a lot of traffic or oh, I think this is going to have high impact or oh, it won't take too many dev hours to do this. You don't actually know until you launch the test. And even something as Looking at the data and a test design where you're trying to figure out the minimum detectable effect. Oh, I think this is going to have a 3% conversion lift based on historical data, but it may not. It may have no conversion lift at all or it may have a 20 conversion lift. We don't know anything detect anything. So within each step of this quote, scientific process, it's very fallible and subject to human qualitative opinion. We're influencing, shaping it each step of the way.

Speaker A: Yeah, well, and you know, I've heard the argument that the A's are trained on human data and so they have their own biases. But that's, that's missing the point. We um, can ask a, a language model to pretend that it has stereotypes or biases, but they're very good at uh, stepping outside of that in any particular decision. Now, uh, one of my favorite things to say is that if you're not getting good results from using AI and almost anything that you're doing, then you're probably using it wrong. And um, I'm very aware of that because there are many things, you know, I'm, I've been playing with a number of the tools and trying to see exactly how quickly we can automate these steps because we're an agency. So uh, if we can automate swaths of what the conversion scientists are doing, then um, we can give them more clients and allows us to grow with the excellent crew we've got. Um, but everything, you know, the, the places where I've seen probably the most interested in or the most success in is there are some systems that are ah, pretty good at doing exploitative analysis. So heuristic best practices, um, you know, change the call to action, make the headline bolder, make the fonts bigger, increase the contrast, add white. You know, things like that that are really primarily designed to increase conversion rates where um, we are holding our banner. I guess the hill we're on is the exploratory test where we're trying to understand something strategic or something some gain an insight of the visitors. Um, beyond that I have, I've had, I've been disappointed even in the data analysis that I can get out of the AIs I played with. Where do you see the first places? Um, where are the first places where you see A.I. uh, making an impact in that eight step flow. Um, and then we can talk a little bit about why that, if that even matters.

Speaker B: Yeah, yeah. So it's a really important question and I think it's also very important to say right now, as of timestamp 05-21-2025 it is not there yet. These uh, postulations are uh, conjecture based on data, based on research, based on trends of where I suspect things are going. But I am crystal balling it a little bit because you're absolutely right right now. The output that you get from AI when you try and get it to analyze data, look at trends, those types of things, it doesn't always point you in the right direction. It can lead you astray. And so we can't really trust it right now. It hallucinates, uh, it's going to tell you things you want to hear. There seems to be an inherent bias in it. You can kind of game it to tell you this or tell you that based on past interactions. So uh, it's not completely objective right now. As things develop I suspect that's going to change and it's going to become more objective really over time. And in that sense I think uh, the whole eight step framework is going to be totally flipped on its head. So rather than starting with generation generating ideas, AI will simply predict insights right away. It's going to have a repository of past tests that work and it's going to say, based on what I've seen work in the past, why don't you start with this test idea? Here's your best bet, uh, for starting with your control.

Speaker A: Try this then it may not even have to look at data and trends. It may just be able to statistically predict where uh, a site with this structure in this format where it should start exactly.

Speaker B: And there's a testing platform called AB Testing AI and they started a few years ago and it's the closest that I know to already doing this kind of thing. And what it does is it starts with the control as your last winning version and it iterates and optimizes on that uh, control automatically. So you can. Right now it's limited to copy and images, but some of the most important

Speaker A: things on any page they are.

Speaker B: Yeah. And so it can automatically suggest multiple variants for copying images in a multitude of ways. Basically you can have a multivariate test design.

Speaker A: So it's kind of Bayesian approach to, to copy and images where you start with a, uh, predecessor, um, uh, a conversion rate if you're measuring that or whatever it is you're looking for and then you build on that and that helps you get closer and closer to what the ideal is.

Speaker B: Exactly, yeah. And it just keeps iterating automatically until you get, you know, the highest optimal Lift possible. Basically it keeps going with it. Uh, okay, let's try this version, let's try this version, let's try this version, let's try this one. But it happens automatically. You're not having to run a test, sit there, wait for it, analyze the results, see if it's statistically significant. It does it all for you.

Speaker A: And this is, it is, it is launching tests on traffic.

Speaker B: Isn't is launching on traffic, but it's, it takes out that manual kind of busy work of having to do all that yourself.

Speaker A: So just, you know, the best practice in a B testing is to launch the test and then wait until your estimated end time, end the test and then see what happened. Uh, you literally are going to be able to push a button and um, come back once a month and see what, see how high your conversion rate is.

Speaker B: Yeah, yeah, for sure.

Speaker A: Okay, well that's scary.

Speaker B: So no more generating ideas. Um, what I do and specialize doesn't, uh, matter anymore. AI will do this for us. Much better, probably creating tests. AI is going to do it. It'll automatically whip up the variant, whip up the alternative copy image, whatever, format it for you. Boom, it's formatted into a test. No need for designers, no need for developers.

Speaker A: I have not been disappointed in. And that's AI's ability to code. Wow. Wow.

Speaker B: Yeah, yeah. And there's people at Anthropic who are coders. I know them personally and they say within a year AI is going to be coding better than you know, these are the top of the line coders in the world and they say AI is going to be coding better than me.

Speaker A: Well, I'm a, I was trained as a conversion, a computer scientist actually. Uh, I'm more happy today that I'm not in that industry anymore. Not, not directly.

Speaker B: It's also one of the industries where, you know, you're going to have to really adapt. There's, there's a lot of them. There's a lot of jobs that are going to be gained through this if you pivot appropriately. But a lot of jobs that are going to be lost. The projection is 78 million jobs will be lost.

Speaker A: 78 million in what time frame?

Speaker B: Within the next five years. But let me pull up the stats. So I'm not misquoting here. 170 million jobs will be created out of AI and the revolution and evolution of it.

Speaker A: I'm curious, where are those, where are those jobs? Um, they're clearly not going to be, you know, engineering prompt does it.

Speaker B: Yeah.

Speaker A: So these jobs are going to what those jobs are going to be, I, I think.

Speaker B: And we can look at them individually specifically if you want, but a lot of them are going to be overseeing the AI that is automating the process for us. So while AI may be, for example, creating test variants for us, we're still going to at least initially want to have that upper hand and say, uh, let's just vet this and make sure this looks good. And we actually want to launch this test for the client. And so we're going to all going to become managers.

Speaker A: All become middle, middle level managers of AIs.

Speaker B: Yes, yes. Our team is going to be a whole bunch of AI agents that will be managing rather than a lot of people.

Speaker A: Um, I hope they have nice personalities, hopefully.

Speaker B: Yes. Yeah. So I, I think I said 78 million. It's 92 million jobs that will be lost, 170 million gained. That's the projection.

Speaker A: Okay. Okay.

Speaker B: So, you know, it's not all Dire Straits, but there is that disparity there, which I believe is 78 million, if I do the math properly. And, uh, you know, what do we do about those 78 million jobs that end up being kind of, you know, in no man's land?

Speaker A: M. Um, that's a lot of. What do you think it is so

Speaker B: open to work on their LinkedIn banners. Yeah.

Speaker A: You've said that empathy is going to be our, um, our superpower. It's going to be human. Uh, empathy is going to be our strategic advantage, I think is what you said.

Speaker B: Say more about that, I think right now. Yes. In this current paradigm that we're in, where we're just shifting to AI, uh, now I think there's three factions or three camps of people out there. There's people like you and I that are forward thinking and looking at this, looking at the big picture down the line, five years, 10 years down the road and projecting and going, what's going to happen? Where do I fit in? How can I carve something for myself to still stay relevant? And uh, we're the people that are going to be able to, you know, figure it out and go along. There's people who are just kind of saying, yeah, AI is going to make an impact and okay, I'm willing to accept it and okay, maybe I should start thinking about this. And then there's a whole camp of people still that are saying, yeah, AI is happening. M. But my job hasn't changed. I still continue to do everything exactly as I have already, so I don't need to worry about it. Right now, and those are the people that need to worry. They're the ones with their head in the sand and they need to get their head out of the sand because the storm is coming in ferociously and it's going to just catapult us all into this whole new world. So the, um, shift towards having a really empathetic, intuitive, human centric approach, I think is a first step that all of us should be embracing, because that's initially where we're going. We're going into a world where AI is going to start automating some of these tasks that we're doing. It's going to make it more efficient, more effective, maybe at first more error prone, and we'll need to be vetting the work that it does, but, but ultimately those things will get refined and likely will outperform what a human can do as we work towards that. What AI is not good at now and likely won't be ever good at is really understanding us, uh, people, because it's not a person, it's a computer. And what people have is emotion, empathy, intuition, gut feeling. These primal things that make us human, these instinctual things that set us apart from any other species. They're what really are our consciousness. And that is how we can understand and appeal to other people and motivate them to convert and persuade them. And all these wonderful things that we do as marketers is by understanding their needs, their pain points, what keeps them awake at night, all these things. And we can appeal to them through stories and emotions and motivating calls to actions and that type of thing. And AI will never be able to generate that because they just don't understand how people think. No, they can attempt to replicate it and get better and better at it, but ultimately that's what gives us the unique power. It's our strategic advantage right now over an AI.

Speaker A: It's very optimistic, but at a doubling every 3.3 months, and that's probably going to accelerate. I, I think I would, I'm a little bit of a catastrophist, but I, I think that we'll be able to simulate those things sufficiently that we, uh, we won't be able to tell the difference.

Speaker B: The.

Speaker A: Ah, yes, Turing singularity.

Speaker B: Absolutely agree with you. AI likely is going to become sentient, which means it will have its own form of consciousness before we know it. Probably, you know, I'm just spitballing here, but within the next ten years, um, and it's going to be taking over in a world that we can't even Imagine right now having human empathy. And that human connection is not going to suffice in a world where AI is becoming increasingly prominent in the work that we're doing and in, you know, just communication and outreach. And the trend that's happening right now is not human to AI communication, but starting to go towards AI to AI or agent to agent communication where agents are actually physically talking to each other through their own beeps and bloops. Language that we can't even understand.

Speaker A: And yeah, you showed that we'll have a, we'll have a link to a rather eye opening video in the show notes.

Speaker B: Yes, definitely. As optimizers we need to be looking at that and going, okay, this is no longer just about us and human connection. This is about figuring out how agents are optimized to speak or to talk or to motivate other agents that are doing these tasks. For us, that's an entirely different ballgame and paradigm shift. And so we need to be thinking not just about emotion, which is maybe the first step in the process, but how we optimize for agent to agent communication or agentic AI.

Speaker A: And this is something that we've already created audiences for in analytics. And that is the number of visits that are coming from uh, agentic search, um, a little bit harder to do with Google because they mash it up. But one of the things that um, we're going to certainly see is a drop in the number of humans that are coming to our web pages. Um, and that's something we've got our eye on. Um, it's not going to be too long before well we're looking for the marker when 50% of the traffic to our customers websites are coming from agents. And when I say not that long, we're used to thinking like oh, it's technology happens really fast. So that might be five years, that might be 24 to 18 months. Um, I think there's probably going to be a company that comes out with the Facebook of agents. So it makes agents available to the general public. It's easy to use, it's fun, it's gamified. And uh, suddenly there's going to be this burst of people that are asking their agents to do all this stuff for them. Um, and it's going to leak into the business world. So people looking for conversion optimization or ask their agents of uh, you know, who is uh, who should I for my particular company, uh, consider for conversion optimization and it's going to come back with a single answer. So um, the, there's a standard right now called LLMs txt. It's like the robots txt or the SML or uh, the sitemap xml, um, files that go on your website that tells agents what your site is about. It essentially rewrites your website, um, with a little bit of notation but it's, it's all in English at least for now that's going to start facilitating this and I think that's going to be the, the harbinger of uh, when web pages go away altogether. Um, you think web pages are going to disappear and use an example of shopping. Um, you make a pretty good case tell. Why don't you share that with us?

Speaker B: Yeah, I do and it's a really scary thought as a person who's dedicated the last 15 years of her life to optimizing websites and I do believe that websites as we know it today are not going to exist. And the reason why is because agents are going to be talking to agents that communication is going to bypass any websites. They'll likely be doing so in you know, a browser if you will or GPT like chat GPT and what we're seeing already, this is current day, this is not speculation. I will just pull up the stats. So I have it correctly here. Um, Chat GTP is going to start allowing Shopify purchases directly within the app. No needing to go to the merchant's website anymore. You can just shop within Chat GTP on Shopify sites. AI agents already are being booked for travel. They're being uh, asked ah to sift through ads and find the best travel booking deals. That's just going to increase. So you want to plan a trip, get your flights booked, get your car rental, get your accommodation or all done by agents. You won't have to do anything, maybe oversee it and press the buy button

Speaker A: and like a window seat, an aisle seat, um, um, little things like don't sit me over the wing because I like to look at the ground. You know it's uh, I think about all the subjective things that could foil a plan like that and I honestly have foiled my attempts to, to let an agent book travel. For me it's a personal thing. Um, right. It's using um, using uh, ChatGPT for a long time. It's amazing how much it learns about and does callbacks to previous things I talked about weeks later and I think that's going to get worked out.

Speaker B: So yeah, yeah these, these GPTs or LLMs, whatever it is are going to be learning you. They're going to know you intimately and as you say they really already do. They say things in their memory. They're not shy about telling you, memory updating. And uh, they're going to know your personal preferences. They're going to know Brian wants a window seat. He always sits in row 18. And you know he likes uh, the, you know, whatever American Airlines. And they're going to know these preferences and you're just going to say yes, confirm.

Speaker A: And since we've got an agent doing it, we can offer things like he wants to sit at row 18. It's not available on this flight. Uh, could that, could your agent talk to the agent, the person that is sitting in that seat and seeing if for an extra $25 they'd be willing to move to something else and then charge me $40 extra for that seat and I would be like yeah, I'm willing to pay up to $50 to get the seat I want. And that's what. And it's going to en many different ways of um, booking things than we've, we've thought of. Now there are two kinds of shoppers that we identify very broadly. Um, and that is the transactional shopper. And then there are those people that shop because it's, it's relaxing, it's like watching tv.

Speaker B: It's retail therapy.

Speaker A: Stereotypically it's how a man shops and how a woman shops. You know a man walks in and gets what he wants and a woman is taking in other ideas and possibilities for future reference. Um, do, do we think that those experiences are going away?

Speaker B: Well what I project is that those experiences are going to be catered to you as the individual and the agent is going to know your shopping behavior intimately and present you with the options that you want to see.

Speaker A: So it'll recreate that scrolling and searching experience in, within the agent space?

Speaker B: I presume so, yes.

Speaker A: And that way every e commerce site will look the same because it will be, it'll essentially be creating category pages and product pages for you um, to um, to interact with. But the way you want it, the color button you want and the language that's on the calls to action and uh, uh, the size of the images and uh, yeah I can you know,

Speaker B: on the fly, hyper personalized and you know we, we've talked for many, many years about personalization and getting unique dynamic web experiences for each visitor. I believe that's finally becoming a reality where the website, quote website you're looking at is much different than the one I'm looking at. It's a hyper personalized structured data Snippet. It's not a website, it's, you know, showing you exactly what you want and need to see at the right time. Um, exactly. Oh, I'm sorry, just what you need to convert.

Speaker A: So if we're going to do something on the web, it's not going to be pages, it's going to be experiences. Um, uh, we're still going to have to go to an app or something like that to, to play Candy Crush. Uh, um, but for the most part we're going to be. If we're coming to the web, if it's not the agent doing the visiting, it's got to be, it's got to be coming for an experience. Um, maybe not necessarily just for some sort of uh, transactional result. Are there any um, caveats for the folks in the B2B world? So I think about choosing uh, HR management software solution for your business.

Speaker B: Yeah, well, I think the uh, the idea of like lead gen and book a demo is becoming a very quickly outdated model if people are not the primary users of the web interface anymore. Is an agent really needing to book a demo to see how a SaaS platform operates? Probably not. The parameters are going to change an awful lot, uh, and lead generation is going to change an awful lot because you're not going to be targeting leads, you're going to be targeting agents. So I believe the whole model is going to need to be rethinked and likely a lot of these services and solutions are going to fall by the wayside because agents are going to be taking them over innately anyway and probably. And you know, now I'm really crystal balling it here, but the way that I believe things are shaping up is we're actually not going to be having uh, screens in front of us in the traditional sense of the screen where you and I are looking at a computer screen right now or we go on our mobile and we look at our screen. What the technology companies seem to be going towards, and this is based on announcements from Apple, Google and Meta, they're all going towards these glasses, these, you know, smart glasses that we put on and those are going to have the capability to do the surfing for us, do the recording, do the GPS maps on our phones, all these types of things through these glasses that we wear. And um, the whole browsing experience paradigm is not going to be sitting in front of a computer using a mouse, typing things in. It's going to be voice search, it's probably ultimately going to be neural linked and it's going to be you know, initially the interface will be glasses. Likely as we move into the future it's going to be holographic screens. I suspect that's total conjecture, but so, you know, we need to be rapidly adapting how we think about not only how people are going to use things, but how agents are going to use things. And then how people interact with the agents becomes the next question and the interface places that we do that. So if we're stuck here, back here optimizing websites and call to action buttons or button callers, we're not thinking about, you know, the glasses or a holographic screen experience that an agent is running on our behalf. And so there's this incredible widening gap, uh, between where we are now and where things are likely to go very quickly. And outdated models like a B2B SAS lead generation book, a demo site just is not going to fit in anywhere. We need to be way more progressive, uh, in the way that we think about these models.

Speaker A: Yeah, well in the near term it's going to be uh, people talking to language models. And um, we're designing an experiment for our business where we're going to have a language model, um, option so you can choose if you want to ask questions or if you want to see web pages. We love the web pages because we have so much more control of the length of the conversation because we can show the features and the benefits before we talk about pricing, uh, things like that. The next step is going to be uh, where our uh, web properties are talking to agents. Um, and those, those, those standards are nascent. But right now it's English. English is the API. Um, now these are things that as I said at the outside might be three years out, but if that, when that killer agent app launches, uh, we don't want to be flat footed. What do uh, marketers do today to plan for this? We'll call it a 12 month to 36 month horizon, uh, in which everything is going to change outside of three years. I think that's getting a little, that's too far out.

Speaker B: Yeah, yeah. So first to that point, you know, it's, it's not years we're talking about, it's months we're talking about. And Sam Altman of OpenAI said that a few months back, he said this is a time frame of you know, 18 months, which is three years, but it's not decades of years. We're talking about very short snippets of time and we need to be thinking about that and adapting quickly. So number one, what's an experiment you do focus on human empathy, human understanding, human intuition. I think these initially are going to be our guiding forces that help center and align us into what's important and where we need to be really understanding the human. That's phase one. Phase two is agent to agent communication and understanding that everything that we learned about people goes out the window as soon as it's for agents, because they're not humans, they think about us differently, they think differently than us. Um, and optimizing that agent to agent communication in terms of the structure of the data, the way things are formatted, the language, um, that's used, all these types of things. And then phase three is once agent staging communication becomes the norm, how does the world look and where do we fit into this world? And that's still a gray area. A lot of crystal balling is needed. But I suspect you it's thinking about not websites, not, um, screens in front of us. It's thinking about whole new paradigms and ways of actually behaving. And there always is going to be optimization opportunities. There's always going to be need for experimentation because within each of these shifts in terms of our reality and the way that humans operate in the world, there's going to be need to test and refine and find ways to, to extract the best, most optimal way of acting out of it. And therefore there's opportunities. But you can't get stuck in one thing and say, I'm just going to do this, dig my heels into the ground. I'm really good at optimizing buttons. That's what I'm going to do. It's not going to work all the way through this progression. So being flexible and adaptable and really embracing that experimentation mindset to say, what can I do to optimize myself in this new world that we're going into and looking at that and finding a place that you fit best based on your skill set.

Speaker A: Well, you've done nothing to make me feel any better. I'm still as terrified as when we started this conversation. I knew that would happen. But thank you so much for coming and sharing.

Speaker B: I tried to end on a positive note.

Speaker A: In the end it will be, it'll be a very different world with different roles. But, uh, I'm actually excited to be alive right now, so I'll end on that.

Speaker B: We're very fortunate.

Speaker A: We really are. Thank you so much for spending time with us.

Speaker B: Yeah, thank you. It was great speaking with you, Brian.