Marketing and the CTO - Collaborating for AI Adoption

Speaker A: Whether you're making a marketing campaign or a, uh, sales strategy or what have you, you should be using these tools. But, and this is really, really critical, you must understand what it is doing because if you don't, it kind of defeats the whole purpose. I want to encourage people to use it as much as possible. Now, as a matter of fact, like, if you're not using it, that's a problem.

Speaker B: Welcome to the AI Edge Podcast for Enterprise Marketers, a show dedicated to sharing insights, strategies and experiences from a group of who have successfully implemented AI solutions in a large enterprise B2B software company. Specifically when the context of global marketing and how that effort can connect to sales, IT product and the rest of the business. I'm Eden Porto De Leon and I'm joined by my fellow host, Jessica Ria. Jessica, how the heck you been doing?

Speaker C: I'm good. Happy Friday. It's full swing summer. Kids are home. I'm questioning my life choices. All is great.

Speaker B: Aren't we all, Aren't we all questioning our life choices?

Speaker C: I mean, working from home has its ups and its downs.

Speaker B: Yes, it does. It does indeed. So we have a, uh, guest I'm super excited about. This is somebody we've worked with four years, done a lot of really amazing stuff. He's still doing a lot of amazing stuff. And this is Kit Colbert. Kit is the platform cto. Do let us know right before the show. So I updated his bio. Get some platform CTO at Invisible Technologies, an operations innovation company that transforms how companies are built and run through tech enabled services. Colbert. You know, a lot of people, like would pronounce it Colbert, but it's not. It's Colbert. Okay. Colbert brings over two decades of distinguished technology leadership experience in his role at Invisible Technologies. He spent more than 20 years at VMware, where we all work together. He built a remarkable career starting as an intern in 2002. I love that story so much. Kit, welcome to the show.

Speaker A: Well, thanks so much for having me. Happy to be here.

Speaker B: Yes, and happy Friday too, as well. So, as you have seen, and those of you who are longtime listeners of the show know, uh, that we have a show structure. In each show, we talk about topics and objectives, strategy and tactics, teams and tools. And then we get into business impact and at the end we do a quick lightning round. So, Kit, you can kind of think about what you want your lightning round topic to be. Something that sort of smacked you in the head, knocked your head back, and hey, everyone needs to know about this. All right, so getting into the first section of our show topic objectives. There was a really cool time after the ChatGPT moment. Don't worry, everybody, we're not going to go on the way back machine and spend too much time there. We all sort of in the marketing team at VMware, and we all just discovered, whoa, there's all this great stuff. I think Jessica actually discovered it first. Then I kind of came in and say, we should do this. And like, well, Jessica's already doing it. Okay, where's Jessica? Who is she? And let me talk to her. And so you, of course, were in that as a technologist too, and then saw, okay, there's other parts of the business and specifically we're talking about the marketing team that are, okay, we're excited about this, but it is the very, very beginning. And now we'll talk about the beginning, get that story. But then later on we're going to talk about how that model's matured. But give us a sense of what was that story about when you kind of first saw it, what marketing was doing with it. And then we'll kind of go into what does successful collaboration between marketing, the CTO office look like? Because you were CTO at VMware, Jessica and I were in marketing. We were like gung ho with this generative AI thing. We want to do everything. What was that like for you? Yes, take us back.

Speaker A: Yeah. So this would be back in 20, what, 2022, when ChatGPT came out. And I think before that, I mean, okay, putting large language models aside, you look at more traditional machine learning, and it was something that could only really be done by experts. You had to be some sort of data scientists or be deep in the math and know how to train models and do all these sorts of things. And really what the LLM enabled was for anyone to take advantage of these technologies. It's because it's a natural language interface.

Speaker B: What could possibly go wrong?

Speaker A: I know. Well, it is funny because we're essentially here using natural language as a programming language. And what you've seen, it's. It's funny. There's now roles that are prompt engineers, people who learn to quote, unquote, program in natural language in English. And you'll see a lot of these. They have whole guides on how to do prompting, and it's pretty complex, actually. There's a lot of instructions that you want to give these models, and not only the instructions you give, but how you place them within the prompt. And so there's all sorts of these best practices that are emerging for me what happened was In November of 22 when ChatGPT first came out, I remember trying to get on. I heard about the hype on Twitter, I think it was still called Twitter at the time and immediately tried to get on but of course they were overloaded so it was a wait list. But I def remember when I was first interacting with it, it was just kind of mind blowing. You're like, this is wild. Not that it worked perfectly every time, but it was much closer than I had ever expected it to be that soon. I think what's interesting though is that you do have this sort of inflection point, the ChatGPT moment. But that being said, there were a lot of other companies that had developed their own large language models before that, but they were not quite as general purpose as just this chat interface that ChatGPT or excuse me, OpenAI had created. And so I don't think as ah, someone who tries to stay on top of the technology shifts, I don't think I had appreciated enough what was happening in some of these different realms. And so getting back to marketing, there were a number of marketing tools at the time or companies that had built LLMs that could help marketing folks automate their day to day lives, help create copy, help bounce around ideas for campaigns or whatnot. And so I think it was one of those situations where for me it was very eye opening and I was like, man, I wish I had known more about the potential of this technology sooner than.

Speaker C: Yeah, I mean it's interesting because at VMware we were already onboarding Jasper at that time prior to ChatGPT. I think ChatGPT helped democratize what generative AI is. And I remember listening to podcasts from their CMO at the time that was like, it's easier to explain now because you can all see it. But yeah, there were companies using GPT2, which I think was only available via API at the time for that. So it definitely accelerated a lot of it. And luckily we were kind of ahead of that, having heard about Jasper at a time. And so we put together formally our Marketing AI Council in February of that next year because specifically the brand team started getting questions about whether product marketing teams can use ChatGPT to write blogs. And all of a sudden the brand team starting to realize, oh my gosh, we don't have any guidelines, we need to have a pov. They took pen to paper kind of writing this almost like manifesto of what generative AI meant to marketing. And a lot of the open gaps that they had. Jasper could answer in terms of security and brand voice and all this kind of stuff, but pulling that together, kind of creating this marketing AI council. I'm curious like how you got wind about it and what you guys were doing because you were also putting together an AI council. But obviously VMware had uh, AI in its products for a decade. So there's already an AI ethics policy and like a lot of tech companies already ahead in that space. But the generative AI side was new and still needed a lot of education and guidelines for employees abroad.

Speaker A: Absolutely. I can't remember exactly how I got wind of what you all were doing on the marketing side. Remember it being very quick. I was kind of shocked. I was like, wow, they already kind of got their act together on this one. This is great.

Speaker C: Pat's back m getting Jessica.

Speaker A: Exactly. Well done, well done.

Speaker B: Oh yeah. Jessica's leadership was fabulous on that. And there was a lot of really organizational maturity I think was probably the key thing. It was a great thing to be a part of.

Speaker A: Well, I think to your point though, like the fact that you had onboarded Jasper already and we're starting to use it and it can kind of see the potential. I do think it's one of those things where if you haven't actually used it yourself, then you don't really get it at like a sort of a fundamental.

Speaker C: Mhm.

Speaker A: Well you're like, oh yeah, these computers can do this, whatever. But then you like see it and you start asking it like the weirdest, wildest questions and it like does stuff and it's actually pretty accurate. And you're like, oh my God. So it changes your mindset. You all caught onto it very early. When we started looking at it, I was again sort of doing a lot of work through November, December, maybe even January of 22 to 23, trying to understand it, trying to see how people were using it and so forth. And then it was probably over the next couple of months after that that we started looking at do we set up some sort of company wide AI council. There were just a lot of things happening. People were using it to generate code. Some people may have been checking that in, we don't really know. We had no safeguards or processes or rules around any of this stuff.

Speaker B: The Wild west is.

Speaker A: Yeah. And we had a lot of people using it. Yeah. To generate documentation, all sorts of stuff. And I think there was a couple of different concerns. So first of all, I think one of the things we did well was really being cross functional. So we got everyone in a room together, we had marketing, we had engineering, we had legal, hr. It kind of got everybody together and said, okay, let's think about this. How do we enable the individuals, the employees in the company to leverage this and to really accelerate what they do, but how do we also protect the company? And I think at the time there was a couple of different factors happening. Number one, all this stuff's brand new. So to your point, Yadine, it's wild west, like who knows what's going on. But as part of that there's a lot of open ended legal questions. So for instance, if you are using an LLM to generate code and you're checking that code into your code base, but that LLM was trained on open source libraries or repositories that may have restrictive open source licenses, do those restrictive licenses now apply to the code in your code base, thus potentially opening up your entire code base to be at legal risk? These are the things that weren't really known at the time and there was absolutely no best practices around it. There was really nothing that stood out there. So I think as a large company we had to be a little bit more careful. I think a lot of startups probably didn't care because what do you have to lose, you know,

Speaker B: yes, they're moving fast and breaking things, they're too busy.

Speaker A: But as 12, $13 billion a year company, it was like, okay, we've got to really be thoughtful about this. Actually one of the positive things was working hand in hand with legal. I think a lot of people look at legal and they see them as sort of the group of no. Part of their job is to highlight any risks and concerns and sometimes people take that as a no. But really we were like, help us out here, tell us what do you think the risks are? Let's manage them. And I think we had a really good partnership working holistically across all these different functions in order to achieve that.

Speaker C: Yeah, we had legal in our. They were at least invited to the very first meeting we had as well, luckily, because the brand team was already working hand in hand with legal day in and day out anyway. So I remember that was a really close integration from the beginning. But I think it's interesting what you said because it's part of what the advice I give on how to start these types of councils is, number one, is always bringing people together. And that's exactly what you said organically. I'm sure you've not read my blog post on this at all. Right. But like, you got to bring people together cross functionally and so whether it's a, uh, functional specific AI council, like a marketing AI Council or a CTO or CIO led AI Council, which is how I identify yours, it's bringing people together. But I would love for you to talk a little bit about how you might see the difference in the two. But I think you touched on it, the governance side of things. But also, like, how did you set your charter? But also how are you guys thinking about giving some leeway to the functional teams to also move forward?

Speaker B: Oh yeah, I think we drifted into the strategies and tactics section of the show, but that's a good question.

Speaker C: I'm just moving right ahead.

Speaker A: Yeah, I think the reality is that the concerns and risks and benefits are going to be different for each of these functional areas. Right. As I said, I articulated one example of risk from an uh, engineering standpoint, which is specific to code. Maybe there's theoretically some of those problems like copyright type things. I don't know how pressing they are on the marketing side, but I do think that you've got to make that decision somewhat independently for each group. Can't have a, uh, one size fits all approach necessarily. Instead, what I think going back to the cross functional piece is really being able to trade off tips and tricks and say, here's what worked for us, here's kind of the guardrails that we need and so forth. And so that's what we tried to do. A lot of the AI Council, the company wide one, we did focus a lot on engineering specific issues because we didn't have, I mean, I guess because I ran that one, if I didn't run it, I suppose I would have created my own like functional specific one. But since I ran, it was all the same and we included a lot of the engineering leaders as part of that. My recollection though is that we tried to empower and support you guys on the marketing specific council as much as we could. And 100% we ran in parallel, enabling you all to focus on your issues. But then you, Jessica, joined us in the um, overall AI Council and we were able to keep tight at the hip there and coordinated.

Speaker C: Yeah, I think that's a key thing that I like to talk about. And so I'd love for us to just double click into this for a second too, is that you guys took a very committed stance to course protecting company risk and things like that, but also letting the marketing team do what it needs to do. So even building your own internal LLM, but acknowledging that it would take time to do that. I Remember specifically having this conversation, the marketing team can't wait for that and has to move forward with the tool that's already there. So what advice would you give to marketers who are trying to convince their CTO that that's the path? Because I think now a lot of CTOs are rolling out um, Copilot or Guide or Gemini or ChatGPT, but the marketing team is often saying we need something different or they're building their own internally, which is going to take time too.

Speaker A: Well so that's what we did at the time and what I might do now, two to three years difference, which is like lifetimes in the AI space. Yeah. So what we did at the time was again we had legal do sort of the checks for Jasper and these other ones just to make sure there were no sort of issues there. We also had to ensure for any sort of SaaS service, ensure that our data, whatever sort of data we're giving it was protected and kept confidential and that sort of thing not used for like additional training. But on the engineering side, yeah, to your point, we actually did have to essentially not create our own model, but we took open weight model called Llama from Meta that Meta creates and we basically customize that one for our co generation use cases. We did a bunch of fine tuning to it, but we ran it on prem within our own environment. So we fully controlled everything and sort of had that dialed in. But now, you know, I think it's a little bit different. The industry has moved forward quite a bit and so let's just focus again on engineering. So we've got tools like Cursor and windsurf, which are IDEs, integrated development environments that developers will use and they do all sorts of stuff. Back in the day, two plus years ago you could have a uh, LLM, um, basically finish a line of code for you, start typing, it'll kind of know where you're going and finish it. Now it's got chat interfaces where you just talk to it. You're like, hey, here's the thing I need to go do. So go review this, go look at that, look at that bug report and fix this thing. And it'll just go off for a little while and process maybe like 10, 20 minutes and give you back an answer. Then you have things like cloud code, you've got coding built into ChatGPT, you've got Copilot. So you got like a lot of different tools now which are much more advanced and some of these are starting to provide the sort of insights into the data sets they were trained on and what sort of assumptions you can have from a legal liability standpoint that you just couldn't find before. And I think also the other complicating factor that I've seen now is that at least from the engineering side, competition becomes really, really interesting. So what you find is that the best in class startups are generating probably 90% plus of their code using AI. And so that means that an engineer is now sort of a super engineer, that this like one person can now do the work of five people from before and just be much, much more productive. And so what that means is that these smaller startups are actually able to compete with much larger ones who may not be using these sorts of technologies. So I think that's the other really interesting wrinkle. Whereas before a couple years ago, when it was first coming out, everyone was sort of playing with and experimenting. No one necessarily had a clear competitive edge. But now it is the case, I think that certain companies who have fully embraced this and who are leveraging AI agents for as much as they can do, do actually have a competitive angle. And so I think that puts extra pressure on folks, especially like larger incumbents. They really need to start thinking through how they can generate more and more of their code using AI.

Speaker B: Yeah, I think there's a couple interesting things I wanted and I think we're, we're gotten to the, the teams and tools section of the show. You're wondering, okay, well, is it the tool that actually is doing this? But no, it's actually the way that the organization runs. It's how they workflows, it's how the individual has an understanding of the technology in general. Because like you'd said, some engineers are not using the tool, not because they're not good engineers, it's just because there's either a culture or a potential skills gap. And it almost doesn't matter which tool you're using, as long as it's a high quality tool, it then augments your capabilities. But you have to have the skill, you have to have the perspective, you have to have the habits and you'd be able to know how to use it and be able to monitor it and make sure that generating good code, and that is not a simple thing. So the individuals have those skills. And this goes for marketing as well. We're kind of cross pollinating here. The individual needs to be the one who works differently. And then it's almost like you can insert any tool, but the human is the one who has to be able to have the strategy know the questions, know what good is, and then be able to do that piece. And it was interesting in our story where the engineering, the office of the CTO side was going through some of the same sort of questions and process and maturity that marketing was and that cross pollination was happening. And I was curious to know, from a tools perspective, you talked about the ones you're buying versus build. Was there anything that you were looking at in the marketing side that sort of changed the way you thought about what you were doing on the engineering side?

Speaker A: Well, I, uh, wish we could have just used a tool out of the box like you guys did again a couple of years ago. We were very limited, as I said, very cautious as well about this. We wanted to take advantage of the value, but didn't want to put ourselves in some sort of future legal jeopardy because of it. But I do think that's changed dramatically. Like if I look at Invisible today, we were doing kind of all of the above. We're using a bunch of these different tools internally. We're building a lot of things as well. We are taking models that we can leverage and run not in the cloud, because some of our clients are still very sensitive about their data and don't want the data going up to the cloud cloud. So we've got to take open weight models like Llama again, like now llama 4. It's advanced a bit. And so I think what you see is that you really do want a variety of tools to be able to leverage the best one for the job. But to your point, Yudine, I think it is a big change in the individual that how you think and how you operate needs to change. One of the best things I heard recently, this was a few weeks ago, someone was talking about talking this notion about AI eliminating jobs. And someone said, well, it's not really AI that does it, it's a human who knows how to use AI might eliminate a job.

Speaker B: Yes, I think that's a really, really good perspective.

Speaker A: It is. Because no AI is going to run fully autonomously forever. Someone's got to start it, someone's got to give it a prompt. As we talked about before, someone's got to oversee it. Like these things can't run autonomously forever yet. So in the end you can imagine a uh, startup with like one person who's like the CEO and has all these AI agents that they're setting off working, they're still overseeing them. And so personally, on that topic, I'm more of an optimist where I think that what we're going to see is something akin to Jevons Paradox. Jevons Paradox basically says is the unit cost of something goes down, the aggregate spend actually goes up. You imagine, like the price of something going down. You're like, oh, well, the overall spend on that thing would go down, right, because it's cheaper and you're spending less money overall. But the paradox is that, no, that's not the case. What actually happens is that because the unit price is lower, it means a lot of things you may have wanted to do before were not economically viable, but now they are economically viable, and so usage goes up. So you saw with Uber, Uber is the same way. It got cheaper, it got easier to use, and so the overall spend on rideshare or the taxi type mobility stuff, like, went way up. And I actually happen to think this will happen. If you can think about this, if you can hire someone and it's the power of like, five people, why wouldn't you hire more? Because you can do so much more. You can go out and be more productive, you can build more, you can get out and talk to more customers, like, run more marketing campaigns and get better at increasing the effectiveness of those things. So personally, I think it's one of those things where, yeah, maybe there's like, a little bit of negativity out there about employment impacts, but in general, I think it's going to be a, uh, superpower that if people know how to use these tools, it's going to make them that much more effective and that much more desirable by any employers.

Speaker C: It's funny you mentioned Uber, because there's something that I just read recently too, that was like, also, the model where Uber used to be really cheap got us all hooked. Now we can't imagine life without it, and now prices are like 3x or something more. So to your point on if you can hire somebody who could do the job of five, eventually you're going to realize you can't work without them. And so you're going to have to pay more for them to hire more for them because they realize.

Speaker A: Well, I agree. And someone who knows someone who's really effective using these tools will be worth their weight in gold. So maybe it's not five people that can do 10 people's work. You're like, oh, wow, okay, this person's amazing, so let's get more of them.

Speaker B: Yeah. And I think the saying really, um, is that AI is not going to replace your job. Somebody who can use AI really well may replace your Job.

Speaker A: Yeah, I think until we have some sort of future where this artificial general intelligence we've got like a whatever sort of brilliant future but that's not for a long time and I think in the end it's still a human in there somewhere behind all these things.

Speaker C: So I'm curious for your perspective on the role of the CTO and then collaboration with other teams more so thinking about marketing. There are a lot of surveys out there too on um, like who owns AI transformation or who owns AI strategy within companies and is it the CMO or is it the CTO and how do they work together? How do you see that? Collaboration?

Speaker A: Yeah, it's an interesting question. So in general on the collaboration piece, the way I do it is that I think it's imperative for us on the technology side to really sit down and fully understand the problems and goals of the different teams that are there within the company. I think oftentimes technologists kind of build technology but don't really understand who they're

Speaker B: building it for and that never happens. Cam.

Speaker A: Yeah, I know the key thing about this is partnerships. If you think about the AI council we had, it's like understanding that marketing had ah, a different set of trade offs and risks than engineering does. And so from an IT perspective it's like okay, let's grant Jasper use for the marketing team, that's fine but maybe we lock it down so that we're not using these other tools for engineering yet, which is kind of the approach that we originally took. I also think that oftentimes from a tech perspective you will need to build some, build your own tools for a variety of use cases. And there again I think that collaboration is really key so that the tech team's actually building something useful for whether it's marketing or sales or whoever it is. What I found is that a lot of times these individuals will go off and do whatever they take and it can get like a little messy. Maybe using some of these AI tools you're putting a bunch of data in random like spreadsheets, you know, stuff's kind of spread out and that can create its own sort of risk over time with data management and so forth. And so I think the tech team's opportunity is to be able to look at those patterns, discern what the objective is of those teams and see if they can build a more robust solution that can simplify and bring faster time to value.

Speaker B: Yeah, there was one interesting thing that's off of what you just said, Kit, and I don't think I ever Had a chance to talk to you about this while it was happening. But there were definitely individuals who saw how quickly marketing was moving and they were, let's say, on the engineering side, and they were super, super excited, like, oh my God, you guys have this tool. When can I get that? And there was definitely even people in marketing too, that hadn't been onboarded yet. And so that's going to happen anytime any new tool rules out there. And, uh, AI though, is a very special category of that. How did you handle that sort of asymmetry with the tools that were rolling out? Because you said sometimes, yes, marketing just do this one out of the box, but engineering can't do these others out of the box. How did you handle that, that difference in access?

Speaker A: So I think it was delineating in a bit more detail what could and couldn't be done. So, for instance, within engineering, we said, hey, it's not that you can't use any of these tools until we build our own, is that you can't use these tools for a certain set of use cases. So, for instance, if you're building an internal tool that we're never going to release, like, go for it, do whatever you want, it's fine. We're not worried about that use case. That's not really our core IP per se. But if you are working on a code base that is shipping to customers, that actually is more of a concern for us. And so there the bar is a little bit higher. So I think part of it is finding areas maybe that are lower hanging fruit or maybe areas that aren't as critical or as risky to the business that you can get a start with. And so that way it prevents the dam from like, collapsing on itself. People can start getting their feet wet a bit, start playing with it. And then as you mature to a point where you're ready to start applying these tools onto your most important parts of the business, then people themselves have been trained a little bit and so they're more ready to go and you can get faster realization of value.

Speaker B: Yeah. And so I think that's float into the impact section of the show. Where do you feel like? Because there was a lot of decisions that needed to be made and you just explored a few of them that were really, really critical. How are you looking at the balance of, hey, look it, I see what the impact is of saying, okay, marketing, you can go and run fast with this. And I also see the value in making sure that we're mitigating risk and being cautious too, from an engineering side for certain use cases, and those are differentiated. Where were you seeing being able to articulate the value of what marketing needed to do and why they could do it while still monitoring these risks versus, hey, look at. I see the tremendous impact that engineering can also do. But here are some of the risks. Where were you seeing threading that needle of managing risk, but then getting the impact that you knew you could get out of these tools and deciding who went first and which use case would get prioritized?

Speaker A: So what we did is we set up a process and here's one general approach that we took, which is that there are going to be a set of folks, the early adopters, who are going to want to use this. And if there is a lot of risk, like there was early on, what we did is essentially we intentionally added some friction to the process. And what that does is it means that anyone who's like, uh, I don't know, maybe I'll give it a try, and they're like, oh, crap, there's like this wall of not process. I mean, we had them people fill out like a super long form with.

Speaker C: It was called something like a generative AI.

Speaker A: Yeah, something like that. I remember we worked on that thing. It was like 40 questions long and there are a lot of detail, a bunch of, how is this data being used? What are you doing with it? They wanted diagrams of data flows and all sorts of stuff.

Speaker B: See, now I would just use AI to fill that all out.

Speaker A: Dude, I was about to say, now just like toss that thing into Chat GPT and like, make up something for me, buddy.

Speaker C: My CTO won't know.

Speaker B: Yeah, fill this in.

Speaker A: Yeah, at the time they couldn't do that. But the point was that if you add a little bit of friction, it sort of separates out folks who are unsure, whatever, from folks who are like, very serious about it. And generally speaking, if they're very serious about it, they get the value and they're willing to put in a little bit of effort to mitigate the risk, I. E. Like filling out this form and maybe doing whatever else we needed them to do. So that was a useful way at first, I think, to do that. And we never intended that process to live forever, but it was kind of like, okay, give us a chance to catch our breath, let's see what's coming through. And as we start seeing some of these different uses that people are asking for, we can start to say, okay, can we relax constraints a little bit? But that's like one way that gave us an Ability to sort of discern, okay, who's really into it and who should we spend time with versus who's not so much into it and let's wait for later to engage with them.

Speaker C: Who's dabbling and who actually has a solid business case here.

Speaker A: Exactly.

Speaker B: Yeah. And I'm curious for that intech form if you got some really good insights into what people are using for how they're using it. And maybe that surfaced some of those use cases and then gave you the ability to, uh, sort of crowdsource some of the justifications and potential ROIs for, hey, this is the reason why we should do this. I can see impact now because I'm getting feedback from individuals who are super passionate about it and excited about it and see, see that potential?

Speaker A: I think so. One of the interesting things about it is that forcing people to write stuff down is actually a useful exercise just in general.

Speaker B: Yeah, writing stuff down.

Speaker A: Yeah, I know. I'm a big fan of the Amazon style six pager or two pager, whatever. In my meetings, I don't want people presenting PowerPoints. They gotta come with a document. We all sit there, we read the document and then we ask questions about the document, if we have any. It's just a much more efficient way. But what I found is that people complain and they moan. It's ridiculous trying to get people to do this. It's like arm twisting and teeth pulling.

Speaker C: Reminds me of filling out a brief for marketing.

Speaker B: It's like my kids.

Speaker A: Exactly.

Speaker B: Oh, uh, yes. They're like, why do I have to fill out this creative brief so you can tell me what you're trying to do? That's where.

Speaker A: Well, okay, so that's another sort of friction. I think you felt the brief because it's like, well, this executive whoever's time you need is busy. So make someone fail to brief because it raises the bar. But for me, what I find is that I force people to write it down because, yeah, that's a pain. But they often get a lot of value out of it. Because my experience is that, ah, forcing yourself to write stuff down really clarifies the thinking. And as you're writing like, oh, uh, crap, I actually didn't understand that as well as I did. Let me think through this. But back to the question that you had, Judine. Yeah, I think there was a lot of value that we got out of it, actually that people understood better what they were trying to do. You know, they came in, weren't fully sure, sort of half baked. But by forcing through these like 30 or 40 questions, however many they were, and they all had sub questions that actually got them a lot more clarity around what they wanted to do. It's kind of almost some like pre planning in a way. So I do think there was a lot of value in that. Now again, I wouldn't do that today given where we are like at Invisible, what we do, we make a number of selections of tools based on a variety of criteria. But then at that point it's like I want to encourage people to use it as much as possible. Now as a matter of fact, like if you're not using it, that's a problem. It's kind of one of those things where I see all sorts of evolutions happening. There's been this kind of set of dialogue around doing engineer interviews where typically you'd send them to a site that would have them do some sort of programming task and that would ascertain how good of a programmer they were. And now people are quote unquote cheating on these things because they use AI tools to do it or whatever. And part of me is asking, well, is it cheating though? Because don't we want them to use these tools?

Speaker B: Right, exactly.

Speaker A: So it seems like instead of giving them just a little tiny programming assignment, give them a giant one and say we expect you to use AI coding tools.

Speaker B: Mhm.

Speaker A: I want you to come back in a week or however long it takes you, not just with a working program but with a document specifying the different trade off decisions that you ask the AI system to make and explain why you make them. And then the interview is grilling you on those design trade offs.

Speaker C: And I want you to do it in half the time. Yeah, I love that.

Speaker B: That's beautiful.

Speaker A: Exactly. Yeah. The whole point is that we have calculators. We don't expect people to be able to do long division. It's like, yeah, you know, in theory how to do it, but you're not going to do it every time. Just like use a calculator, dude, why are you going to do it? And so it's the same thing I think here it doesn't just apply to engineering, applies to anything whether you're making a marketing campaign or uh, a sales strategy or what have you, you should be using these tools. But and this is really, really critical, you must understand what it is doing because if you don't, it kind of defeats the whole purpose. Like these AI coding tools will make mistakes and they will introduce bugs and they will break things if you don't know what's going On.

Speaker B: Mhm. Oh yeah.

Speaker A: And so what we need to be testing for is understanding what this thing is doing and can you actually manage it well, or are you just like letting this thing fly off the handle, which doesn't benefit anybody.

Speaker C: So this kind of leans into the topic around AI adoption. And how are you thinking about how do you measure AI adoption or the success of a program? You just gave a good example for how engineers might test. How do you think if the CTO's owning AI? Ah, strategy. And the goal is for everyone to use AI. Like Moderna has 100% adoption on every employee that has a laptop. How are you thinking about how do you measure that and what success looks like?

Speaker A: Well, it's a really good question. And part of it, like you were saying, the goals have everyone use AI. Uh, that's not actually the goal. Right. That is a strategy to achieve the goal.

Speaker C: But why are we doing AI?

Speaker A: The goal's gotta be some sort of business outcome. And we look at this a lot. So again, I'm just starting at this new company, Invisible, and I think it's my fourth week here fresh. But it's been interesting talking with our folks, with uh, our employees, with customers. And we do see a lot of these challenges. Look at the data. A lot of these AI projects I think about to nothing in terms of business value. Like you invest all this money, but like you're not actually seeing any benefit out of it. And so the question is, what's the gap there? And I think a lot of it is it's lack of well defined business cases. It's kind of fomo. It's like if we're not doing something around AI, uh, there's a problem, so let's just dive in and start doing it. But secondarily, I do think it's that a lot of these tools by themselves are not going to be enough just using the tools, like not enough. You got to change the culture or oftentimes you have to take the tool and customize it to really fit into the workflows or other tooling or whatever you have within your environment. And so I think that it's not really looking at things from a systematic point of view, it's looking at things more sort of tactically. So for me it's let's talk about the business value and let's work backwards from that to see how do we get things. And let's ask a lot of questions. And you shouldn't assume that AI is necessarily the answer, but it probably will be at Least in some part. But I think you should challenge yourself to say, well, can we do this in some other way, some simpler, more traditional way? And if AI is needed, then let's go and do that.

Speaker B: So wait, are you saying that you should come up with a strategy first and then see how AI should help you with the strategy?

Speaker A: I know, it is shocking. You know, sometimes it's like the most basic life is not that complicated. Work is not that complicated. It's just remembering to do the basic stuff. But I do think it's easy to get caught up in the technology. Like, this stuff is so cool. Like, it must have value. Cause, I mean, just look at it, right?

Speaker C: Yeah. And, uh, that's why I think IT teams are looking for usage reports as a metric. But the usage report is not actually a metric. Like, they could be using it for their personals. Even if you're using it as a thought partner, that's harder to measure. But like specific outcomes related to the business, related to your goals and KPIs. And that's why I always talk about, instead of working from how do I use AI up to where does it drive value? Starting from where do I need to drive value? And then making your way down to perhaps there's an AI use case here.

Speaker A: Yeah. And I think it's defining what does success look like? And that being very much from a business perspective. And so you might say, hey, I'm running an engineering team. And so maybe it's like, we want to be able to move faster. I was, okay, well, what does faster mean? Define that for me. And it's maybe it's more Sprint points per unit of time, month or sprint or what have you. So it's just keeping those really clear metrics in mind, which again, is business 101. Like, none of this is rocket science, but I do think that there's just so much hype and there is a lot of this fear of missing out. It's, uh, very real. One of the things that I try to practice is this almost counterintuitive mindset that you kind of need to slow down a little bit to speed up. And by that I mean instead of just mindlessly running forward as fast as you can to, like, the new hot thing, start asking some questions and start trying to understand what's going on. And that will take time to do, you would have been further ahead if you just ran mindlessly. But once you realize that you're running in the wrong direction mindlessly, all the rework and, uh, throw away resources, you've done. That all amounts to a lot and is way worse than just taking a little bit of time to think clearly about what you need to go and do.

Speaker C: Yes. And I feel like this is stuff Gideon, you and I talk about all the time too. And this is how we want CMOs to be approaching it from a strategic perspective. But even if we're thinking about the bridge from CMOs getting their CTO on board to collaborate in this together, it's coming at it from a strategic perspective and not just saying marketing needs a tool because we need a tool because we need to do AI. And I feel like what you're saying is the CTO is, should be, but is also saying there has to be a strategy behind this other than just the tech too.

Speaker A: Yeah, it's easy to say. It's much harder to do. And you see it happen a lot where this kind of, this group thing starts happening. Or maybe no one really feels empowered to question it because, like, oh, someone else must have figured this one out why we're doing this sort of thing. And so I do think it's imperative that you create a culture where people can ask these sorts of questions. Be like, hey guys, what are we doing here? Let's have a conversation. I think it's one of those things where it can be sort of self evident that it's the right thing to do until you actually start asking some hard questions. And it's like, well, actually, is it the right thing to do? I don't know anymore. And everyone kind of looks at each other. And so I think as a leader, it's incumbent to create that environment where you ask hard questions, even about things that seem, quote, unquote obvious.

Speaker C: Yeah. And if they often say, right, the smartest people in the room are the ones who sit back and ask the questions, even if it makes everyone uncomfortable. But it also pressure tests your strategy and what you're doing too.

Speaker B: Yeah. And then you don't go off, like you said, running in the wrong direction. So this has been great, Kit. I think we're now in the lightning round section of the show where we just say it could be something that happened or it could be something that just you've been thinking and musing about and saying, hey, look at, this is really something that's been top of mind for me. And so I'm going to kick it off and give you some, uh, time to maybe to mull yours over. Maybe you already have one. But it's funny you'd mentioned it kind of at the top of the show with prompting and how important prompting was and prompt engineering. I was actually just putting out a LinkedIn post today that was basically saying, stop prompting, because I think people need to do is let AI prompt for you. Like you said, there's tons of detail and there's really great stuff that you can do if you do it correctly. And I'm like, well, why don't we just move in the direction we want to. Like, pretend like you're just writing an email to somebody who's like an agency that's doing some work for you. List out everything that you needed to do. But at the end of that email, say, generate a prompt that I can use in Google Deep Research that will create a plan and be able to execute on this. And then the LLM will create the prompt that you basically did in your email. Like, I'm just going to send an email to somebody. Everyone does that. But then at the end, just tag that little thing in the output should be a prompt, and that'll dump that into Google Deep Research and it will create amazing things. Or the prompt first will create something really amazing. Then that prompt is actually a project plan. And then I'll go and post it, say, everybody, this is the right plan. All the stakeholders that would normally involve in the process, they'll take a look at it. Yeah, but tweak this, but add this. Well, we don't want that people comment on that. You integrate the comments into the prompt that the ad created for you and then dump that in and then you get beautiful outputs because you have at the core all the opinions and perspectives and knowledge and experience of the individuals within your team. Crafting something that then is get fed into something like, oh, I'm gonna go and run with it. Great, it's gonna go and run with it. But like you said, Kit, something can get run the wrong direction, but if you spend the time CR something for it and getting on the same page, then that becomes magic. So that was my sort of knock my head back, aha, uh-huh moment. I'm just going to start talking to it like I would normally talk to an agency or anybody else and skip the 12 or 14 different meetings that I would have with an agency and collapse that whole timeline into about 15 minutes of stuff. So that was my aha, uh-huh lightning round thing for the show.

Speaker C: Yeah. And everyone needs to be a prompt engineer to some extent. But I think the difference is on the engineering side, actually building and fine tuning a system is very different from a marketer going out and using it as an assistant tool.

Speaker B: All right, Kit, do you have yours?

Speaker A: Yeah, it's not so much actionable, kind of like yours is. But the thing that came to mind first was a meeting I had earlier today. One of the things I'm really enjoying about this job is I'm just learning a ton. And it's invisible. It's like a super interesting company where we have kind of two lines of business. We have one that we serve the model builders, like the big people that make the LLMs that all of us use. And then we have the enterprise side. And on the model, well, actually we have this whole area we call AI training. And we have thousands of these human experts that provide feedback to the models to make them better. And I didn't know much about this, you know, I kind of assumed, somewhat naively, that these are just interchangeable folks, people like me, like normal people. It's like the old Silicon Valley show joke. Is it a hot dog or not a hot dog? Half like a human? So to do that you're like dog,

Speaker B: pig, dog, pig, loaf of bread.

Speaker A: I don't know exactly. But the thing I learned today, I was like, pretty astounded. Some of these projects are fairly complex, comprising hundreds of different individuals that are very advanced, like PhDs and like astrophysics and stuff. It's mind blowing. And we've gotten to such sophistication in these models that they need people, uh, with really advanced degrees to be able to actually give them feedback and help to correct them and help to make sure that they aren't hallucinating and are doing the right things. Within these human agents, we have people that are like project managers that are coordinating work across all this. I was like, wow, it's like a symphony of effort here that we're providing to these model builders to help them train the models. The takeaway for me is that we talk so much about AI as this sort of non personal, non human thing, which if you look at it, it is kind of wild. The first time you see it, it's kind of mind blowing. But as we talked about, it's very human in all aspects. As I said, AI is not replacing your jobs. A human using an AI is replacing your jobs. Not only that, it's going to enable people to do more work and create more jobs in the future. And not only that, it's actually humans who are working to collect all their knowledge to put into the models. And so I think it's just this interesting thing where we tend to think of it very outside of humanity or whatever. But in fact, it's like very much weaved into the core of what we're doing. Again, personally, for me, I think can be extraordinarily empowering. So that was like a super cool, interesting takeaway I had here on week four.

Speaker C: I'm excited to follow you guys more on that because I think people worry a lot about the ethics of AI systems and the impact on humans, kind of the dark side of AI and all of that stuff. Even just publicizing that you're adding jobs.

Speaker A: Absolutely.

Speaker C: Even in that sense. But also they're not like mundane jobs. If you're going to say whether how to build a rocket or something amazing is hallucinating. You have to know what you're talking about. So you have to employ people who have really high value skills. So that is fascinating to me. So I'm looking forward to following you guys more and I think there's a lot of storytelling there that can be had on almost like the human side of the brand that you guys are building as well. Very cool.

Speaker A: Absolutely. Yep.

Speaker B: Yeah, yeah. Was that your lightning round?

Speaker C: We can go with that. You didn't if you want. Team Human.

Speaker A: Yeah, Team Human. There we go.

Speaker B: Team Human. Okay. Those are lightning round of steam human. Okay.

Speaker A: Empowered. Empowered humans with AI. There we go.

Speaker B: I love it.

Speaker C: Yeah, it is very empowering. There was an article I posted yesterday about upskilling like a project manager function in a lot of marketing jobs have just kind of come down to project management, moving tasks from one place to another. How do we not only upskill with AI, but upskill those marketers in marketing functions and content strategy and understanding the brief process and all that so that they actually can draft content themselves in house rather than just move content from place to place. For example, and Myla, who is on our marketing AI council too, in kind of a field marketing digital execution function, abm or she was like, it's so empowering actually when you can give that ability to marketers who don't normally get that. So thinking about it, not as, oh no, I don't have this job anymore. Oh, it's something more to learn. It's. It's so empowering. And if you have that mindset, almost like that growth mindset, but going into all of this and thinking about how AI can empower you, I feel like I don't know what's keeping people from moving forward.

Speaker A: Yes, it is mindset. You're absolutely right. This notion of you can look at any fact of reality and bring judgment or perspective to it. And it could be a negative thing. Oh, uh, the sky is falling, the world's ending. Or it could be a positive thing. I think the growth mindset, you lean into the positive, saying, look, everything has its ups and downs to it, but how can I leverage this to grow myself as an individual, to grow in my career? And it's clear that if you are not leveraging A.I. uh, you're doing yourself a disservice that you're not taking advantage of what's out there to help you grow and prosper in your career. Yeah.

Speaker C: And who doesn't want to feel empowered?

Speaker A: Exactly.

Speaker C: I love that theme to end today, Eden.

Speaker B: I love it. That's a happy theme. Empowering. All right, well, this has been great, Kit. We're getting to the end of the show, so I wanted to thank you for the fantastic conversation, and thank you very much for joining the AI Edge podcast.

Speaker A: Yeah. Thank you guys so much for having me. I really enjoyed the conversation. Yeah.

Speaker C: And hopefully this was helpful for marketers out there. You're thinking about how to approach your CTO or how a CTO is thinking about things. So thanks for helping us cross that bridge today.