CELab - Ep 181 - Kelly Mullaney - From “Team of One” to 10x Impact: How AI Is Rewiring Customer Education

Narrator: Do you find it harder than ever to keep your customer training material up to date? If so, you are not alone. That's exactly why Learn Worlds has become such a standout in customer education. It gives you the flexibility to build training that feels like your brand, not like a boxed in LMS template. And honestly, the analytics are some of the best I've ever seen. You get real insight into engagement, progress and ROI without having to become a full time data analyst. You can also chat with AI to immediately discover anything you want to know about your learner's progress, course trends or revenue insights. Just ask it questions and let it pull that data for you. The integrations are smooth, especially if you're running HubSpot or automations across your stack. And the platform is actually enjoyable to use for both admins and learners, which I know sounds wild for an lms. But here's the thing I appreciate the most. Customer education teams consistently tell us that LearnWorlds helps them move faster without relinquishing control. It is learning that scales alongside your product and business. And if you need to even train employees or partners, you can do it within the same account. Just create a new school and you are good to go. The platform is very user friendly, so you are not relying on devs for changes that you might need to make, which gives you so much autonomy and flexibility. So if you're ready to upgrade your academy experience or rethink how you deliver product education, check out learnworlds.com today and enter the code CELAB15 for 15% off off the first six months of your subscription.

Adam Evermescew: Welcome to C Lab, the Customer Education Lab. That's right, C Lab stands both for customer Education and uh, certainly excellent, where we explore how to build customer education programs, experiment with new approaches and exterminate the myths and bad advice and in my case, bad acronyms and initialisms that stop growth dead in its tracks. I am Adam Evermescew and I am honored to be here today with Kelly Mullaney, Head of AI Guild at Juniper Square. Hello Kelly.

Kelly Mullaney: Hey. Thanks so much for having me.

Adam Evermescew: Thank you for joining. Um, I want to jump right into it, but also just by way of, uh, uh, starting with those pleasantries. It's not just pleasantries, actually. You and I have been in the same communities for a really long time. At this point, I would say back to the late Paleolithic era or something like that. I remember very distinctly some of those early days in the Slack channels. And so actually it's strange. It's strange that we haven't done this yet.

Kelly Mullaney: It is really odd. I've been admiring your work forever. You're one of the first people that I heard advertising for customer education and speaking about what it can do and obviously wrote a book on it. And it is interesting that we haven't, uh, aligned yet and found time to talk.

Adam Evermescew: Well, thank, uh, you for saying that. And equally, I think in the community you have always been one of the people who, when you speak up, it's credible. I think you're very active and generous in sharing your point of view and really helping people. But similarly, I keep developing an increasingly strong conviction about this, that especially with now a lot of, uh, people just popping prompts sloppily into, uh, chatgpt or what have you. It's easier than ever for people to just share on LinkedIn and places like that just because they can. But there's not a whole lot of substance behind it. And what I think that you did before our AI enabled future, but what I see you even doing more of, I would say now, is when you're popping up, you're sharing things that are super practical and tangible and helpful and I think actually helping show people in our field where the future is and what types of things can be built and how they, uh, like how we can deliver value with them. And that's really why I wanted to talk to you today.

Kelly Mullaney: I love it. Yeah. My idea when I share is to basically show what's been built and not talk theoretically. I think that for a lot of customer education people, um, maybe it's cool to look at AI through the lens of electricity. When electricity kind of came out, like people didn't get the value out of it for about 30 years later because they basically just took an existing process. They're like, okay, we're just going to build electricity on it. And they didn't get that value for like 30 years. And what they kind of found out is let's rewire the factory, let's completely redefine how it's being done. And I think AI is the same thing. It's like not going in and just inserting it into an existing process. It's reimagining it. And so when I show my builds, I'm really just trying to show how you can reimagine your organization around AI.

Adam Evermescew: Uh, yeah. But again, I mean, I think that's great because for a lot of people in our field, either people haven't been able to come as far yet with the technology that exists, or the technology moves so quickly that there's always something new to discover. And I think frankly there are a lot of folks out there who are AI curious, but maybe haven't had the opportunities or the at bats to really build things in the context of their organization that really deliver that level of value. So, really eager to jump in with you to that. But first, maybe it would be helpful to set a little bit of context for kind of how you got to where you are today. Because, I mean, first of all, AI Guild, that's a super interesting and intriguing concept. But also even going back to when I met you in the community, you, uh, were at that time. I remember, uh, your handle was KellyInvestnet. And uh, now you are Kelly Juniper Square, as if we should be defined by our companies. But, uh, can you talk a little bit about, um, how you got into this in the whole place and uh, how that brought you to where you are today?

Kelly Mullaney: Yeah, so my first job was the most boring job in the world. So I started out as a tech writer and I started back in the day Microsoft made a tax product, so it competed with TurboTax to help you prepare taxes. And I started there and my job was to go down as a tech writer every line on a tax return and define what you'd enter. So like on line seven, entering wages. And I used to joke to my friends that like, if they couldn't sleep, just call me up and ask me about, uh, my day. Because taxes isn't the most, most glorious thing. But that kind of took me off into the world of fintech. And so after that I joined envestnet and ran the customer education department there and then ultimately ran it for all business units. So for people that don't know Investnet about, I think it was a $5 billion company at the time, maybe 5,000 people. And I ran, uh, the education function for that. So tech writing, internal enablement, E learning, onboarding, all of that stuff. And eventually I joined, uh, Juniper Square to run the customer education department. And due to my aptitude for AI, I'm now running AI Guild, which is helping people get the most out of AI, either by building it for them or empowering them to build themselves.

Adam Evermescew: And uh, is that on top of your existing customer education, uh, scope, or is this actually.

Kelly Mullaney: No, I actually moved a few months ago, um, moved over to helping people build. You know, I figure, I figure I can probably provide the most value in the organization by helping people, you know, 10x their productivity. And so I think it's, it's like the next phase of Enablement, in my

Adam Evermescew: view, what a cool move because also, I mean, first of all, I don't see a lot of people making the move from like external facing customer education into like internal enablement, but like, especially doing it through the lens of moving into AI enablement. Like, I think that's such a cool opportunity to take the skills that you've honed in the customer education world, but also then, um, uh, well apply it to actually helping people kind of like move into the AI future in a way that's like exciting yet, uh, comfortable, let's say it is.

Kelly Mullaney: And some of it's just dumb luck that I got in there. It's like when ChatGPT launched and kind of kicked this off. Uh, and what was that November of uh, 2022. What did it do? Well, it wrote, right? And so everybody's sitting there, hey, it's going to take tech writing's jobs and all that. And so I made the choice then to just kind of face the person that was going to take me out, so to speak. And through that I was kind of first. I've just been using it longer. As you said, people haven't had the at bats. Well, I got them really, really early and I think it kind of just worked out for me that I just had years and years of using it, but because it was going to impact my job. So clearly.

Adam Evermescew: Yeah, that makes sense. And maybe this would be a good time to talk about this too, because in the pre show you and I were talking a little bit about this. I think neither of us are really doomers about customer education jobs when it comes to AI, But I'm curious more about your point of view on that.

Kelly Mullaney: Yeah. Do you remember the CRAAP test in college when you were writing papers and what you could cite, it ran through the CRAAP test. I'm not sure if you remember that crap test. Uh, the crap. It's like C R A, A P CRAAP test. And one of the things that it did to kick out a source is to basically go, okay, does it have clear bias? So you couldn't cite that, uh, in your paper? And I think the same kind of thing's going on with AI. It's like, do we need to be listening to Dario and Sam Altman on the impacts of technology on the economy? Probably not, because they have a clear bias to say that. And I lean more optimistic. I look at like, all right, who are all the top economists out there and what are they saying about the impacts of AI and technology on the job market. And it's really, really clear to me that it's sure there might be some short term pains, but long term we're going to get job categories we've never seen before and a ton uh, of new opportunity to open up. How do you see it?

Adam Evermescew: Yeah, no, I mean I see it similarly in the sense that, I mean even if you look within companies like OpenAI and Anthropic, like they, they have customer education teams, they have digital CS teams, they have skilled team, right? Like they're, they're doing this work and I know, well, I don't know, like, I believe that like maybe if you ask them they might have a more nuanced point of view on like why they've built up some of these functions in the sense that hey, you know, when AGI, uh, really hits and manifests, maybe we don't need any of these roles anymore, but if they have them now, first of all, I think that speaks to even the companies who are on the frontier are still doing this work and there's still a need both for this function and for humans who have expertise in this function to do it. But also what I'm seeing is in a lot of cases you have now these small customer education teams who are AI enabled. Uh, we actually had uh, an episode that just aired recently which was about running uh, a customer education team of one with an agentic education approach which was super uh, cool with Jess Katz and listeners, uh, uh, check that one out as well. But um, those teams have always existed. It's actually in some ways the anomaly in recent history is for customer education teams to turn into these gigantic mondo teams with like such huge headcount. And that might be in some ways related to the scope of the team or the level of delivery that is needed or covering across multiple markets or uh, doing you know, live uh, uh, paid training and things like that, like actually needing, like needing to fly people out places. Well that, that still exists, right? It's not like the humans have gone away and as long as humans exist in businesses, like those things are still going to be needed but for the, the scalable stuff. I mean at least the way that I'm seeing it, small, lean customer education teams have always existed. They will continue to exist and the technology that we have now is just going to make those same teams more powerful. And in a way what that's hedging against I think is more like those teams overinflating over time.

Kelly Mullaney: Yeah, I see it the same way I could have probably given the Said the same exact thing, I said it before. So I love that we're aligned there. Yep. I think, yeah, people are going to be managing agents and how content gets generated will probably shift over time. Meaning like, okay, take like a release note, um, like a good basis of that is going to be the why and then you're going to have and customer education. When you insert yourself into the sdlc, you know, you're asking all those questions along the way to fuel what you're going to do and then you build your output and it's not like 100% of the time that we're the first person asking that question. It's probably happening early in the sdlc. The product manager probably is in grooming, telling people why we're building this. And so if you can build the systems to capture that knowledge, you can generate things like that automatically. But what you do after that is going to be a human running an agent, I believe.

Adam Evermescew: Yeah, I agree. And like, to that end again, one of the reasons why I really wanted to talk to you is because you have been building and sharing a lot of these, uh, you know, agentic or Vibe coded products, uh, and experiments that you've been working on. And so I would love for you to walk me through, um, you know, let's, we can take a few of them as examples and first of all think about, you know, how you got the idea to do, to do some of these. Would love to hear more about, you know, some of the different tools that you've used. Because, you know, you know, kind of scrolling through the timeline and looked overall, uh, your, your AI tech stack seems to have changed a little bit over time too. Um, and also then thinking about like, what do we do after, um, you know, these products are released and now we're responsible for maintaining them over time because I think we should probably talk a little bit about like the total cost of ownership of uh, you know, Vibe coded products that we build and maintain versus, you know, paying for uh, SaaS subscriptions. And you know, there's a, there's a, there's a push and pull there.

Kelly Mullaney: Yeah, yeah. The buy versus build math really, really changed with what Cloud desktop and Opus 4.5. So the first thing I do when I approach a build is I'm, I'm really mindful of the fact that like 95% of AI initiatives fail. I think that was like, like Harvard or something came out with that. And I think it's because people are building tools for specific use cases or they don't enable on it appropriately. And I really spend time thinking about what's a tool that is applicable to everyone that everybody can use. An example of that might have been, um, I built like a help center. I was basically like, okay, do I need to keep using Zendesk guide or do I need to go and buy like Mintlify? My idea from that was what if I could automate writing how the how. Like, maybe I don't get the positioning stuff when, when writing, but what if I can write the how? And my thought process behind that was, well, what knows how something works, what the code does? Uh, the idea that I wanted to solve for is if somebody could check in code and I monitor the git repo, can I pull the code, bring it into my context window and generate an output? That's what I did in one of the things that I showed is just checking code. It goes through, it writes a, uh, release note and it writes everything. Building on that, ah, what if I went and built a connector over to JIRA and I grabbed the ticket in the EPIC or the Initiative and pulled in some of the why, like, could I get myself 80% there? And that's really what I want to focus on.

Adam Evermescew: Yeah. And this was one that I was super interested in as well, because it looks like, I mean, first of all. Well, I think you even mentioned you were thinking, uh, of releasing the source code at some point. But like, um, I'm interested in the tooling that you used for this and kind of how you went about building and iterating on.

Kelly Mullaney: Yeah, so I build generally in cloud code, desktop. Uh, the database I often use is Supabase, but I'm flexible on that. And that's just backend as a service. And it's setting up the database in English in the same way that I'm generating the code. I'm just talking to it. The plugin that I prefer for my SDLC is called Superpowers. And what it does is it does a really good job of asking you questions about what you're building and understanding the problem so that when you get to the end, you don't have to iterate on top of it. And I like it because AI can take an input, you give it, but that's really limited by the things you think to tell or questions you think to ask it. And what Superpowers really does is it asks you a bunch of clarifying questions about what you can build. And oftentimes those are things you haven't even thought through when you're going to approach the problem. So Superpowers, Supabase and then Claude Desktop and then oftentimes I'll bounce my ideas off of other LLMs just to refine my thinking. Like I just will use another one and be like okay, this is what Claude's telling me. What do you think Gemini? And that really helps refine my thinking.

Adam Evermescew: Yeah, yeah, that's really cool. And so in this case then you have it uh, actually scanning um, um GitHub as uh, uh, different commits are happening to see what's actually updating in the product and then use that I'm assuming with some predetermined uh, prompts as well to then know like okay, based on that, here are the things that I should look for, here are the things that I should try to document on it. Or if someone also uploads like a PRD or M a changelog or something like that, then it can do the same. Is that generally how it's working?

Kelly Mullaney: Yeah, that's basically looks at the commits and it determines if it's a material change. And for me what that means is if there's a front end component or it changes some meaningful functionality. There are a lot of check ins that are going to be bug fix that some companies advertise, some don't and don't matter. So it analyzes that and then it kind of runs it through essentially prompt engineering as you said. It basically is like okay, here is exactly what you're going to do and create the outputs but it also goes out and fetches uh, from some other sources so it looks at like Jira for example and pulls and stuff there. And then I built in all of the different style guides so you can define how you're going to, to write. So an example would be like pull in the Microsoft style guy for technical publications and pull in Chicago and then write it that way. So yeah, it just writes the whole thing. Is it 100% perfect? No. But is it 80% there?

Adam Evermescew: Yeah. So that raises I think then a different interesting question. Um, I want to come back to a different one that happened in my mind. But while we're on this, so what's happening afterwards? How are you doing that additional 20% refinement or is the 80% good enough to print? Are you doing manual review and editing? Are you running it through another agent that's double checking? Tell me about that.

Kelly Mullaney: Yeah, I believe strongly in human in the loop. And so it's just picking it up and then verifying its output is essentially it. I think one thing that occurs to Me is I don't know if help centers as they're designed today are going to be needed in the future in the same way. And if tech writing isn't going to be the presentation layer any longer, it's going to be the training layer. So it occurs to me that a AI can do something that just fixed text cannot, and that is it can ask you a question. And as a tech writer, that has always been the bane of my existence, is that I can't speak to you like I can't talk to my audience. And my belief is that I can write the right question. So I picture a world in which the tech writing is not actually surfaced to the learner. And what happens is it writes customized articles and help content for you through its, you know, like through its interface, like normally help center, uh, you know, how can I help you today? Similar exactly to how like Claude AI works. It's the same kind of thing. But if it can ask you questions and then build the knowledge base articles just for you, so much more powerful.

Adam Evermescew: Yeah. And so what I hear from what you're bringing up in this case as well is that increasingly, especially when we talk about documentation and, well, yeah, let's say documentation and help centers, I guess, is that just as much as we have been writing for human users of the product, we are now writing more for what a lot of people are calling like aeo, uh, or let's just say searchability and discoverability for other LLMs and AI products to ingest. Which means that on one hand the accuracy still needs to be high. Right. We can't not have a human in the loop process because AI is still prone to hallucination or might even just not get something entirely right because it doesn't have the context for it. But that the output. What I'm hearing from what you're saying is that it's not as much optimizing for human readability anymore as making sure that can actually get ingested into other AI products that can contextualize, um, and then support the user based on that.

Kelly Mullaney: That's exactly it. It's basically building the corpus that AI is going to feed from picture. I don't know, you're an educator at HP and you're handling printers. If somebody can't print, you can write a knowledge base article or build a video on possible reasons for that. But are you going to get down to the edge case like somebody's, somebody's like on vacation and they're thinking that they can just print to their computer at Home. You're probably not going to have that in like, number one on your troubleshooting article. But AI can get there. And so it's going to make us so much more powerful. But we do need to write essentially the corpus where it's going to have enriched company information in there to answer it properly. So it's not guessing when it doesn't have context.

Adam Evermescew: Yeah. And I think that's super cool, like looking at some of the features, um, that you incorporated into this and maybe you can expand on some of these too. It's not just about having that content generation workflow about monitoring all the commits and deciding what to document and producing the initial documentation and going through the editorial workflow. Like, that's table stakes. Um, that's already awesome. But that's also table stakes. Um, but you've also incorporated into this, like an AI chat assistant, so the customers can ask those questions in plain language and get answers. And then based on that, also look at search gaps to figure out where there should be additional documentation or new questions to answer. And then of course, like an analytics layer on that too, to be able to actually have some observability for what people are doing in those docs. I think that's super cool.

Kelly Mullaney: Thank you. Yeah. My idea at first was basically going, what if I could just make a help center AI, if fixed articles just went away and it was just going to generate them on the fly. As I started using it as a user, I was like, yeah, that's a heavy change for somebody to wrap their mind around. So then what I built is, I built, let's do the AI layer on top, uh, of a fixed help center, so that way you could have your choice. And I felt like that made adoption a lot easier for people.

Adam Evermescew: Yeah. Well, because I would imagine that if we think about the technology adoption curve, even if it's possible today to generate a completely bespoke help experience, there's still going to be a lot of customers out there. When we think about customer preferences from a behavioral standpoint, they still want to know. Here is a doc. I've gotten to it. This is the source of truth. Someone has looked at it. I know that this is right.

Kelly Mullaney: That is right. And I think the other part that help centers do is if you have a left nav or some kind of navigation structure, it shows you what's possible. It's like, oh, I see this section. I didn't know I could do that. So you learn by browsing the page in and of itself. So there is value in that, that I don't know how to solve for. But I do like the AI component that in and of itself it is self enabling use Claude for example. It's always saying like, I don't know, you give it a spreadsheet, it's always like hey, do you want me to build this into a PowerPoint deck or remove duplicates or something like that. It's always prompting you and showing you what you can do next. And I believe that if we can build that into our help centers, adoption can go up just because you're learning by using it.

Adam Evermescew: Yeah, and I was thinking even as you were saying, um, you know, kind of that last example, that increasingly that will probably start to also take the role of the left nav and understanding more of the art of the possible. But like the part to me that always feels a little bit tricky about saying oh well this means that um, you know, uh, fixed help centers will just go away eventually. Well maybe eventually on like on some timeline. But um, I. So much of the customer's experience with our products is contextual and if you are in the help center because you're having a uh, moment of uh, crisis, that's very different from you being in the help center because you are trying to like discover a new use case or maybe you're like more emotionally open to it at the moment. So I think like right now viewing it as an, and not an or like not a replacement for the products that we've already built is probably like also like even just a helpful way to look at it. Maybe not even like a stopgap.

Kelly Mullaney: Yeah, yeah, I agree. I think that Anthropic has a help center. I use it all the time. I can ask the questions through cloud if I want to or I can go to their help center and I do. Um, it's hard to change those motions. And then help centers, they do have some of their value. But I think you do need to offer the choice. Like let me give you a better mechanism for consulting and then let me get you some pre written release notes for example that uh, are written with the positioning the company wants.

Adam Evermescew: Yeah, well and that brings up an interesting point too because even working with Anthropic, for instance, you can ask Claude to answer uh, a question for you. You can look at Anthropic's help documentation and you can also work with their support team. But even Anthropic is not necessarily rebuilding something like uh, Intercom fin. Right. Like for, for the AI based support that product exists. It's uh, well now I think they actually have their, their own model powering it. But like, the reason I'm thinking about this is also because we talked about build versus buy. And I'm curious like when you think about making these sorts of investments for your team or like whether it makes sense to build and maintain a help uh, center product. For instance, instead of working with you named a few of the vendors earlier like Zendesk, Mintlify, et cetera. How do you think about that calculation and how do you think about the ongoing maintenance of products that you actually build yourself using AI?

Kelly Mullaney: Yeah, so I generally look at it through the lens that AI can get you 80% there for most things. It's not necessarily good at the design side of things, but it can get you 80% there. Like I was able to replicate 80% of Zendesk guide in a week. And I keep that in mind. And the other thing that I look for for potential disruption is if you're just a company that is just a layer on um, top of a frontier model, I think that those ones are easier to disrupt. And a good example for that is like competitive intelligence.

Adam Evermescew: It's like ultimately a clue competitor, right?

Kelly Mullaney: Yeah, it's like going out and just looking for market, market signals and then finding a way to get that to you. Sure there's some special sauce and like probably having subscriptions to tools that I don't have or like you know, being able to scrape LinkedIn without getting kicked out but at the same time it's ultimately just doing agent searches and then pulling it in. And that was one that I actually just spent last weekend doing. I just basically made a uh, Slack channel called Competitive intelligence. I have 37 agents that uh, manage agents, cloud manage agents that go out and look monitor all of our competitors and then give a daily intel brief into a Slack channel. And then I have a uh, self improving feedback loop for that where babies could be putting forward over the message to a feedback channel and then the bots just learn from all the feedback that they're giving. Anyway, I just look at it like ultimately if we can get you 80% there and you're charging us 30, 60 thousand dollars a year, I like to revisit that math.

Adam Evermescew: Yeah. And basically say would I take 80% of this, that I don't necessarily have to put a ton of work into maintaining versus having the hundred and pay this much for it and still in some cases the vendors aren't going to uh, deliver everything you might want out of that product anyway.

Kelly Mullaney: Exactly. Yeah. I think it's going okay. Is this one where I want to be able to control the roadmap? Like when you buy SaaS, you're buying into a roadmap and some companies haven't produced the roadmaps that I value. Like speed is such a moat now and if they're not cooking along and stuff, well, maybe I can move faster. And that really weighs in. And so there are companies and tools that I've looked at that aren't moving and then there are tools that are part is a really good example of a company that moves quickly. I would be happy to buy into their roadmap versus a storyline or something like that, which I see is moving along slower.

Adam Evermescew: Yeah, this actually brings me uh, to something I wanted to ask you about as well. So, uh, thank you for the free transition which I'm now stepping on. Um, you, uh, a couple months ago, um, also built basically an elearning course that you Vibe coded, um, without using Storyline or one of those products and you built it in Claude code. And so I'm curious first of all again, maybe take us through a little bit of the process of what building that app was like and I mean it probably would even look a little bit different today with cloud design. But um, how you might think about that versus like using a product like Parta, which you, you also just mentioned.

Kelly Mullaney: Yeah, what I, what I would say is anything that is a critical business function. I, I'm not to the point where my confidence is that uh, Vibe coded app is going to work perfectly all the time. I'm one person. I don't have all the testing resources and everything to really refine it. So would I trust my critical infrastructure to it? Not yet. Who knows what it's going to do in the future, but not yet. And I consider elearning courses to be critical infrastructure to my team. So in that case, yeah, I would choose an enterprise grade vendor, um, for it. That being said, my mental model around doing that is ultimately that we don't need to be limited by what tools say that we can do. Like, you know, if you, you see a rise force, every rise force looks. Do you ever feel like your product

Narrator: team ships faster than you can build training? Yeah, me too. That's why I've been checking out what the folks at Learn Experts are doing. Their AI platform called Leah, uh, takes her existing content, docs, slides, wikis, recordings and turns it into full training programs, publishing multiple modalities to nail blended learning

Kelly Mullaney: for your customers 85% faster than traditional methods.

Narrator: And get this uh, when your product updates, Leah, uh, can update the content faster,

Kelly Mullaney: existing outline, even your end of

Narrator: course, test or exams.

Adam Evermescew: Yeah, Learn Expert shares that they're also. I guess what I'm hearing you say is there's also a distinction between now

Narrator: that's time back to actually focus like

Adam Evermescew: on one hand, let's be honest, to

Narrator: finally take a lunch break.

Adam Evermescew: The legacy products like Skilljar and others like let's say our streets rise and things like that, like check it out, the limitations of those things

Narrator: very clear.

Adam Evermescew: You see the limitations, what can be done there. And I think you also even called that out in your post. Right? Like why are we limiting ourselves and why are we also continuing to pay for those tools if they are actually meaningfully impeding the experience for our learners and what we can create and creating additional work, um, uh, for us to maintain them over time. But then we can kind of go one step forward to say, okay, well if we're that limited by some of these legacy tools that are still critical infrastructure, can we work around them by using either like in your case, CLAUDE code or like now? I think CLAUDE design would also be able to help a lot with these types of use cases. Um, and do that. But there's kind of a third step, if I'm hearing you right, to say, well, but let's remember they're still critical infrastructure and we still need to be able to produce these reliably. We still need to make sure that they're uh, replicable on brand that like if we have multiple people in the company working with them, that they can be updated and maintained in time. And that's when again, like the script would flip to say if there are vendors out there that can move fast enough to not give us those limitations, then it's worthwhile again to actually go with a vendor instead of trying to build and maintain all of these as one offs. Uh, uh, is that the equation?

Kelly Mullaney: That is exactly how I'm seeing it. I think a lot of these legacy tools have gotten comfortable in the fact that they felt that they had a moat and what that stopped is from them moving quickly. So you have a choice with it now. You can partner with somebody who is moving fast and is using AI to generate a lot of their code and testing it and treating it as real enterprise grade. Or you can bolt on top of those legacy tools. But I think the moat is moving fast. It's also having great service and great infrastructure. And so for tools that maybe you don't need that, uh, sure, Vive coding. 80% of the way there might work. But am I going to trust building my full E learning course in Claude code and having that work flawlessly 100% of the time? No, I would probably choose an enterprise grade vector there.

Adam Evermescew: So for instance, when you built that, you built it as an experiment, but not necessarily intending to fully productionize that more to challenge the legacy LMS and authoring vendors.

Kelly Mullaney: That's exactly it. It's basically to go, uh, you know, Claude could be coming for your lunch. So speed up. That was really the warning shot I had. And then it's also, you know, there are companies that can't go out and uh, afford these tools. You know, they don't have the funding for it. And so would I rather have something that got me 80% of the way there or nothing? Of course everybody's going to choose to 80%. And so it was to inspire those teams just getting BO draft to go, hey, you don't have to do nothing. You can't do something. Let me show you how.

Adam Evermescew: Yeah. Which goes back to something we were talking about at the top of this call, which is so many customer education teams especially who are still in an early state, are still customer education teams of one. Uh, they're still working really lean. They still might not have the tools. And I'm even thinking, for example of when I, I've told the story on the podcast before, so I'll keep this short. Like when I was at Optimizely and we didn't have an LMS yet, I came in, we were a customer education team of two and we were basically just scrambling to make uh, the help center and video. We didn't have the AI tools at that point or, well, nothing that would be recognized as AI today. And um, we built the first version of our academy basically as a prototype on an instance of Zendesk, uh, help Center. And so if we were in the same position today, we could make a much higher fidelity version of that, either CLAUDE code or let's say a similar tool. We don't need to make this just about Claude, but you're right, it is possible. And the other point that you make in your post that I really like is that even if you look at a lot of the traditional LMSs that are out there that are reliant on scorm, they're only giving you course enrolled, course completed, uh, maybe the score of uh, a quiz or something like that, and the fact is now we have the ability to measure so much more. And I think the vendors who win in this space are the ones who are going to outpace what we can now collect ourselves and not just stick to that kind of, you know, to your point, like, very limited imagination of, like, what we could track in the past.

Kelly Mullaney: Yeah, I think, uh, you know, the power of lms, right, is the reporting. I mean, like, at its end, it's got a presentation layer and a, uh, reporting layer. And if you're just like. You can't even track, like, session time in a lot of these, it's hard to track, like, oh, how long were they actually in the course? Like, you're like, how did they pass in two seconds? When you realize, oh, if they were on a different tab or something like that, it messes. It messes up, uh, your metrics. And with AI, we can do different types of reporting. I think a lot of people probably in the space have had the idea of going, well, I can just write it all with API calls in the future. And then you actually, five years ago, you tried that, and you realize that is really, really, really hard. And hiring the staff in order to fuel that vision is really, really hard as well. Well, now it's not. Yeah, you can hire Cloth or you can hire OpenAI or whatever you use.

Adam Evermescew: Yeah, totally. And so to that point, maybe we can think about, um. Well, you actually described several different technologies, even within the CLAUDE stack. But we can genericize them if we want. Right. We talked about CLAUDE code, we talked about CLAUDE design, we talked about managed agents. Uh, we talked about CLAUDE desktop. Um, I think a lot of people know claude, kind of the consumer chatbot product. Um, if you're talking to someone who's a team of one and maybe getting started and wants to explore these tools more, could you give a quick rundown of maybe how some of these tools are similar or different from each other, or how you might decide which of those tools to use to accomplish your goal?

Kelly Mullaney: Yeah, and that is, uh, a real challenge for a lot of people that I see is basically, okay, what do I use to solve what? And I'll just quickly walk through the anthropic stack because I think that is probably the one that gets a little bit confusing. You have CLAUDE AI, you've got CLAUDE cowork, you've got CLAUDE code, you've got CLAUDE design, you've got managed agents. So what are they? I'll walk you through each of them. CLAUDE Chat is just CLAUDE Chat. That's where you ask it a question and you get a reply. CLAUDE code. And especially the desktop, there's kind of two flavors that there's a desktop app that you install and use and then there's just uh, a terminal based. The one I'm talking about here is the desktop app part of code. And that is where you generate code in plain English. You just basically go, hey, I'm thinking about building an e learning course from this script that I want to be reportable. Can you help me build that? You just talk to it just like that. In fact, I would say that I use speech to text almost exclusively when vibe coding just because I want it to almost have access to my ID or my verbal thought or like my thoughts. And so anyway, that's the one that you're generating code Cowork is kind of a bridge between those two. It, it is basically the idea, uh, it's built on top of plot code and it's the idea that it can help you automate workflows when maybe you don't need a front end to the software you're building. So an idea of that if we want to go, is like, hey, run this skill, which is kind of like a pre written prompt and go out and do a competitive intelligence and it will do that for you. And you just call it by typing slash and going. You can also make your own skills. Then on top of that what they have is, are called plugins and plugins can combine skills and then what are called MCP or connectors and those are the systems at which it can connect to. So an example of that might be like Slack or Jira.

Adam Evermescew: So the skill works like when you were talking earlier about having your uh, Help center product scan the code base for commits like that would be through an mcp.

Kelly Mullaney: That's exactly it. An example for that one might be like, hey, go out and use the Atlassian Rovo mcp. Go grab the requirements for this EPIC and bring them in and then execute a skill to write my release note. That's just something really, really simple. And then the idea of these plugins is you're sharing them out to your. Then we have Claude design. And Claude design is really at its core kind of like a Gamma competitor. But the idea is that AI is really, really good at writing code. But for some reason it's decided that design is not what I'm excellent at. And an example of that might be like I can write that entire Help center product in a weekend, but then if I wanted to format a pixel perfect PowerPoint deck or slide deck, it's just like, no. That was their answer to it. And it also does that in Software engineering and what I mean by that uh, is that you can tell what a vive coded app looks like. They have their tells and the idea of that is well can I load it up with my design system and how we've made enterprise grade design and then just sync that down into our products to give it our unique flavor and polish on it. And then finally managed agents are basically just it's code that you can deploy to their servers which saves you a lot of time for the infrastructure. I think probably long term it's going to be like an open clock answer but right now it's just shared memory and basically uh, agents on it.

Adam Evermescew: Yeah. And being able to like run some of these automations for instance without having your laptop open and running all the time or like waiting for it to catch up when you open the laptop again is.

Kelly Mullaney: That's exactly it. So in my competitive intelligence right like I don't want to keep my laptop going um for like if I had it on like a cowork skill or something like that I wanna, I wanna be able to just you know go to sleep and then wake up with my digest and my, my Slack channel. So it was a perfect answer for

Adam Evermescew: that and, and like I think again it's probably also worth calling out. We just walked through the entire um, anthropic stack which like again I also agree is like a super useful, useful one and also for a lot of the work that we do in many cases super appropriate, super sufficient but um, for our audience as well. I know we have a lot of uh, Gemini users out there who are also then throwing things in notebooklm generating uh, uh images with nanobanana et cetera et cetera. I know a lot of people shop around a little bit between the different uh providers and I even saw that you in the past uh, you had an example and probably would take me a moment to find which one where you were using a different stack as well. I think you still had Supabase and uh, a couple of the other and lovable. Yeah. So I'm kind of curious how you look at some of the different alternatives out there and how often you're maybe switching between the alternatives versus going with I guess the consistency of sticking within a specific stack.

Kelly Mullaney: Yeah, it's something I struggle with. The arms race of uh, uh AI is real gosh maybe eight months ago or something like that. ChatGPT was the front runner and everybody was all pro OpenAI and then what was it? Gemini came along with their newest version like a month later and dunked on OpenAI and then after time, Anthropic came out with Opus and it was better. And so there's always going to be an arms race. And the way, the way that I see it is you have to be open to tool choice. Um, because if not, you're always going to be behind if somebody pulls ahead. But if that's not possible, I'd say where the models are today, I think they're all really, really good. Yes, Claude probably is going to excel at coding and they, a lot of the other tools don't have a perfect answer for that, but they will over time. And for a lot of uh, what most people are doing, I think just find what your company issues you and work for it. But if I were starting today and I had choice and I was in my personal life, I would probably look at Claude right now just because its toolset is pretty robust.

Adam Evermescew: Yeah. And you also bring up a good point in terms of looking at what your company actually has approved. Right. Because you don't want to be like putting a bunch of company data into like a third party party, uh, third, uh, party A.I.

Kelly Mullaney: that is real. I think, uh, that's the complaint I hear often from talking to people at other companies. It's like, well, we can only use Copilot or whatever. Um, and so there is frustration, I think, with people about what tools to be given, being given. I'm lucky at Juniper Square that I get all of Claude and every employee gets all of Claude. But at the same time, um, there was a time when Claude desktop didn't exist and CLAUDE wasn't the best. And we were doing great things just using OpenAI and using Gemini. So you can get value from any tool. It's not the tool, it's probably the person using the software.

Narrator: No.

Adam Evermescew: And I think again, you've shown that really in your approach about thinking how far you can get towards a use case that you want, want to accomplish, throwing different combinations of tools at it to get the job done, that shouldn't be the limitation. And at the same time, for anyone who might be a little bit nervous about AI or feeling like, oh, I have to keep up with all of these tools and I have to know all of these things. No, these are all options that are out there. But if there's something that you've started with that you're more comfortable with and that's your way in, that's fine. Or if your company uses one thing and that's what you have to use to solve the problem, that's fine. And I think one of the great things that you've shown throughout this conversation is that it really is more about your thinking about the problem statement and then figuring out how do we iterate towards solving that, how do we get towards that 80% than knowing the exact capabilities of every single tool.

Kelly Mullaney: Yeah, I think that in customer education in particular, I think the most well aligned to take full advantage of AI. I would say that most of us aren't coders and so we see opportunities and not um, the potential problems that you have. And that optimistic mindset is really, really helpful. And the second part of it is that customer education has leaders that are intellectually curious. I don't think that a lot of people got into customer education going I want the top paying job in the world. If not they would have went into sales or something like that. And so they have that intellectual curiosity. And what makes good AI users and bad AI users is the questions that you ask. And customer education leaders are going to be so excellent at asking the right question. And I think that's really the core of it. And then the other part of it is the only way to get into Carnegie hall is practice. And we've talked about how AI ah, is self enabling and just use it and it's okay if you fail and your first coded app is a throwaway app, but just get your app backs and use it. Use it, use it.

Adam Evermescew: Yeah. And again love that you jumped into it early and we can see what you're doing now as a proof of that and I think still as a proof of for customer education in the future what that human in the loop is still going to look like because yeah, I mean as much as we can say there might be a day where all of this is fully automated, hard for me to see that coming so soon that any of us needs to be so worried about that. Probably better for us to figure out how to be creative with the tools that are available to solve um, some of those parts of the job that have been traditionally difficult or limiting for us either because of our access to information or because of the tools available or because of where the knowledge sits like whatever it is. Right. Like I think like proper use of these tools and creative use of these tools is going to make things easier for us even though like yes, in customer education, like we're still always going to have to be scrappy. It's, it's. But that's been the case, you know,

Kelly Mullaney: that's always been, that's always been the case. It's Just going to amplify your impact and make that uh, make that easier. As you said, look at what you know, Anthropic has customer education. OpenAI has customer education and they're using models that we don't even have access to. They're way ahead of us. And they still run the function for a reason because.

Adam Evermescew: Well, and I know the people on those teams, like they have incredibly talented people doing that work. So it's not, you know, it's not, it's not like uh, they, they've like devalued that uh, function either. Like clearly they see the value in doing this.

Kelly Mullaney: Well, exactly. And for those doing the video based learning too, I think a good signal is. What did OpenAI do recently? They probably had $10 billion in their video side of their business and they just discontinued it because they were basically like, okay, the compute costs in order to do this are just too expensive. And that makes sense to me. It's like how many images would you have to create to create one second of video? What is uh, that like 60 frames or something like that? The compute costs are gonna outweigh everything else. And that's why they just threw you know, $10 billion or something away. And so it's not even coming for video yet. It's too expensive.

Adam Evermescew: No. And at the same time, like for those of us who are producing video based education, for instance, the tools that are available to us now, like Cluso Heygen, not uh, even getting into like some of the voice Like 11 labs, all of that. Right. Like there's so many great tools now available to us. Even though you're right, like OpenAI deprecated, what's it called, Sora. Yeah, that um, maybe in the consumer world that cost benefit analysis doesn't work out, but we do have at this point pretty fantastic point tools available to us to make that side of our jobs easier. So if anything comes back to what you were saying about it, more unlocking possibilities and overcoming some of the limitations, especially for teams that aren't fully resourced with huge production studios and uh, uh, um, full teams working on this as

Kelly Mullaney: a. Yeah, that's exactly it. And then I would say that it's also imagining a world in which engineers are 10 times more productive and code is coming out at 10 times the rate and it's going, does my workflow work in that area? Does it scale and is it going to lead to like cognitive overload? And so it's reimagining what customer education could look like at a time where people are pumping out 10 times X as much code. And then as you said, how do I do that with a scrappy team?

Adam Evermescew: Yeah, totally. Well, Kelly, like, again, thank you for your time and sharing your expertise and also being, for so being so public, uh, sharing on LinkedIn what you're building, maybe just to close things out. Um, do you have any other advice that you would offer to other customer education professionals or customer education leaders who are, um, maybe getting into AI, maybe have some early use cases, but really feel like they want to go deeper and maybe just don't know exactly how to do it yet?

Kelly Mullaney: Yeah, I would say the first piece of advice that, that I would give is don't try to like, insert AI into existing processes. Like, you can go, okay, here's my sdlc. And let me just try to, uh,

Adam Evermescew: which I think we should probably define at this point. Right? We've said that term a few times. Software development life cycle.

Kelly Mullaney: Yes, sir. Okay, great. So how your software is being built, like, don't just go, okay, I'm going to insert AI in there because it seems like a good opportunity. It's to imagine what the world for education and your company could look like with AI infused in it. And the second part that I would say to people that want to get started is don't be afraid. Just jump in and use it. And know that your education mindset is going to make you so powerful with this stuff because you're going to ask the right questions and you're going to understand how, how users think extremely. Well, sure, your early stuff might be. Might be throwaway and that's okay. There's learnings in the failures. I've done so much throwaway work and burned so many tokens, but I learned every single time and it's helped me level up. So just use it.

Adam Evermescew: Amazing. Well, Kelly, again, thank you so much. I agree, by the way. It's all scary until it's not. Yeah, uh, well, cool. Thank you so much for joining us and for sharing and um, I mean, again, like you, you inspired me in terms of, uh, thinking about what can be done with AI in our field. And I hope for our listeners that this was also as inspiring, if not more. And if you listeners want to learn more, we have a podcast website at, ah, Customer education. You can find all the stuff there that you would find on a podcast website. Uh, and you can also find us on LinkedIn. Kelly, where would you like to be found?

Kelly Mullaney: LinkedIn is great. Okay.

Adam Evermescew: We love LinkedIn and we also love Alan Kota, who does our theme music. You, uh, can see him, uh, play in, like, six different bands if you are in the Austin, Texas area. So definitely, uh, check him out. And if this helps you out, please subscribe to our show. Wherever you subscribe to podcasts, and especially on, uh, Apple Podcasts, we would love a five star review. Um, and if you are listening right now and you're like, I haven't done that yet, um, maybe you can take this moment to open the app and just leave a cheeky little five star review. There's no time like the present, um, because that really helps us keep this thing going after so many years. And of course, to our audience, thanks for joining us. Go out and educate, experiment, and find your people or your managed agents. Thanks, everyone. Thanks, Kelly.

Kelly Mullaney: Thank you. It's an honor.