Stop Guessing on Launch Day: Meet Your New Message Testing Framework w/Shannon Kearns

Shannon: This is Zoe, and you're listening to Product Marketing for your.

Speaker B: Welcome back to another episode of Product Marketing for your. Today's guest is Shannon. Shannon is a GTM M consultant who specializes in helping B2B and B2C tech companies launch and reposition products. And she's got a very specific obsession, making sure your messaging is actually tested before you bet the launch on it. Because here's the thing. Most PMMs pressure test the product, the positioning deck, the sales pitch, but the messaging itself, it just ships. Shannon's got a framework that fixes that and some real numbers to back it up. So if you've ever crossed your fingers on launch day, hoping your messaging lands, this one's for you. Okay, quick break and then we'll jump into my conversation with Shannon.

Adam: This season's partner is PMM Camp, the number one resource for product marketing leaders with access to newsletters, leadership frameworks, and a vetted peer community. It's where small aren't senior PMMs like you go to become indispensable. And most of this season's guests are actually campers themselves. So consider this your sneak peek into what life is like as a camper. Go to pmmcamp.com and join me and 9000PMMs and counting on our journey to become the next wave of product marketing leaders.

Shannon: Product marketing for you. Product marketing for you.

Adam: All right, I'm excited this conversation because I spent almost two years of my life getting in the trenches on a B, testing and testing in general and understanding what resonates with people. But, um, compared to that, I am still just a little basic kid when it comes to what Shannon, my friend Shannon here has developed over her time as, uh, an expert in this. Shannon, welcome today and thanks for dropping by.

Shannon: Thanks, Adam. I'm really happy to be here.

Adam: Sweet. I'm going to get right to the good stuff because there is some goodness, some stories, real life stories. You have a template that you use and developed over time that you're going to share. Um, I do not want to rob you of any precious time, but before we get started, um, you've done, I think, over 15 plus launches and counting that might even gone up since we talked last.

Shannon: Probably not since we've talked.

Adam: That's what I'm talking about. What is it about message testing specifically that you just can't let go of?

Shannon: Yeah. So for me, it is the fastest way to move from ideas, hypotheses, gut feel, whatever you want, call it into actual data backed things that work, that you can then scale. So When I'm thinking about product launches specifically and things move so quickly these days, especially with product launches, you don't always have time to get a ton of insights on the front end. You don't have a ton of time to necessarily talk to customers and things like that. So um, it's really helpful for product launches to be able to do some message testing in the market as you are launching a product so that you can very quickly test multiple angles, learn what's working iterate scale that on the acquisition side, but also bring that information back to the rest of the team. And I think for me we do a lot, I see a lot of message testing on the product side. Maybe in app, I've seen it done with uh, even a research team prior to launch acquisition does it 24 7. Right. But when PMM is owning it, especially for a launch or a major repositioning or maybe you're moving up market, it gives PMM that central point. PMM is the central point of contact. And we are then able to feed those insights back to the rest of the organization in a really structured way so that everyone is learning about this product, product and what's resonating as you're rolling it out. So for me it's the feedback loop also it's the need for product marketing to own it. And it's just so much goodness that can be learned from message testing if you set it up correctly.

Adam: And that's, that's one of the biggest things too when, uh, over the last, I'd say four to six years, we're in, we're in 2026. Right. Uh, that the idea of a launch doesn't end on launch day. It was that way for back in the day. Right. But now back in the day, good old days. Yeah, the good old days, uh, where we had to handwrite everything on paper. Um, the idea of statistical relevance plays a key role in understanding whether or not a B testing is successful. Um, but that gives you some extra leeway and understanding of like, hey, it's not the launch day, we have to know these insights. It's extended out which is great. And if you're interested in statistical relevance and a B testing, there's a ton of great resources on the interwebs that we will not get into that this year. But for me, you obviously have a lot of experience and you are hyped, uh, up about it. Was there a situation specifically that made you say hey, yes, this and made you kind of like figure out what we're going to be Talking about in a few minutes around your framework.

Shannon: Yeah, so I use, I come from a demand generation background. So on the acquisition side think like Google search ads and meta ads. Well, before it was meta, um, might be aging myself but um, I come from that background but specifically from a product marketing perspective. I was in a situation where my hand was kind of forced. We were rolling out a product that was more so being developed to address some limitation in the market. It wasn't really grounded in consumer insights. It was kind of a means to an end for the business. So I'm over here like about to brief my marketing team on this product that doesn't have a lot of consumer insights driving it. Um, I'm like, I don't really know what to say about this. It was a direct to consumer product and so I was like, is there a way that we can test this in the market before we go all in on it? And so I worked with the acquisition team, I worked with the marketing analytics team and we figured out a nice little way to use the acquisition side of the business which again direct to consumer, you get super fast learnings. And we stood up a couple of rapid tests and then we use that to inform the broader rollout and it worked so well that now especially if I'm in a PLG organization or it's a direct to consumer with a quick sales cycle, I love having this message testing touch point because again we don't always have the opportunity to get all of the insights up front to get the perfect message to market. So as product marketers we have to understand how to continue improving our message. Because to your point, the launch does not end with on um, launch day. That's when it starts. And these feedback loops need to continue happening for two, four, six weeks thereafter.

Adam: That's great. And you touched on the B2C direct to consumer side of things. As a B2B marketer my whole life, um, I've always been jealous. I love and adore B2C marketers, especially Prop marketers. Do you find that there's fundamentally any difference when looking at message testing, especially when it comes to a launch between the B2C and the B2B space.

Shannon: So I've actually done less in enterprise B2B. I think it has more to do with how the buying cycle happens. Again, if it's more of a LG organization with a faster sign up into a free trial or something like that, I think it's great to uh, it's a great playing ground for message testing similar to direct to consumer. But, um, it could also work for B2B. You, you really have to work with the acquisition team to understand what is the metric that you are optimizing for. So in B2B, that's probably not going, you're probably not going to optimize for customer acquisition cost. Right? Because that's like, that doesn't make any sense. But maybe you can work with the team on something that's more engagement based, like a click through rate on Google search terms or something like that. So I'm not an expert by any means in user acquisition and all of that, but I know how to work with an acquisition team and an analytics team in order to set up a message test that's going to give me the insights that I need in order to feed my ongoing messaging, um, work across the other marketing channels. Does that make sense?

Adam: It does. I'm all for harmony and teams supporting each other. Yeah, I've always been on the more of the Enterprise B2B side. So it's like you talk start on PLG. I'm like, that scares me. I'm good. I like my long cycles.

Shannon: Yeah, it's interesting. You'll have to try it out and let me know like when I go through the framework, you can give it a go. Or maybe your team is like, no, you're crazy. Don't do this.

Adam: You're wild, Adam. Uh, wouldn't be the first time I've heard that. Okay, the framework, um, let's get into it because this is the good stuff right here. Um, and what Shannon's doing is pulling this up on screen. Now if you're watching this on YouTube, you'll get to see us and our faces and this example in full. Glorious video. If you're on Spotify, maybe we're feeling frisky and putting this video on Spotify, but you're probably listening to audio. If that's the case, the link to this template is going to be in the notes for this episode. So Shannon, when you're ready, let's get going.

Shannon: Let me share this. So just a quick note, and this is kind of what we were talking about is. Can you see my screen? I tried to zoom in. So, uh, what we were talking about earlier is that message testing is also about feedback loops. It's about understanding what is working. But even more so, it's about understanding what isn't working. So in my, in my framework, um, I have like past performance data in here. So you see performance context. What data do we have that might inform how tests should run. This is a cycle of learning. So maybe you have old data that you want to put in here that informs things. Maybe this is the data from the last message test that you're putting in here that's informing. Work with your acquisition team and see what they want. This framework, uh, is a little bit more detailed than maybe what I would deliver to a client, depending on the client. But um, always start with existing data and how that can inform message testing. And then of course you want to talk about your, and we'll talk about how to actually get hypotheses and things like that. Um, but the next thing is your test design, um, just to kind of help people understand what this means. And again we're talking about message testing on the acquisition side versus testing in app. Um, this is more advertising versus testing on a website. So, um, for the test design, typically how I've seen this be successful is you want to run a couple rounds of concept tests with whatever allocation of budget. Um, and then you essentially do three to four ads per concept. So your hypotheses informs the concept. And then out of that concept comes three to four ad variants. You really want to have like multiple variants. You can't just have one ad for one concept because that's not going to give you a valid output obviously. So your team can kind of help you set up what exactly the test looks like in the duration. Again, if we're thinking about PLG or direct to consumer with a faster sales cycle, I might want like, you know, I've been at organizations where they're doing $500 per UM concept across a five day sprint and that gets me enough data in order to make a decision. So work with your team and then also work with your team to understand the primary test metric. So how are you actually going to define success of this again? Short sales cycle. Maybe you can do return on ad spend. Maybe you need to move it up and just look at cost per click, which is more of an engagement metric. These are directional results. A lot of times they aren't stat sig. But again the goal here is also to inform product marketing and get a read on the market. It's not necessarily to do user acquisitions job for them. This is so that pmm, um, can get a better understanding of what is resonating with what markets and so that we can bring those insights back into broader marketing efforts. Right. We're not trying to optimize user acquisition right now and do their job for them. Um, does that make sense?

Adam: Yeah, absolutely. I was going to say, uh, this is all great. And then the next part is the testing framework, right?

Shannon: Yeah. So this is the testing framework.

Adam: No, you go for it. You're the expert.

Shannon: Okay, awesome. And this is like where it really gets important is how you set up your hypotheses. And from what I've seen is a lot of times people have a hard time actually setting up hypotheses and then maybe they don't really understand what they are. So what you'll see here on the page, I just included some really simple guidance on what hypotheses are. So it's all about, um, these seeds that you find in input data. And you find recurring themes in that input data that you're evaluating and you're turning those into predictions that you can measure. And that input data is coming from things like previous message tests, if you've run them, competitor ads, um, Reddit reviews, or if you're in B2B G2 reviews. Real customer conversations, of course, is a big one. Customer support tickets. So you're basically ingesting as much information as possible that you have around that product. And again, if this is a new product, you're not going to have a ton from your customer base, so you're going to have to get creative. So ingesting all of that and finding those recurring themes across your data and those are then turned into hypotheses that you can measure via, ah, tests. And so some of those are if you find themes around, like, who does this person aspire to be? Like, what do they want to be seen as a part of? Um, do they react more? This is one of my favorite ones. Do they react more to losing something or do they react more to achieving something that's really big? And B2B, um, what is the role of trust? This is a huge one and direct to consumer and a lot of PLG too. How can we tap into trust and test messages around trust and so forth. And so then once you have your hypotheses, you just put together your actual framework that you're going to pass off to either a creative team or a acquisition team. This is a lot more detailed than what I might give to a more advanced company. Um, but what you see here is obviously you have the concept name, um, which I usually just name whatever the driver is. So you have the concept, then you have what type of lever you're pulling here. Are you tapping into emotional, functional or social? What is the driver? So what is the behavioral trigger that's going to actually make that person behave Differently, you can think of that as like, oh, loss framing. Like that would be a driver. Right? Um, then you have your market insight or your customer insight if you're lucky enough to have had customer data here. This is a really nice tidbit to give a creative team just so that they know where this hypothesis is coming from. And also if you have a hypothesis, you should have an insight that's, that's backing that hypothesis. Um, and then you write out your hypothesis. Have an example here. Like by, by framing the product this way, we expect X in the consumer to drive action. That like a very simple framework that people can use. And then this is where it kind of. You may or may not need this. You can give an example static ad message. If you are, if you have a really advanced um, acquisition team, they're not going to need this. But um, you can give some creative guidance. I always like to be very light on this and not too prescriptive. And then you can also do like any notes or guardrails. Um, again that depending on how advanced your team is with uh, their acquisition and how comfortable they are with this type of message testing, especially for new products, sometimes teams appreciate being a little bit more prescriptive and then if it's relevant you can pop in some inspiration. This is really only if you're doing uh, maybe some static ads on meta as your message testing. Like if you're doing Google search, probably not going to be a whole lot of inspiration but I'll pause there and see like what clarifying questions. What did I breeze over that I should have double clicked into?

Adam: No, this is great and I think it covers really the basics of jumping into the world of AB testing. Uh, because there's so many directions, so many variants that you could test this hones in on it. A couple questions. Uh, you mentioned could be running for a five day sprint, could be longer. Uh, do you usually have a base frame for statistical relevance when you're approaching like is it timeframe, is it amount of impressions or is that really dictated by really the project that you're working on?

Shannon: Yeah, it highly depends on the project and I would take that guidance from the acquisition team themselves and, and again we're not looking with this to optimize. Um, I'm not looking to get the absolute best like customer acquisition cost. I'm looking for directional insights that help me as a product marketer understand what's resonating. And so for me, what, uh, because that then feeds the rest of our marketing efforts because I don't, I Don't need to help user acquisition do their job. They'll continue optimizing. Really the algorithms will continue optimizing for them. But I'm usually comfortable with something that isn't necessarily stat sig and I'm typically comfortable with leading indicators like the cost per, like a cost per cl. Excuse me, a click through rate. Um, some companies might be like no, we're optimizing for acquisition cost on all tests. We're not doing anything where that isn't part of the plan. But um, for me I like, I'm comfortable with just directional data.

Adam: And to your point earlier that what's being released in a launch is honestly probably going to dictate what you're tracking too. So if this is a, you know, a huge release from established brand, I would expect something a little bit different as opposed to a new startup who's launching ads for more or less brand awareness and learning, uh, as opposed to converting customers. So. That's an excellent point, Sharon.

Shannon: Yeah, exactly. You've got it.

Adam: Sweet. I think, um, all this is fantastic. Again, this will be linked in the notes for anyone who wants to grab this and make a copy and use it on their own and mess around. But there's one thing I gotta ask. This is pretty developed, right? So my assumption here ah, is Shannon, that you've got some stories or some numbers from real life usage that maybe you're anonymized for the sake of logo usage but or not call them out like what has been some of the cool things you've seen come out of it or maybe even the failures that you've learned from.

Shannon: What's interesting about the message testing is that it is a lot of failures. Um, uh, it's a lot of leaning really hard into differentiated messages and quickly finding out that they don't resonate. And to me that's a really big win. But um, some of the ranges that I've seen. So I did this for a client recently. I actually just set one of these up like two days ago for a PLG client. But um, I have a direct to consumer client that has a luxury physical product and we were optimizing for actual customer acquisition costs. And this is crazy because their CAC on average is like 130, $130 for a thousands of dollar luxury item. It's amazing. But um, the ranges that I saw for customer acquisition cost for the message testing that we did and this was only across the four concepts, we've got several left and I think we only did two ads per Concept because they were constrained on budget. We saw a customer acquisition cost ranging from $80 all the way up to like 200 plus dollars across the concepts. So they can be so wildly different, but with an average of 120, that $80 cac makes me very happy. But also that 200 plus $1 is like, whoa, okay, maybe we actually don't understand our market as well as we thought. Let's figure out what's going. Because it was around, um, a luxury angle, we were leaning really heavily into luxury language. So now I'm gonna go, like, dig deeper there and say, okay, what's actually going on? Is our audience not really interested in that sort of luxury positioning? Because that's a really important learning to bring back to the organization.

Adam: I agree. Yeah, it's be. It's so cool. I always love this. I have no patience for it, but I love it. Um, so thankfully there's experts like you, uh, in the world, Shannon. This has been a treasure trove of goodness, especially when it comes to the testing or ab testing part rolls into product launches. Um, if, if anyone's listening to this and say, hey, I want to learn more, I want to follow Shannon. I want to understand how she does this better. Where is the best place to, for people to find you?

Shannon: Yeah, I think just on LinkedIn. I'm pretty much there 24 7, so. And grab the framework, make it your own. Um, I have a cloud skill as well that I can. That's very super easy, where you can basically dump your insights into it again, the customer conversations, the web scrapes, whatever. You can just dump your insights and it'll get you like a starter framework, if anyone's interested in that. But find me on LinkedIn. I'm always there.

Adam: We're all chronically online, as it feels like. And also, uh, you're a fellow camper, a PMM camp, so yay for that. Uh, I'll leave. I'll swipe those, that, uh, cloud scale file and I'll link in the notes too, if you're up for that. I would love to share that out for anyone who wants to mess around with it.

Speaker B: Sure.

Shannon: Uh, sure.

Adam: Shannon, we crushed it. You crushed it. Um, thank you for being so gracious to spend a few minutes with me today. Drop by, share your insights and especially your templates and your know how, uh, with whoever's listening to this. I really appreciate it.

Shannon: Awesome. Um, thank you, Adam. I appreciate it too. I hope it was helpful.

Adam: Yeah, it absolutely was. Uh, and for those of you who are still listening, this has been another episode of prop marketing for you. I can't officially say this, but give us a quick follow on Spotify and give us a 5 star review if you feel generous. Takes about 2 minutes to do. Other than that, Shannon, thank you. And for those of you listening, thanks so much and catch you on the next episode.

Shannon: Sam.