#192 - Slowing Things Down to Speed Up Research with Jared Forney of Okta

Speaker A: So my strategy with working with legal is like, I try to come as early in the process as I can, even if I'm just coming with scraps. It's like, here is something I can see coming and here are the things that we anticipate might become problems in three to six months time. Can you help me piece this together? And that builds so much goodwill because it's like, hey, it's fine that you're bringing pieces. That's more than most people come to me with. But you're coming so early that it gives us time to work collaboratively together. Whereas another partner, like more like maybe a product management partner or a technical partner, might come more looking for final finished goods. Right. You know, they're like, I want to be able to take something and run with it. And so like how those stakeholders come to me at what point in the process, what their kind of idea of a deliverable looks like, that kind of shapes the relationships I have with each of my stakeholders that I work with and how I interface with them. But it all comes down to building that trust over time because that's when, when you need to communicate change. Because change management is a big part of our job too. When you have to communicate change, particularly big change, that's where that, uh, trust becomes the capital that you use to help drive that change.

Speaker B: Hey, this is Erin May, and this is Carol Guest, and this is Awkward Silences. Awkward Silences is brought to you by User Interviews, the fastest way to recruit targeted, high quality participants for any kind of research. Hello everybody, and welcome back to Awkward Silences. Today we're here with Jared Forney. Jared is the principal research operations person at okta, has written and talked extensively about research operations and on the cutting edge of what is next, which is what we're going to talk about. We're going to talk about the changing role of research in research ops, which are always changing. But that change is happening real fast, uh, right now and in real time. So we're going to talk about the latest and greatest. Jared, thank you so much for being here with us today.

Speaker A: Yeah, Erin, thanks so much. It's a pleasure to be here.

Speaker B: Awesome. All right, so let's dig right in and kind of, uh, orient the conversation around. There's a lot changing, there's a lot to think about. Of course we're going to talk about AI, but what are some of the most important areas for research and research ops to think about when it comes to upskilling and evolving right now?

Speaker A: Yeah, there's a lot there's a lot there. So it's a great question to start on. I think, you know, the things that I know I'm thinking about a lot and my team's thinking a lot about are two kind of key functions. You know, you mentioned scale. So scale is really like one of the first ones is how do we grow not only research's impact, but the scope of our work and the scope of the people that we impact. And I think the other one that we really focus on is rigor. You know, especially those two things sometimes feel diametrically opposed because Ed, uh, there's an increasing, there's always an increasing demand for research and research insight, which is a good problem to have. But at the same time there, especially with the, all of the new tooling and of course AI enhancements that are coming with access to research tooling, there's an increasing demand for people outside of traditional research roles who have a desire to do research and research like activities. Kate Cowsey famously, uh, referred to these people as powders, people who do research PWDRs. The thing that's really most prevalent with that is the upskilling is coming from more than just the research team, right? We're having to educate and help guide a new legion of these folks who are doing research activities, right. Who may have uh, had limited exposure to research methodologies before, who are not used to conducting research of their own at scale in a certain degree. So that increasingly is the challenge of not only researchers performing the day to day work of research itself. You know, whether it's generational, exploratory, the messy stickiness of people in general, but also like helping to enable, and I will use the D word early here, democratize, um, research outside of our traditional roles. Because as we've seen with the AI revolution in general, everyone's roles are blurring together, right? Product, product roles are blurring with design, design is blurring with engineering, research is blurring with both those fields. And so like increasingly we're all being asked to upskill in different ways.

Speaker B: Okay, yeah, great. So that's a great place to start. You, you mentioned scale and rigor and I, I can imagine a sort of two by two. You can imagine there's like an inverse relationship between these. How do you think about them? Like, is it possible to have our cake and eat it too? Can we get more scale and more speed and more rigor? Are they always, you know, opposing each other? How do you think about the trade offs there?

Speaker A: Yeah, I mean it is a balance, right? I mean, uh, there's there's. I think the, My answer to that would be like, it depends on how big your cake is and how many layers. Um, I, I think it really, the, the balance comes from understanding and having a really fundamental grasp on what your goals and objectives are. I think that's what something that, like, as ironic as it is, sometimes I feel like that gets lost amongst higher levels of stakeholders of like, there's this frenetic energy that comes with all of these AI adoption models where it's like, well, we're already behind if we're not adapting all these tools and doing this and doing that. But sometimes we lose track of the why and to what end. Right. And so really what I've thought a lot about at OKTA and what, like, my team has thought a lot about is like, getting back to like, what are our underlying goals and purposes for wanting to adopt and use these tools to begin with. Because we're giving, we're being given an unprecedented amount of access and tooling that is like something that's like a sea change from even where we were two years ago. The level of access has gone up tremendously. But I think what has been a little bit more of a fast follow and what has been something that's been more of a work in progress organizationally is thinking about, okay, like, what is this going to enable us to do? What objectives is this going to accomplish? So before we even think about how do we make sure we get the quality while scaling, at the same time, we have to, like, fundamentally go back to what, um, our goals are. So how is this going to help us reach out and effectively serve our customers better? In what ways? You know, is it about being able to reach out to more participants more effectively? Is it about being able to get more insight per participant? Because we all know recruiting is very difficult and still in this day and age, is a frequent bottleneck. Is it about being able to streamline the delivery process? Right. Being able to put insights in context with where people are, where they're working. Those sorts of things, like thinking about those underlying goals and working backwards into the tooling stack is really how I think about it.

Speaker B: Okay. Yeah. Great. And is that, uh, sort of meta question of why are we even doing this in the first place? Is that something you've been able to, I guess, democratize? And does that question sit with research, with research ops, or is that sort of, you know, a behavior, uh, that can, can be taught?

Speaker A: Yeah, I mean, it's, it's something. It's. I think it Works best. Right. When it's a collective, uh, understanding. Right. Like when everyone's on board with that. I think most people would agree. When everyone's on board with why are we doing this, everything tends to go better. But I, I do think it's what if I'm speaking particularly to research and research operations, it's a unique time in our field where we have a lot. We have an outsized impact on that decision. Right. Because we're so close to the tooling stacks that people are so focused. But. And like internally at uh, Okta, just to use this as an example, like, we have an entirely new product development life cycle that's really built around fundamentally the AI enhancements. It undergirds all of it. But more than that, it really, organizationally we stopped and realized that we need more gates in some cases because this is moving so quickly inasmuch as we're automating and allowing for more things to be systematized, more agentic interactions, whether it's like, you know, building product requirement docs or reviews or that sort of thing. At the same time we realize we need in some cases more human checks than we did before. And so really that's where the opportunity lies collectively across the product organization is. It's a good time to almost stop and pause for a minute and really think about what your process looks like today. Before you just kind of pour. Guess. Pour gas on the fire.

Speaker B: Yeah. Slow down to speed up. And that can be a difficult. You gotta be careful how you message that to who. Right?

Speaker A: Yes.

Speaker B: Yeah. And I know that's something you've talked about a lot as well, is rethinking kind of who are the stakeholders and departments that you need to be working with in these changing environments, in this democratized environment. Can you tell us a little bit more about that?

Speaker A: Yeah, I think one of the more fundamental things about research ops that I tell people is it's. It's a currency of trust. Right. Like that is really our part and parcel. Business is about building trust and confidence in the systems that underlie not just research, but product development as a whole. Right. And so that currency is something that's a little bit hard. It's not as fungible as maybe something like an insight or something that's more concrete and quantifiable. But as long as organizations are composed of people in relationships with those people, that is going to be our business. Right. And really a lot of what I think about is like how as a research operations person, can I help drive those conversations in directions that will help build trust internally. So, you know, when I'm thinking about developing a tool or building confidence between stakeholders, a lot of times when I'm building those relationships, whether it's with legal, whether it's with our product managers or designers, it's always in the back of my mind I'm like, I'm trying to increase the trust that those stakeholders and those partners have with not just research ops, but the trust that they have in the process. So with Legal, Legal gets bombarded from every direction. Right. Uh, I have. My heart goes out to a lot of people in Legal because a lot of times when most people, I would imagine, are interfacing with legal, it's because something's already on fire or about to be. Something's about to combustion. And so a lot of times it's, you know, everything is reactionary for them. And, um, so my strategy with like working with legal is like, I try to come as early in the process as I can. Even if I'm just coming with scraps. It's like, here is something I can see coming, and here are the things that we anticipate might become problems in three to six months time. Can you help me piece this together? And that builds so much goodwill because it's like, hey, it's fine that you're bringing pieces. That's more than most people come to me with. But you're coming so early that it gives us time to work collaboratively together. Right. Whereas another partner, like, more like maybe a product management partner or technical, uh, partner might come more looking for final finished goods. Right. You know, they're like, I want to be able to take something and run with it. And so, like, how those stakeholders come to me at what point in the process, what their kind of idea of a deliverable looks like, that kind of shapes the relationships I have with each of my stakeholders that I work with and how I interface with them. But it all comes down to building that trust over time because that's when, when you need to communicate change. Because change management is a big part of our job too. When you have to communicate change, particularly big change, that's where you, that trust becomes the capital that you use to help drive that change.

Speaker B: Yeah, and trust, of course, has always been so important with stakeholder, uh, management. You know, one of my big learnings doing this podcast for all these years has been the research you do internally is as important, if not more important than the research you do externally. Right. So you're always getting to know these folks and what makes them Tick, what are their motivations? And this might change over time. I'm curious, you know, have any of these observations changed for you with AI, with this speeding up of product development with democratization, you know, is it, you can imagine there's a temptation, for example, to forego getting ahead of the legal earlier because we're in such a hurry, but is it perhaps even more important to take the time to do that?

Speaker A: Yeah, yeah. I mean it's a big change. Right? I mean that's what's like perhaps been kind of most arresting, uh, about like how fast this is moving. It's like even parties and individuals who would normally have been more cautious or uh, more conservative in their deployment, everything is sped up drastically. It's where it's kind of surprising and it takes a little bit of recalibration, like, oh, you're on a completely different gear than you were before because we're all feeling that pressure, right?

Speaker B: Yeah.

Speaker A: And I think part of what uh, comes with that in that in recalibrating and recontextualizing that relationship is one like again to build that trust where it's like, hey, I know you're under a lot of time pressure, so am I. We both are in the same boat together. And knowing that it's like when you need to move fast on something. Again, let's get to the underlying reason behind that. And do we really need to move that fast? Because that's another aspect of this where making especially higher level stakeholders aware of like here are the things that feel urgent, but it's not as urgent as we think it is. And there are of course, like, you know, like nobody likes to hear about like, well, the what ifs. Right. The consequences of this. Right. They want to focus on the outcome, the potential outcome rather than all, uh, necessarily all the risks. But I think at the same time, if you help people realize that there are really like to, you know, like as you said, like going slow to go fast. Right. And it doesn't necessarily mean everything grinding to a halt either. It's just like very considered like in our product development life cycle, very concerted checks at certain points in the process where there's concentrated amounts of human oversight that allow for all the other AI enhancements to happen. Right. It's like we're not substituting one for another. It's an and. And really what that brings to the table is this opportunity for not only to slow things down through those gates, but, but it helps people feel validated. Right. Like that. Because that's obviously the Other existential concern is like, this AI is going to take my job. Right. But it's like, no, your role becomes more important than ever as we apply fewer and fewer. Like as we automate more and more, those concentrated checks where your expertise comes into account becomes exponentially more important. And so it's like to help people understand that and help our stakeholders understand that, we're concentrating the human attention and effort now much more than we were before. It was maybe more diffuse and spread throughout the process. Now it's like really focused.

Speaker B: Yeah. I feel like, was it last year everyone was sort of building these, right. AI agent org charts. It's like, well, where's the people? And now, you know, we've. Everyone's using AI. We're seeing a lot of AI slop. And now we're seeing human in the loop here and here. Right. And really mapping that out. And so you see these kind of corrections, adoption, correction and this kind of healthy back and forth, I think, happening as AI adoption scales.

Speaker A: We're seeing pullbacks in various ways. Right. We were just talking a little bit before this, the first major pullback that I'm sure a lot of organizations are experiencing. I'm like, holy cow, tokens are expensive. We're seeing now we all have token budgets and oh my gosh, this is more expensive than people. Uh, so there's a lot of, in the broader market, a lot of pullback first fiscally.

Speaker B: Right.

Speaker A: With that, I think will also come a realization that like, a collective realization by the market of like, there are things that AI is good at and things that it's really not good at. And that's where again, the specialization and focus of human attention and human action becomes all the more important.

Speaker B: And it feels like Research Ops has a very unique role to play in that. Right. Because you know, I'm imagining, and this is an oversimplification, but let's say five years ago, Research Ops were making a lot of templates and documents and we're revisiting them as processes and methods change. But they have a shelf life of a while. Whereas now, right. It's what we're building AI native systems or. But the technology's changing what we're learning about where to put that human in the loop. So how has that changed? I guess that process of setting these systems and setting these templates and figuring out how to democratize research.

Speaker A: Yeah. And you know what's interesting about that, Aaron, is like in as much as like. Yeah. I mean, you know, five years ago I'd even say as little as two years ago. It's, it really underscores like in the last 18 months things are just fundamentally different. But you know what's really interesting about that is some of the core motions aren't really all that different. I think that in the other thing I think about when I think about and frame agentic AI in particular is it's really just another stakeholder, it's another user. The only difference is they're just not human and can operate at scale orders of magnitude more in terms of actions than a single person can take. But the same guardrails, the same types of provisioning actions, the same types of policy and governance guidelines still apply. So really what a lot of what it's done is it's not so much of like, oh, we're not doing these types of activities anymore, we're focused all on these instead. Really it's just a reshaping of the actions we take. So maybe things that might have started as templates and recruitment guides now become prompts. Right. It's taking the same skill set and applying it in a different context. I know we've also been experimenting with synthetic users a lot, which I know that's also a term that is very, it's a very hot topic right now and uh, myself included in terms of the skepticism, but it's really pretty incredible to see how you can apply for certain types of like very early stage work. You know, even before you would typically bring something in like an evaluative study where you'd be bringing actual participants in the room doing it almost like as a pre test sort of thing. It's been a really interesting exercise in understanding not just what the capabilities are, but like how do our current templates, our Personas and user profiles and our past study work? How can that be leveraged and utilized in this new context? And like what are the pros and cons of that? And again, how is it serving that process in our product development life cycle to get us to that end goal of connecting better with our customers?

Speaker B: Yeah, and this is a great example with, uh, Nick just shared in the chat, we did a uh, recent report on synthetic users. And what's clear is that in many cases the hype is ahead of actual usage. The folks who are ahead in usage are at uh, some of the forefront. But then you also have maybe some of those powder who don't have a lot of guardrails who could be maybe not using it in the best way. And so research, research ops really do have this unique role with Synthetic users with AI technology in general to share best practices guardrails and keep those up to date by getting hands on with them. Right. This is not an ivory tower thing. This is uh, we're learning too, but we're going to share what we're learning.

Speaker A: Yeah, I mean I think that that's the other really unique opportunity we have. And you mentioned also like doing research internally. That's something that like I come from a research background so I kind of. A lot of people go the other way. They start as research operations and then go into research. I went the other direction. Yeah, I do miss talking to customers every day. Like that is one aspect of my job I do miss. But like what. Where I found some fulfillment in that is actually talking internally. And I think it's something that we all don't do often enough because we have stakeholders too. Right. We have our, our customers, our users are our folks internally. And there's just so much we can learn from how our colleagues work in what they care about. And like as you mentioned, just like having this co learning opportunity because uh, it's one of these very rare paradigm shifts in the way we work that like everyone has something to contribute. Right. And it's kind of a great reset and equalizer in that way that it allows us all to have kind of an egalitarian voice on certain parts of this experience and we all can bring different types of expertise to the table. And that's where I think the role of research operations in particular is so central to this because it's about learning and insight fundamentally and how again how we can improve our products and better fulfill our users needs. But it's also an opportunity to all learn a lot more about ourselves and how we work as an organization. Because of course the products that go out in the world are a direct reflection of the internal corporate culture that builds that product.

Speaker B: Yeah. I want to ask just a couple more questions and then we'll move to the Q and A portion. So you know you mentioned that you went from research to research ops, right. In a lot of fields you're seeing a lot of compression. That's got to be one of the words of the year, right? Of different roles. Right. Uh, a given function can handle more breadth because of AI, because of the need for speed, whatever it might be. How are you seeing that play out in research, research ops? Are you seeing these roles blend together? Are you seeing more specialization? Are you seeing them cover uh, into maybe some of the design and product management areas? What are you seeing there?

Speaker A: Yeah, I know just speaking personally, it's gotten a lot more technical in a hurry.

Speaker B: I don't know about you, but I have my first GitHub repo.

Speaker A: Oh yeah, like we're, yeah, I did that. I mean also like one of the things that I just. On that, on that note in particular, like I, I would say like I wouldn't consider myself git expert by any means but like one thing that's thing that I did early on, um, was I really early on started doing personal projects, right. To get comfortable with the repository and git workflow. I had dabbled as a hobbyist in the past, but part of what I find most engaging about what AI is capable of is like what I think many are calling like the personal software revolution, like building products for users who are one of one. I think that's also really a really great way to get used to the workflows and things. So that when I went into work I was already familiar with some of the nomenclature and that really helped. But all that being said, had to get very technical very quickly. Adopting a lot more of the basic engineering structure and language, not just to communicate with our engineering teams, but even just to like do the work of automation and engage with these tools. Because as much as they're accelerating into a consumer minded direction, these are still fundamentally very technical tools. Right? And we're all kind of, especially as we dip in to more and more of this and kind of seed more of the control to these systems, there's a lot that we're executing on that we don't fully understand. Right. As, as end users of it. So being able to uplevel your technical skill I think is something that's like increased, been increasingly important. You know, I spend more time configuring like APIs and things than I ever did previously. But at the same time I think the other thing that I've had to really sharpen my skills on more is written communication. It's something that like you're constantly working on. But I think a lot of people have gotten more sensitive, and rightly so, to AI generated written communication. Right? We've all learned to detect it. You know, we know what M M dash itis looks like I used to

Speaker B: use them and now I don't. It's been nice for me, right?

Speaker A: I mean, and there's like places where it's appropriate, right? Like, I mean there's places and uses where like, you know, most people are like, okay, it's like a routine systemic communication. It doesn't need to be necessarily hand tailored. But frequently we're like starting to already see evidence of this of like what is deemed a valuable interaction by people is one that is inherently human. Right. One that flaws and all is inherently like a person to person interaction. And so it only underscores why that skill set is so important to be able to clearly communicate an idea, to tell a story, to be able to resonate with people. Because again, as long as we are building, as long as people are building software for other people, people will be an essential part of that process. Right. And so those are the skills that I think about, even though they seem kind of diametrically opposed to one another. Like it is something that's both like highly technical and highly human at the same time. So those two things I feel like really go hand in hand in terms of what I've been doing more lately.

Speaker B: Yeah, it's really. The HCI has never been more hci. Right. I mean it's like on exponentially so. But um, and then the last thing I wanted to ask you was, you know, you've talked a lot about sort of reconsidering how insights are collected, stored, accessed and shared. Tell us about that.

Speaker A: Yeah, so that's how I got my foot in the door in research operations was through knowledge management. You know, I think for a lot of people that's kind of a common front door. Knowledge management and repositories are always an evergreen topic because we have to store that knowledge, that insight, that collective corporate brain somewhere. I think what I'm finding in the evolution of that, um, we've gone through um, two different repository systems in the time, we're dovetail users currently. But I think really what's changing about that is where we might have had single systems of record before a ah, repository in this monolithic sense of here's the place where insights live and everybody goes to it. With the advent of mcp, you know, being able to agentically connect to disparate sources of information to then be very quickly amalgamated into a whole, a repository almost becomes another hook, right? Another source of information maybe. You know, you've got your product telemetry over here, you've got your repository over here, you've got support tickets here and then you've got like maybe a um, a Slack channel that is coming from AE contacts with your customers over here. Right. And all of these things could be united by something like mcp. And then it becomes less about the individual sources themselves and more about orchestration. Orchestration has been the hot topic of the last couple months. And I think what's exciting about that is that's one, that's the stuff I'm really into. That's kind of like why I got into this role is for workflows. Because the thing I tell every vendor I work with, and I'm a broken record about this, so if anyone's heard me talk before, you heard this is crazy, I always tell them, don't sell me a platform, sell me glue. Because what more and more we're moving away from, in my view, is this platform organization, these monolithic stores of vast quantities of data that become a single access point. And instead it's about how nicely do you play with others. It's more than just about data portability, right? It's about how are you contributing to an, uh, organization's workflow as a whole, right? So how well do you integrate? Is it API, is it mcp, is it data piping in one way or another, webhooks, what have you. That type of orchestration becomes exponentially important now in the age of AI because it's only getting, it's only accelerating. And now when we look at vendors, a lot of it, the first question is like, well, do you do these things? Because if you don't, it becomes a harder proposition to be able to like, because the first thing that goes to your mind is manual work. So that's just like, I think the paradigm shift that we're seeing with that too.

Speaker B: Yeah, but, but Jared, how am I going to charge you 100k if I don't call myself a platform?

Speaker A: Well, I mean, a lot of times it's like, you know, and I mean, of course, like, budgets are always another aspect of this. And there are, there are lots of ways to provide value, right? I mean, and there are certain, there are certain aspects and centralization of things. You know, there, there, there is a place for hubs, right? But it's like hub and spoke, where a lot of, like, you know, it's not everything we expect to be, you know, you know, floating in the ether. Because there are problems with that too, right? If it's too disconnected and too floaty, right? It becomes hard to identify centralization points. Like, even in our workflow, we have two core pillars, right? We have like, our intake and management flow, and then we have our data, data store, repository and distribution, right? So like, you know, it's, it's not all just this massive web of interconnectedness, but the more integrations you have, of course, the more points of failure you have too. So A lot of what you have to balance is identifying how many sources do you want and again, what underlying goals are you supporting? Right. If your organization works better in a single hub, that may work great for you and your organization, depending on what their needs and goals are. So it's just something I'm observing more broadly, but to your point, like there is a time and place for centralization of certain aspects, certainly with participant management

Speaker B: as one Awkward interruption. This episode of awkward silences, like every episode of awkward silences, is brought to you by user interviews. We know that finding participants for research is hard. User interviews is the fastest way to recruit targeted, high quality participants for any kind of research. We're not a testing platform. Instead we're fully focused on making sure you can get just in time insights for your product development, business strategy, marketing and more. Go to userinterviews.com awkward to get your first three participants free. I think it'll be interesting too to see how the bundling and unbundling and rebundling will play out with all of this too. Right. Um, because we're in, I guess you would call it an unbundling of data sources. Sort of plugging in. Right. And being portable. But um.

Speaker A: The season of skus.

Speaker B: The season of skus. But uh, anyway, yeah, I'm curious how much you mentioned orchestration. The other word that comes to mind for me is sort of like algorithm building. And what I mean by that is you've got 20 sources of truth for information now, which gets primacy, who builds that and where it's accessed. Is that the kind of work you're doing and is that uh, very different than the work you used to do when you think about managing knowledge?

Speaker A: Yeah, I mean it's definitely a challenge because it's like how do you decide what is canonical? Right. Um, because all, you know, a key aspect of knowledge management is uh, what, what I've heard effectively called data gardening. Right. We have to make sure that things are pruned and neat and old studies are deprecated and like new, you know, new studies are balanced with work that has like long tail significance. Right. So a lot of that comes with like a reexamination of like uh, what is kind of the effective half life of certain methodologies. Right. We look at things that are canonical, generative studies, things that are like foundational research, things that inform like our Personas, for example, our buyer and customer Personas are more, maybe more foundational work tends to change a little bit more slowly versus something like a usability study. Which sometimes is more of a flash in the pan and like, you know, has immediate value, but then it dissipates quickly as things are iterated on. So, you know, there's a, there's a, uh, rebalancing that happens with that. Especially when some of these systems through MCP don't always have the ability to differentiate between that. They're not able to make that distinction necessarily. And so what becomes important is not only what we're putting into our repositories and these data stores, but also like you said, what primacy and value we're assigning to it is something that like, is actively being discussed right now. There's, there's ways, we're talking about doing that through taxonomy. There's ways that, that are a little bit more blunt forward. Uh, like blunt where it's like, does it even get included at all? Right. Is it something that lives outside of a repository simply because its half life is so short? So that's something that's very top of mind right now.

Speaker B: Yeah. Okay. Yeah. Great. All right, let's take some audience questions. We'll start with, uh, one here from your friend and mine, Caleb Loosebrip. What was the single best tactic and argument that successfully got your team to slow down and invest in governance?

Speaker A: Uh, yes, thank you Caleb for the question and thank you to everyone who submitted questions. I'm looking forward to answering more of these. Ah, uh, slowing down. The biggest thing I think is really that focus on rigor and quality. Right. That's something that's like a collective discussion, um, that's happening particularly as we scale. There's an understanding that, uh, more research faster doesn't mean it's good research. Right. It doesn't mean it's contributing in a way that necessarily is helping us move forward. And so a lot of what we're doing, doing to encourage that governance. And this is like where our research team kind of led by example, like I'm really proud of how our team did this. When, when this AI adoption really first started to accelerate internally within our company, we realized like, we need kind of a, a central statement, a print, a set of principles and a way to plan our flag in the ground, knowing we're planting it on shifting sand. But like, we got to start somewhere and especially before it's decided for us. And so a lot of it came down to like, here's how we define quality, here's how we disclose AI usage, here's when, um, we do adopt it, this is how we use it, and this is how we don't use it. And I think by setting those principles and socializing them, that made the case for. Ooh, I never thought about that. Like, from an organizational standpoint, it's like, that's a really good point about we shouldn't necessarily take things at face value. Or, you know, we should narrow our scope and maybe focus on small language models that are only citing our research rather than using broader LLMs that are calling outside of our organization and using outside sources that maybe are a little bit more of a black box. The way we kind of impose that or suggested that sense of governance wasn't even necessarily by saying, slow down. It's just like, here's how we're moving, right? We're like really intentionally showing like, yes, we're crawling, walking and running, but we're really over disclosing. We're being really transparent in how we're taking each of those steps as we accelerate.

Speaker B: Right. And like you were sort of mentioning before, in aggregate, everything is without a doubt moving faster. But not everything needs to move at a breakneck pace for that to be true and just being really intentional, uh, about that. And I also love that you took the time to define what quality means, because I think that's one of those things that can be so hand. Wavy. Right. It's like we have to be rigorous, we have to have quality, whatever it might be. But if you actually take the time to socialize what we mean by that, people will be like, well, yeah, of course we want to do that, you know.

Speaker A: Yeah. I mean, it's like, yeah, shared definitions are so important because there's so much flying around that like just even taking the time to come up with a universally set agreed upon set of definitions. Just that, uh, exercise alone, I feel like is so healthy for organizations to have. I can't even imagine the ability to just. I can't say we've had a unifying workshop in that way, but looking back, that's something I maybe wish I would have done. It's like get all these people in a room for an hour or two and just, uh, come out with a degree set upon definitions. We did it more incrementally, but we got there all the same. And I think that's what's really helped everybody start to. Even as things are flying around, there's chaos sometimes. We can always. We have the set of uniform definitions and checks we can return to.

Speaker B: Yeah, that's great. Okay, we've got a zoomed out question here from Natasha Duncan, uh, at Boots, user researcher uh, the question is, any tips for getting buy in for research operations in general and getting the time to do the tasks?

Speaker A: Yeah. So I, I'm going to make maybe an assumption here. So Natasha, I'm sorry if I'm making an assumption that maybe you're on a smaller team or possibly a team of one, but if you're not, that's great, but hopefully this will still help you. I think that the argument that because I was in this boat too, it's like, how do you make the case for doing research, operations activities when there's so many other pressures, so much other research to be done? The argument that I used and when I realized I couldn't do both jobs well at the same time is making your organization, uh, aware of the fact that this work is going to get done either way. Like the operations work has to be done and the question is, who's doing it and what is it preventing them from doing instead? So when there's not a dedicated operations person in place, the first sign that you need one is that your researchers are doing it right. So they're doing those operational tasks, whether it's participant coordination, incentive distribution, knowledge management, procurement, all those tasks. And they're not doing research right, they're not talking to customers, they're not distributing and sharing insight, they're not helping inform product decisions in that way that operations work is going to happen whether or not you have a dedicated person. And the case I made to my organization is like, what do you want our researchers doing more? And it was a pretty easy answer for them. We want them doing more research. Of course. I'm like, well then this is the case of where if you have someone who's dedicated at those tasks, a couple things happen. One, obviously they do, they have more time to focus on the customer experience. And then the second thing that happens is it unlocks all of these potential process improvements that acts as a flywheel on the rest of the research process. Right. And then you can start talking at things about like now we're very fortunate in the fact that we made what started as a, uh, research ops team, uh, of one, just me. We now are like getting ready to hire our fourth person. And what, that's what's really exciting about that is one, it shows confidence in the organization that this is a role that's valuable. But the other thing is it's allowing our ops people to start to specialize. So we have someone who's just focused on democratization and socialization, or, excuse me, democratization in general. And Research enablement. We have someone who's just focused on our beta programs, which is like turning into this huge aspect of our um, of our research experience and product development lifecycle. And we have another person who's like focusing more just on the participant side. And so I think really what that specialization starts to allow you to do is like, it builds the case for being able to further scale and you talk about force multiplier for being able to expand your research operation and really just the ability to learn as an organization.

Speaker B: Yeah, yeah. So researchers are going to get to do more of what they're supposed to do. Research. The OP stuff they have to do, they no longer have to do. But you're also going to make that stuff faster and more efficient, which researchers of course don't have time to do to think about those systems. Right. It's blocking and tackling and then you can specialize, enforce, multiply all that stuff. I do think it's the golden era for operations of all kinds, particularly because of AI and the need to democratize and the technology and all the systems. I'm curious, uh, you know, in marketing land, there's. We've got GTM engineers, we have content engineers now. Everyone's an engineer. Has this spread to research yet or are we still.

Speaker A: I think we're, I think we're getting there, right? I mean, and it's like, this is what I always like, say that like so, so much of research operations borrows fundamentals from design operations, which borrowed fundamentals from DevOps, right? So like, we're all inheriting, um, language and uh, rituals and things from organizations or from functions that came before us and still exist today, obviously. But what's different is how we're leveraging those traditions, right? And how we're putting our unique flavor on some of those rituals because it is a different function. I've seen some talk from peers in the industry of like, yeah, I'm seeing a blurring between, uh, design ops and research ops. And how do I make the case that this is a separate function? And it really comes down to, again, like, your organization size can have an impact here. Like, I could see a world in which some, sometimes someone might be like kind of forced to straddle a little bit, but it really comes down to like, who is your primary audience that you're enabling first and foremost. Right? So for like a design operations person, it's designers, right? It's in the name DevOps, it's developers. Right. You know, with research operations, first and foremost, my core stakeholders on My research team, those are the people who I am first and foremost responsible to. And then we start talking about spheres of influence outside of that. And really like being able to make the case for that separate role really comes with that maturity curve of your organization. Like where do you. That's a great uh, place to. A great question to ask yourself if you're trying to make the case for Research Ops 2 is where do you think your organization is on that research maturity curve? Right. And just because you're not far along, if you find you're like in the earlier stages of that, that doesn't mean that there's not still an opportunity for research operations. It's just that your framing is different. It's more about, maybe it's more tactical in nature. Right. Than it is strategic. The goal I think for most people and where I know I want to be is more of a strategic function. Right. Because it's about thinking for the long term. But like, there's still a lot of pragmatic stuff I do every day. I, I joke this role is like at ah, equal Ah, times 30,000ft and three inches. Uh, so there could be a lot of thrash that comes with that. But it's a really important part of the function is being able to think at that high strategic level and also work on the nitty gritty in the same day. Awesome.

Speaker B: Um, all right, our next question comes from Anon, who is a uxr. How can reops teams enable more, faster research when timelines are more compressed than ever?

Speaker A: Yeah, this is the seminal question of our time. Right. Uh, you know, I think it really comes down to like redefining what fast means and like redefining what delivery means in that context. Because, and this comes again, I know I keep cycling back to like, what are your goals? Right. What is your organization's goals with that research? What's driving that need for quick insight? Is it a go to market launch? Is it some uh, external pressure? Is it a competitor? Is it something about an opportunity that we're seeing in the market? You know, there's all sorts of underlying causes that drive that frenetic need for speed. And by starting to understand that, that's where I start to work backwards to like methodologies and then to, and then to tooling and processes and things like that from there. Because there's a lot of value that you can get obviously from like, you know, there's a million usability tools out there that'll promise results in an hour. Right. Um, and you can get something that resembles insight research in that short period of time, but it often turns to be kind of fluff, right? It's something that may not and fluff at best, and at worst it's dangerous in the sense that it is something that masquerades as insight that could be then taken and run with and applied to the company's detriment. That's the real. That's one of the dangers of AI, right is it's so convincing. Whether it's through hallucinations and through the faux detail it provides, it could be really dangerous for things in the name of speed to take and run with. So a lot of what I try to focus on with teams like that when they're asking, when they're facing pressures for speed, is like I have a philosophy, I think about of uh, one, uh, hundred fifty, fifteen and really it's like, what does 100% of this process implemented look like? Okay, now cut that in half. Okay, now cut that to 15% of what it was. You know, it's like the classic MVP model, but applied to your own research process and methodologies. It's like, what is that 15% version of a methodology? What does a 15% generative study look like? You know what I mean? If you're trimmed down, if you have to boil it to down the essentials, what does it look like? Uh, are there opportunities where, okay, maybe you have to make trade offs with participants, maybe you're doing a little bit more internal research than external. If you're talking about participants, maybe it's, you're doing slightly smaller sample sizes. Maybe you're using AI tooling to help speed up the synthesis where it would have been a little, a little more hands on and more taxonomically focused before. And there are trade offs with all of that. But the idea being so much of what our operations function is in, uh, the service of speed and efficiency. It's about building trust and how can I use my methodologies and process and tooling to not only help enable our teams to move faster, but to deliver trust even in smaller pieces that'll give my stakeholders the confidence to give us more time and space to move from that 15 to that 50 to that 100% implementation of that process.

Speaker B: All right, I think we have time for two more questions if that's all right with you. Okay, here we go. This one is a, uh, few sentences here, so bear with me. This is from Kittist and Kittist is. I don't know if I have your job here. You are a former uxr okay. We often hear how AI agents can help streamline our workflows and save us time on the most minute of uh, research tasks. But I want to know what have been some of the challenges and limitations of using AI in your work? What are the things we should know and set up guardrails for?

Speaker A: Yeah, it's a great question. I think the limitations, right? I think that like the, the first and foremost when we hear the most is hallucination, right? It's the, the overconfidence that comes with being able to take something out of context. Uh, for an AI system to take something out of context over prescribe weight, being able to generate really convincing looking but factually inaccurate insight. Right? And so it's like, that is like one of the risks that we have that we deal with is maintaining that quality. I think another aspect that we have to be careful of in that same vein, especially as we talk about agentic AI, is so much of it gets up, leveled early in the problem space defining process. So like, think about like a product requirements doc, right? Like fundamentally that's a template, right? Like we do product requirements docs over like PM and other parts of product. They write those docs over and over again and they follow the, generally the same format. It's a template. And so templates are a great avenue for agentic generation because we can start and spin up something as a, as a framework to work off of from there. Um, but as. And we've even started to see some evidence of this too, it's like, oh, we can do these PR docs generations and then we could start pulling in sources from what we know already and injecting them into this document to then push out something that's like, hey, this is ready for review, let's do it. And part of the risk that we see with that is, okay, it's great that you're like starting with the template and you're plugging in from all these sources to help inform this document. But if nobody's checking where those agents, those agent hooks are coming from, what prompts they're using, and it's just being dropped in and then taken verbatim as truth, that's where the risk is. And so a big part of like the automation and templatization, a big part of what we do and what our responsibility is, is to make sure there's good fundamentals in those templates before, right? Whether it's the prompts themselves, the MCP hooks that we have into the tooling, how those um, hooks and Prompts are designed so when it's ingesting the tools, it's doing it within a certain set of rules and frameworks. And then also, I think part of our job is education, too. I often, uh, think about the importance of building good fundamentals. I think that so many people are in this rush to learn new things quickly. There's a lot of fear that people are going to be left behind. And so we just adopt things and run with them because it's like, we don't want to be the person who wants to take the time to learn something more, uh, thoroughly, because we're just in such a hurry to demonstrate results. And so I think really where we have an opportunity operationally, as someone who's like, watching all the pieces fit together, including when they don't fit together well, we have the opportunity to raise our hands and say, like, we all need to take time to explore and learn together and make mistakes together. Right? Like, that's something that I think we really need to be able to have the space to do more of as we experiment is we need. We need to learn the, uh. We have to have the space and opportunity to make mistakes in a controlled way. And so as researchers and things, that's where the guardrails. That's what I'm really trying to consciously put more guardrails around is like, creating that space for mistakes to happen in a safe and controlled way. Because when we were able to learn from those and demonstrate, like, hey, look at this. Look how crazy this. Look this, uh, crazy thing that happened in this. In this walled garden, like, what if that got out there? Right? That'd be bad, right? Like, it gives everybody a chance to slow down and give themselves the permission to be a little bit more thoughtful, to take the time to, like, learn and apply better prompts, better systems, and, um, ultimately better processes.

Speaker B: That's great. And I think that's so needed. And, um, there are so many mandates to learn AI and use AI and just having those support systems, whether it be dedicated AI help or operations help, but, uh, providing that safety, that time, those tools, so folks can learn. Because learning isn't supposed to be fast, right? It's messy and hard and full of failure and all those things and.

Speaker A: Absolutely.

Speaker B: Uh, so that's great to hear. All right, last question. Any tips on getting into the UXR REOP space as a newcomer with minimal experience? This is from Rituja.

Speaker A: Yeah, so I think this is just something that, like, if you're coming in as, like, a new grad, you Know, I think it's one, like, I. I have. My heart goes out to you. It's a. It's a tough world out there. It's a really tough time to be in the market right now. And, you know, even just as someone who is also part of the hiring tracks, it's unprecedented, right? I would. I. I feel like I would struggle in this new world that we're in. So by. By, like, heart goes out to anyone who's in this role right now. And I. I talk to and mentor a lot of people in this space too. And I think the things that I would say, if you're not coming from a research background, I think a lot of people are kind of fret that it's like, oh, I just, like, don't have this background and I'm worried that, like, it won't be applicable. And like, the thing I tell people is that your life experience, every experience you have, can apply to this field. Um, uh, one example I'll talk about briefly is I talked with a guy who was looking to get into the research space, and he didn't have any design experience or background in design, and he was trying to move and take that next step. And I'm like, well, what did you do before this? And he's like, well, I was a metal worker, and I did this, that, and the other. And I'm like, what do you mean you don't have design experience? Like, you've worked with your hands in a physical medium with all sorts of constraints and techniques, and all that stuff is absolutely applicable. You just have to recontextualize it in this new world that you're in. And I think that's a really important message that I have for a lot of people who are looking to go into research and even research operations, is that you have more experience than you think you do. The important part of what and the hard work comes from translating that and convincing others that that experience really counts for something and that you bring a unique perspective to the table. Because, like, I didn't start in research operations. I actually started as a designer researcher hybrid. I've gone through three career changes in my time here at Okta. I've been here nine years, and I've changed roles three times. And so, like, each of those jumps, I had to convince my manager or my manager's manager that, like, this was something that was not only important for my career development, but was also going to be a force multiplier for the organization. Right? It's like, by helping me make this transition in this role here is what I'm going to be able to do. And that case is something that takes time to make, and you have to have the right stakeholders and things to help you through that process. But ultimately, when it comes to moving into research operations, a big part of what I say is, like, if not me, who. Right? And what are we giving up by, uh, having somebody else do this that's not centralized, that doesn't have the context, that's distributed throughout the organization, that's scattered and siloed? Right. That's how I often tell people to frame that conversation if they're starting to move into research ops or make a case for that first hire.

Speaker B: Yeah. And I'll tell you what, too. And to your point, it is a difficult environment, no doubt about that. But I think experience arguably counted more than it does right now, where everyone's having to reinvent themselves. And it's really that growth mindset, that willingness to learn, that passion for the craft that's always counted. But I think, uh, that'll take you a long way now, too, as well as demonstrating learning some of these AI skills and things like that, that not all the people who've been around do have or have taken the time to learn.

Speaker A: Right. And, uh, even the AI stuff that we're learning now, in 18 months, who knows if it's even going to be relev anymore?

Speaker B: You're the world master, right?

Speaker A: I mean, like, to your point, Aaron, like, really, it's like the passion for learning and development and that, that, that accounts for a lot, right? Like being adaptive and being able to, like, shift gears and being, well, being able to, you know, dive headfirst into ambiguity and a new space and being comfortable with being uncomfortable is a really important skill set. And something that is sometimes shows up as an intangible, you know, on, on a resume. But, you know, it's something that you really, I think when you can speak to it and tie it to experience and of course tie it to impact, because, you know, in the, in the process, that's ultimately what everybody, like, is looking for when they're doing, hiring, at least at the top level. But when you get in a room, and I've seen this, like, with, with people I've mentored, when you get in a room, the ability to tie that story into that willingness and passion to learn and grow is the thing that ultimately gets you to where you need to be.

Speaker B: Well, awesome, Jared, that takes us just about the time just to close. Where can folks find you? Are you active on LinkedIn or X or.

Speaker A: Yeah, so I'm on LinkedIn. Uh, feel free to connect with me on LinkedIn. Um, I also have my own, uh, personal site, um, @jared forney.com where I do writings and musings every once in a while. Those are probably the two main places you can find me. I'm always happy to try to where I can like, chat with folks. I love talking to people in the industry. I was just at the reops rally around Research Ops last week, uh, which is a really great conference. Uh, hopefully we'll see some, uh, some things come out of that. So I try to do some networking around. But please come say hi to me. Um, connect me on LinkedIn and I'm, you know, if I have the opportunity and time, I love to chat with people in the industry because it's just a really exciting time to, to be in our field. Um, and I think it's. There's no better time like the present for research operations professionals in particular.

Speaker B: Yeah. Hear, hear. Well, thank you, Jared, so much for being here. Thanks to our audience for joining. And even if we didn't get to your question, we'll try to respond, uh, in our follow up. Uh, thank you everyone. Have a great rest of your day.

Speaker A: Thank you. Uh,

Speaker B: thanks for listening to Awkward Silences brought to you by User Interviews theme music by Fragile Gang. Hi there, Awkward Silences listener. Thanks for listening. If you like what you heard, we always appreciate a rating or review on your podcast app of choice. We'd also love to hear from you with feedback, guest topics or ideas so that we can improve your podcast listening experience. Fans, we're running a quick survey so you can share your thoughts on what you like about the show, which episodes you like best, which subjects you'd like to hear more about, which stuff you're sick of, and more just about you, the fans that have kept us on the air for the past five years. We know surveys usually suck. See episode 21 with Erica hall for more on that. But this one's quick and useful, we promise. Thanks for helping us make this the best podcast it can be. You can find the survey link in the FM episode description of any episode or head on over to userinterviews. Com Awkward Survey.