Why Zapier Requires 100% AI Fluency (And How They Measure It)

Speaker A: M One of our biggest episodes this year was with Zapier's CPO, Brandon Samut. For today's essential Heretics 101 feature, he demonstrates AI agents automating, bar raiser interviews and performance reviews while explaining Zapier's four level AI fluency framework.

Speaker B: What's up, everybody? Welcome back to another episode of HR Heretics. Brandon Samut. What's up, man? It's good to see you. Where I want to start with you is talk about, like, AI fluency. And you had said on LinkedIn that AI fluency is a requirement for 100% of new hires at Zapier. Why is that the case? And how do you guys go about

Speaker C: measuring that AI fluency is a requirement for everyone? We're hiring in the company for a couple of reasons. The first is it's essential for fulfilling our mission. Right? Uh, Zapier's mission is to make automation work for everyone, and AI is a meaningful new way for us to make that mission real for so many more people. A second reason why that matters to me is more about, like, Zapier's tacit commitment to people we hire. You think about if you're in the job market right now and you're thinking about whether to go to this company or that company. You know, a lot of us have a leading sense that wherever we spend the next few years of our career is going to have a lot to do with the kind of opportunities we have for many years after that. And gosh, I would really encourage all of us, like, make sure we go somewhere where that's actively investing in making sure that we know how to use AI to, uh, supercharge the thing that we do best in the world or at least at work.

Speaker A: Brandon, I was listening to a podcast on the way up to the city today and similar thing that a company was saying. It's a requirement for folks to come in with some sort of AI. You know, I'll use fluidity because we're using it here. But their bar was curiosity, and I'm wondering if yours is something different.

Speaker C: You have to be a little bit more than curious to get a job at Zapier, but I appreciate that as a starting condition. So I would call it curiosity, like necessary but not sufficient, if that makes sense. At Zapier, we have kind of like four levels on our AI fluency rubric, and the first is unacceptable. It just means you just, you're not both from a skill and from a will perspective. The second is capable. And when we mean capable, that Means like you have some of the skills and some of the know how of like how to apply AI in your daily work. The third and next level in the rubric comes up that's called adoptive. Adoptive is, oh no, you're applying it, but you're applying it primarily to the work as it's conventionally done. So it's like, you know, the work is traditionally done a certain way. I'm using AI to practically improve my productivity or the quality of my output is, uh, by the way, like that's pretty impactful. But it gets even better because the top tier in the AI fluency rubric is called transformational. And that's for folks that are showing us that not only do they know how to use AI to uh, impact the work they do as it's traditionally done, they are also able to completely rethink how the work gets done in the first place. So it's almost like a, like a clean, safe, but you're starting more from uh, you know, like uh, you're kind of zero basing the whole thing and being like, gosh, if we were to start from scratch with the tools and know how we now have, how would we do this work?

Speaker A: Thank you for breaking that down. A lot of the issues I see out there right now is just literally like an inability to simply break things down so we can touch it and feel it. And that's really helpful.

Speaker B: All right, well, let's get into the meat of this. So how are you guys deploying AI and what do you think are like the most interesting use cases you want to show?

Speaker C: What we're looking at is Zapier agents. Right. And just to define agents really quickly, agent has the uh, kind of underlying LLM capability that we all know and love, but with the ability to automate tasks on your behalf. So at Zapier, like many organizations, we do a bar raiser interview. This is uh, a, this is intended both to apply additional rigor to hiring. We don't hire a lot of people every year, so everyone should be amazing. A, uh, bar raising addition to the team and importantly, it's, it's done by an exec.

Speaker B: Is that C level exec or VP plus? Like how do you guys think about it?

Speaker C: It's VP plus. Yeah, it's like the department heads effectively

Speaker B: and, and just contextually. How many employees is Zapier?

Speaker C: 737.

Speaker B: 30. And how many VPs do you guys have?

Speaker C: VP plus? We have 10.

Speaker A: Yeah, that was like my eyebrows just raised.

Speaker C: Love to see that. Now as great as These Bar Raiser interviews are. They're a bear to prepare for when you're like a, uh, busy exec because you're supposed to be looking across all of the prior interviews and scorecards and identifying, like, specific things to dig into. The non AI assisted way of doing this looks very inconsistent in practice when you watch, there's recency bias. Like, maybe I'm reading quickly through this scorecard and stopping more on this one. Like humans are messing with. And so is preparing for a bar raiser without a copilot.

Speaker B: I've had this experience a million times, by the way. It's like, oh, hey, Nolan has to interview this person. And five minutes before is when I'm scrambling and looking at their LinkedIn and trying to get my act together.

Speaker C: That's right. That's exactly right. And so what this agent does is I will point it at a series of scorecards for a fictional candidate that we'll look at in a second and it will draft the Bar Raiser interview guide. It's reading all of the data in this browser tab about Leslie and, uh, everything we know about her from our interview. It's also looking at her resume and it's creating the interview guide. And what's interesting about this one is it's been, it's been prompted to not only add questions that'll help us, like, really round out our perspective on the candidate, but it's explaining why it chose that question.

Speaker A: That's incredible because before I saw this, I was like, the questions are great, but you're still flying blind. Like, why are we doing this? What are we worried about? What are we trying to validate or devalidate.

Speaker C: That's right. The speed of iteration is so fast. Like, you know, I'd like an extra sentence on why each of these questions was chosen. All you gotta do is come right back on here and update your prompts and then retest. It takes, takes three or four months.

Speaker B: And so, Brandon, there's like a spectrum here. Some companies for their Bar Raiser interview have like the exact same questions, but it doesn't change the fact that you actually want the full download of what's happened in every conversation. I met with Joe Ed recently who runs, he runs recruiting at hbiq. And one of the things I learned from him is they're actually recording all of the interviews with metaview, taking the transcript, putting the transcript into ChatGPT, and then with the backend prompts, basically, like saying like, here's context on our org, on our product. How do you think they're going to fit in. What are the yellow flags, what are the green flags, what are the red flags? But another spectrum is go summarize these interviews, tell me about this candidate via the resume. Right. And you could do that with one click every single time, you know, an hour before an interview. And so there's just so many different ways that you can actually tinker with this thing, right?

Speaker C: A hundred percent. You also have a lot of choices around how much of the human you want in the loop. After it generated the interview guide, it stopped. It asked me to read it and say if I liked what I saw, if I wanted to make tweaks. If you tested this enough and you're like 99 times out of a hundred, it stops and asks me if I'd like to make tweaks. And I say no, because it's just, it's right there, it's spot on. You can take that step out and it will, ah, automatically create the document. And so this, this one I use regularly when we're hiring folks into the people team. That's that beer.

Speaker B: This is what I've been saying. I would show up unprepared to 95% of the interviews where I was the final interviewer. I'd say 97 out of a hundred times, the candidates getting greenlit from me. And this is a way to actually add value into that process without me being the person who's like, oh my God, I don't have enough time in the day to prepare for these things.

Speaker C: Well, now let's talk about another. I think something all of us can relate with, which is we're writing whatever our company's version of our performance review is. And usually that includes a self, uh, reflection. Right. I'm actually about to do it next week for myself. And I will tell you, like, I gotta go back into the vault to like, remember all the, you know, what did I, what did I do, what did I learn? Accomplishments and so on. And this is another place where like, humans are messy and imperfect and, you know, like, we're all, you know, apt to have, you know, recency bias, forgetfulness. So one of the things we're curious about is like, how can we help automate evidence collection for coaching performance reviews? What I'm trying to solve for is like, when I get a piece of feedback or when something good happens that I've contributed to, I want a running documentation of all of that so that I don't have the blank page problem when it comes time to write my, whatever it might be. My self evaluation for a performance cycle. My case for promotion. Zapier does almost all of its communication in Slack. What I want to do, I want to use a very specific reaction Slack. And that's what this trigger is. So when Brandon Slack account adds the blob notes reaction to anything in Slack, it's an emoji. It's an emoji. It's an emoji and I'll show it to you in a minute. And when I tag anything in Slack with this emoji, then do what's in the instructions, right? So it's like classic prompting, like role trigger. So, and again, there's the trigger again when I, you know, react to a thread, analyze the conversation, create a summary focused on what it's about, how it relates to my work or performance, and then it includes an example of what I'm looking for. Uh, I actually added this one last week. Include a link back to the Slack comment. So I have it for reference as I'm clicking through if I want, like more of the original material. So it's just, it's amazing, right?

Speaker B: Which, by the way, you didn't have when you first built this. That's what I like. The thing I got hung up on a couple weeks ago was letting perfect be the enemy of good and just like, get going. And then when you realize, oh, it's actually not perfect here, go make the edit, which takes 30 seconds, right?

Speaker C: That's exactly it. And in fact, that's exactly what I did. And it really did take literally 30 seconds to make this update. And then I tested it, right? And that's one of the things, of course, you know, we're finding so helpful is like you kind of treat an agent like you would an employee, right? I'm giving this a job to do. I think I'm being clear in my instructions. It's time to test. But the speed of like, build, test, learn and improve is so fast. And again, it doesn't require like, special skills or at least harder to develop skills. There's definitely some skilling involved in this, but wow, is it relatively straightforward to pick up compared to learning how to code.

Speaker B: The other thing is making me think like, so this is, you know, this is today's iteration, the future iteration, right. Is probably going to be living in the background of Slack to where you're not even going to have to hit this emoji you're about to show us, right? And it's going to just kind of be idling back there and then you're going to send. It prompts things like, hey, what are my blind spots? Right. And how I communicate with the rest of the team are like, you know, what do you think I'm doing really well at? Or what do you think I can be better at? And it's just reading all of Slack all the time and essentially like filling in your gaps, being your on demand coach.

Speaker C: You've got it. Yeah, I think that's right. And in doing so, by the way, like, it. That would be like the, I think the highest level, like agentic workflow is it's doing work on your behalf with little to no human LED triggering.

Speaker A: Well then also I had, is. Is a person, right? Imagine, you know, no one, you're, you're pulling up me and I'm an AI agent and I'm talking just like this. And I'm not only doing all that, but now I'm talking through the feedback with you. You're almost like mocking through a conversation or going into more detail about the feedback.

Speaker C: That's right. We're going to be testing something internally in the second half of this year along these lines where it's kind of like, you know, coaching in the flow of work would be a good way to describe it. Where it notices that something's been discussed or something's going on in a meeting and it will basically say, hey, sounds like X, Y, Z came up in that meeting. If you're looking, for example, for a way to like give some feedback about that, like, do you want to talk about that, do you want to role play it? Or hey, like, you know, looks like you were facilitating that meeting. I noticed that it didn't sound like the next, uh, steps were super clear. Here's what I mean by that. Like, here's what I thought I heard. Like, is that right? Like if, if you think that's, if you think there's something to that you want, you may want to, you know, follow up with the group, make sure those next steps are clear.

Speaker B: Wow. I mean, so all three of us, by the way, do coaching and you know, you can imagine, right, if you have like all of the NDAs and like legal BS figured out, but you can actually imagine like coaching clients opting in for these sorts of things and you're getting actually data from like what's actually happening versus what the coaching client is telling you and to help improve. So it's, I think a lot of times people get really worried about, oh my God, like the agents are taking over versus really like, you know, it's like steroids and like really turning this coach or whoever it is, like using the data into more of somebody who can actually help you and help your performance.

Speaker A: Hey everyone. We'll be right back in a moment after a word from our sponsors. AI change is how work gets done. Recruiting is no exception. Medaview is the AI platform purpose built for recruiting. If you're looking to level up your AI adoption and see real impact from intake to offer, it's a no brainer. Medaview's suite of AI agents works for you at every stage of the hiring process. Their platform powers instant job post creation, AI powered, uh, candidate Q and A, insights on pipeline performance, compensation and so much more. I've seen firsthand how tough it can be to extract consistently high quality data from candidate conversations. I've been in candidate debriefs where interviewers told us about their feelings instead of what actually happened in the interview. And I've seen VPs and CEOs block hires because they didn't learn anything from skeleton scorecards. Medaview changes all of that for every candidate in every pipeline. Bring radical efficiency and precision to your hiring process with perfect data from every conversation. Medaview covers all of your compliance needs, works with all of your existing tools and takes less than three minutes to set up. Join talent leaders at uh, more than 3,000 elite companies like Brex, Deal11 Labs and Quora and check out Metaview. Just head over to medaview AI heretics to talk to one of Metaview's product experts or get started for free. You'll be up running and AI enabled in minutes. I talked to a founder this morning who's, I mean I feel like these are, you know, this is incredible. Xavier is incredible. I talked to a founder this morning who's trying to create that and like a coach in the workflow, like a person. And I said, you know, the, the day that someone figures out how to, how to create 10 of me in an organization that can talk like this to 10 people at once, a hundred people at once in the company on demand and their workflow.

Speaker C: It's coming, it's, it's super interesting. And so, you know, take a look at the, at the screen right here. So this is, this is slack at Zapier. This is our general channel, it's our all company channel. And you'll see Wade does a, uh, what he calls his Friday five. It's like five really interesting things going on around the company. And he had a sixth one this past week because we did a big launch moment for what we're calling HR by Zapier. They're really starting to package up some of the use cases that we're finding helpful inside our own company and making it easier for other people or HR teams to put, uh, those to work in their organizations. And so just a shout from Wade to the team and I, and if you'll notice right down here, oh, there's that reaction blob of notes. That's the trigger when I use that emoji, not when someone else uses it. When I use it. That's the trigger for the agent. And with that in mind, I'll show you what happens. Here we go. So when that happens, I have a Google running Google Doc and it will drop in the summary. Now, I did this yesterday because it takes for it to run the loop. It takes like 15 minutes or so to pick up on it because I don't have it running, have it running like periodically to save some compute. But you can see right here, when I triggered it yesterday, got a little summary of what it was about, little context on the overall kind of wrapper, and then a link directly back to the message that the evidence is pulled from. And you can imagine using this throughout, you know, whether you do semiannual reviews or annual reviews, just tagging stuff up throughout an extended period of time. And then when it's time to sit down and make that promotion case or write yourself reflection for your performance review, like you've got what is ultimately the thing that takes the most time if you want to do a really good review, which is having a fairly comprehensive set of evidence to build on.

Speaker B: Um, yeah, I mean, my advice to people and what I tried to do was I let. I literally kept a running note, right? And I would try and add to that thing once a week, once every couple weeks. But, you know, turns out life happens. You get busy, you forget, right? You have all the human problems and you just like, don't do it. Or like, maybe you're running from meeting to meeting and just clicking a little emoji is a perfect example of like so much faster, so much easier to be able to capture what you need to capture.

Speaker C: 100% sick. Let me make one point about some of these workflows that y' all were hitting on earlier indirectly, which is about employee choice. And you take some of these workflows, right, like this one, and there's a big difference between, you know, our company telling everyone, you gotta use it in us, like treating this more like a M market, right? Which is if my team makes A credible case for how this is gonna help our people be great, be recognized, you know, have a, uh, performance review that feels like more comprehensive, more fair, higher decision quality, then folks are going to pull this off the shelf and use it.

Speaker B: Yeah. The effective marketing piece is actually something I hadn't thought about and, you know, said a little bit differently. Brandon, a lot of CEOs are saying you must use AI. I better send this message too, and people better be using it and teaching me how to use it. But what you guys are doing is saying like, no, no, no. Leadership, like needs to be native in this and we need to do a great job of teaching everybody how valuable it is and why it's so dope so that it's so obvious for them that they should be using it.

Speaker C: I think that's the higher bar, but I think it's the right one.

Speaker B: Yeah, but actually it takes leadership doing real work. And, and this is like, this is the, this is where we are in this cycle is like, there are very few companies still that have real life use cases like Brandon is showing us today.

Speaker A: Yeah, this is, this is so helpful. I mean, for one of the biggest things for me, maybe for the audience is, is just to demystify how simple these use cases actually are. Because again, you hear the big cloud of use AI and HR and everyone gets, you know, deer in headlights, Google eyed, paralyzed. And I understand why. But again, like, we started the conversation breaking it down and just unpacking exactly what you mean so we can touch it and feel it. It's like, oh my gosh, not only is this demystifying it, but I want to actually go do this because I

Speaker C: understand it just interpersonally. But also, even if you're just, even if you were just interested in maximizing the company's performance, you want to maintain a sense of belonging across the team

Speaker B: and you tinker as you go. Like, that's the other thing I want people to take away from this is like, yeah, like testing stuff like that. But like, even if you can't, ensuring that there is a human in the loop step, which most of these you have shown, Brandon, have a human in the loop step. Right. You could just do it live and be like, didn't love that one. Like, let me kick this off right now to get this off my, you know, my plate and then take five minutes to go update the prompt so then next time it doesn't make the same mistake.

Speaker C: Exactly. You know, if you're interested, I've got one more, please. Okay, this next one is the most sophisticated build. It's not one that I built and it's not one I could build even today. But I'm sharing it to make a couple points. And the first point right as I pivot over to it, is that, you know, within a company you don't need 100% of folks to be, you know, deep, deep like power builders of these use cases. You need enough within each set of use cases to be able to build and troubleshoot and maintain the workflows. You know, the rest of us, you know, we can pull that stuff off the shelf and put it to work. And if you think about it that way, it should really increase, uh, all of our sense of possibility about what we can do. So here's a neat one. This is a AI one on one meeting coach. This is for managers who in many cases do one on ones or coaching conversations with their direct reports. And what this one does is it listens in on those meetings with the consent of both people. And uh, it takes notes. It then analyzes those notes for very specific qualities that we believe make for a great one on one or proxy and then it automatically provides coaching and feedback to the manager after the meeting. So we talked about human in the loop earlier. The only human in the loop moment for this one once it's turned on is hitting the record button. Everything else is automated and you can see here a couple things. So this is a workflow that blends automation with AI and if we're just to walk top to bottom here, it's, it's, you know, looking out for the recording once it's generated. Uh, it then importantly waits because Zoom or Microsoft Teams takes a minute to develop the transcript after the meeting. So it's intentionally waiting for that transcript to drop. It's then taking the transcript and then there are a couple code steps. And I give credit to one of our learning designers here on the people team, Emily Mabee. She and a couple teammates put this one together. But again, Emily and our team, we don't have developers on our team, but Emily was able to write zapier code steps. So in other words, just write code to do very specific things that aren't hard coded into our automation schema to do some really interesting things around structuring the transcript and reorganizing it, uh, getting clarity on who's the manager and who's the direct report. Like it's just pretty incredible. What she.

Speaker B: Calculating the, the talk to listen ratio is really interesting.

Speaker C: Oh yeah. I mean this is. And that's custom like, you know, like Emily made that using code by Zapier with the assist of an LLM. The other, like icing on the cake here is that as you do more and more of these one on ones and generate more and more of this feedback, it actually databases it and then provides thematic feedback about, hey, like over the last 90 days in general. Here are some themes I'm seeing. So, you know, here's something to keep doing, here's something to start doing or do something different.

Speaker B: Wow.

Speaker C: To give you an example of the output, I had a one on one recently and call it 45 minutes after the meeting, once the transcript was created and the feedback was synthesized, I get this in Slack and you can see here there's like kind of oh my God, you know, so overall strengths, do betters. And then the feedback trend isn't super useful in this example because I don't have a ton of data in the table that's aggregating. But you can imagine if you'd done like 20, 30, 50, a hundred meetings of this type, you know, over some period of time, you know, you would get genuinely useful, like thematic feedback.

Speaker A: Yeah, yeah. And you could emoji that or whatever and put it in your notes that you were talking about earlier.

Speaker B: Yeah. This one though, Kelly, is like, this is scaling HRBPs, right? This is like, hey, we managers suck. Every company has a problem with shitty managers, right. It's the hardest thing because you're not in that room to give them feedback. What Brandon and his team are doing here is they're essentially saying, well, as long as you press record, which again, uh, you know, six months from now, that's going to be the default, right. And that's. They're not going to even have an option. It's going to be how we work. What's going to end up happening here is they're going to get feedback from AI that is, that has the tinkering on the back end from us to tell managers exactly what we expect and what they did well and where they could grow. And then we're going to have it on our back end as well to be able to manage all of the things that we can't see. That is incredible.

Speaker A: Well, we said earlier, meeting them in the, uh, meeting them in the workflow

Speaker B: of their day, in the flow of work. Yeah, wow.

Speaker A: Yeah. Scaling us and then, all right, I'm the hrbp. You give me a hundred of these. I now use this as an aggregate dashboard instead of running.

Speaker B: Well, actually, instead, uh, even. Even. Yeah. So to Your point? Right, Right, Kelly. So then the next thing you're going to do is you're going to say like, okay, here's all the managers and all the feedback. Stack rank. My best managers, like stack rank managers, one to a hundred. Brandon, this is so cool.

Speaker C: Well, you have so many choices to make around this too because, you know, you, you get to choose. You know, is this one of those kind of on demand, off the shelf type of workflows that folks can choose if they want to sharpen up their one on ones? As a manager, is it a, is it a required workflow? If it's a required workflow, you have yet another choice to make. Actually, either way, you have another choice to make around. Like, are you collecting the metadata to in fact, you know, understand who your most or least skillful managers are, at least as it relates to doing one on ones. How does that influence trust in the workflow? How does that influence whether folks actually use it? Uh, like a lot of interesting things there. Our current theory of action for this one is we have kind of a golden path for this so folks don't have to redo all of this goodness that Emily and colleagues put together. No one has to make this up. Or again, it's a template, right? You copy it, you off your own stuff and you're good to go. But it's still on demand. So we don't require that folks use it. Right now we are making a case. Again, use, actually using like user stories, right? This is just like marketing, right? It's just inside your company, like anecdotes, like I'm using this, here's how it's sharpening me, like et cetera. And importantly, we're not currently doing anything with the data. Like we're not collecting it in some extra place. Different ways to approach it, all with their merits and challenges. One thing I highly recommend, no matter which way organizations go on a question like this is to be very clear and transparent with your team about things like this. What data is being collected, where is it being stored, who is it for, who else gets to see it? And if other people other than just me are going to see this information, what will it be and not be used for? And again, like, you can have lots of interesting conversations around like the right answers to those questions for a specific use case in a specific organization. But the most important takeaway is be clear and be transparent because that's the type of thing that builds trust and inspires folks to try more of this stuff out over time.

Speaker B: Wow. Yeah, it Reminds me of, you know, when I was in high school, we would have optional practices on Saturday, and we'd. We describe those practices as manned optional because, like, ultimately, like, they were kind of mandatory. And if people aren't using it, like, that's kind of a signal itself. But what you guys are doing such a good job of is just saying, like, this is the, like, listen to these people talk, right? And if you don't use it after listening to these people and it's been a couple months, like, you know, we'll have a conversation then. But, like, I love the idea of encouraging people to use the stuff that's just off the shelf that you guys are doing, that the people team is adding value to the organization. And it's like, this is the whole idea is like, it's turning us all into the Ironman suit versus the AI Overlords are doing the job. Wow. All right, Brandon, where can people find you?

Speaker C: It's the best place. That's where we're going to keep sharing use cases and things we're learning along the way, including the hard stuff that doesn't work. So you can see around those corners.

Speaker B: Ted, I didn't ask you this before the show, but we are going to require that you come on the show every couple of months and give us the latest updates and the stuff that you're building so we can share it with the rest of the world. World.

Speaker C: Oh, challenge accepted as a good excuse to make sure we have a good point of view on, uh, the stuff we want to showcase. Sick.

Speaker B: Thanks so much, Brandon.

Speaker A: Thanks, Brandon.

Speaker C: You've got it. Good to be with you.

Speaker B: HR Heretics is a podcast from Turpentine, the network behind Econ 102, Moment of Zen, and Turpentine VC. Subscribe five stars. Share it on Apple, YouTube, Spotify, anywhere you get your podcasts. All the things.