Teach Your Team to Thrive with AI feat. Hannah Calhoon from Indeed

Speaker A: I just got tingled. I got super excited about this. I want to lean in. Uh, can we, like, call this Vibe hiring or Vibe. Let's coin the term on this podcast right now.

Speaker B: Vibe hiring.

Speaker A: Yeah, Vibe hiring. I love it. Yeah.

Speaker B: I don't know. My marketing team is like, oh, God. Oh, God.

Speaker A: This is Next Gen Builders, the show for the growth and product leaders of tomorrow. Everyone's talking about AI transformations, but how do you actually teach 11,000 people not to be afraid of robots? Today, we're going to go behind the scenes at Indeed to see how they get their employees AI ready. Spoiler alert. It's not as simple as just flipping a switch. It takes deploying AI that actually helps people instead of replacing them, and combining the best of both humans and machines to move the business forward. Joining us today to talk through it all is Hannah Calhoun, VP of AI at uh, Indeed and head of AI Innovation. Welcome to the podcast, Hannah.

Speaker B: Thanks so much. I'm excited to be here.

Speaker A: Well, really excited. I mean, let's just dive right in. You've got a pretty interesting rollet, Indeed. I don't think I've heard anybody with that title before. Tell us what you're up to.

Speaker B: So listen, Indeed, I mean, I think, like many companies has been trying to figure out over the last couple of years what AI means for us. Um, and for us, there's sort of three different lenses of impact. There's first, this big question about what does AI mean for the labor force and the job market? Um, you know, lots. Lots of talking about that. Indeed is the world's largest, uh, hiring marketplace. We've got more than 600 million job seekers, 24 million jobs, 60 countries. And so we have this very interesting front seat to the global economy and how AI is starting to impact labor and roles and all of these sorts of things. So there's this question for us, there's a macro question, which is, what is AI going to mean for sort of the context in which we do business? The second level, of course, is what does AI mean for our products and services? Um, we help people find jobs and we help employers find talent. And both of those are processes that AI has a tremendous opportunity to improve and disrupt. Um, and so we're thinking a lot about how do we leverage the best pieces of AI to make it easier and faster and more enjoyable to chart a career or to find the right person to fill a role. And then at the third level, the sort of most micro level, there's this question about what does AI mean for us as a company and the way we do our work and for our employees experience of working at Indeed and being part of Indeed. And how do we make sure that we are taking advantage of everything that AI can do for us in terms of business productivity and outcomes and also just making work better for the people who work for us. My job is fun because I get to spend time thinking about all three of those levels every day and also the places that they intersect with each other.

Speaker A: So I mean, you're really partnering across the company to figure out how Indeed uses AI, but also how does the market use AI too, or how does that work? Exactly. How do the two work? Hand in hand?

Speaker B: Yeah, we are, I think, like many companies, working to bring AI into our products and services in ways that are thoughtful and actually value additive. Right. So job seekers and employers don't want a bunch of AI hype. They want AI that's actually going to make the product experiences better for them. And I think we've done a lot in the last year and a half to roll out new features and we're really excited by some of those results. And then also, like many companies, we also recruit talent and manage talent and try to motivate talent. And so it's, it's really interesting to be able to watch sort of like think of it both as a product and also as an experience that we're going to put our people through and to be able to test some of those same product experiences, some of those same patterns for sort of AI human interaction inside our own operations.

Speaker A: That's great. I mean, basically your team becomes almost a petri dish for what's going on outside and you get to learn and experience what others are doing too.

Speaker B: Yeah, we're really lucky and it's a lot of fun.

Speaker A: That's really cool. Um, how far along are you in the transformation of Indeed? Maybe internally, uh, in adopting AI and maybe what are things that have worked and other things that have been maybe more challenging?

Speaker B: Yeah, so I would say that we're in the middle of our journey. We spent a lot of last year doing smaller scale experiments and pilots, trying to get tools into the hands of all of the folks that work at Indeed and giving them space and guardrails and sandboxes so they could play around and tell us what was useful. And I think at this point we have a series of hypotheses around the largest, most interesting opportunities. And now we're really trying to go after those at scale. So a lot of our focus last year was our sort of key kr was usage. Right? Like what percentage of indeedns are using an AI tool in their day to day work on a weekly basis? For example, this year we've really shifted gears and said okay, usage is great, but usage doesn't drive business impact. And so we've said okay in a core set of verticals where we think there's a lot of opportunity. We not only want to drive usage, but we, we really want to drive. We want to measure like what does the usage get us? Are we saving people time? Are we improving the quality of decision making? Are we improving like customer satisfaction scores for example? Right? Like what, what are the things? Like AI is great, but why is AI great? But we aren't at a place where we've like rolled that out at scale and it's working and it's perfect and I can like, you know, set up shop and go home. And so we're not at the end, we're like, we're squarely in the middle. There's a couple of things we've learned, but one of the biggest ones has really been around like what does it take to get the, the maximum value of, of your investment in these tools, um, out of the tools and out of your workforce? Um, we have launched a bunch of different AI tools and every time we release a new tool we create a chart where we look at what percentage of the usage is driven by what percentage of the relevant employee population. And out of the gate it is a beautiful pareto curve where 10 or 20% of the early adopters, um, are driving 80 or 90% of the usage. Right. It's exactly what you would expect. And getting from that Pareto curve to deep integration of a tool into your workflows and therefore maximum impact on your business requires going to all of the other people and like actually being able to sit down with them and say, here's why this tool is useful for you in your job, in your workflow. Here are ah, three or five ways that you should try using it in, you know, as part of your job. And that requires like very function specific, role specific training that takes into account the work processes and the journeys that our employees go through every day. Um, and so we have done, we've got a great sort of like AI101 foundational curriculum that's designed for everybody at. Indeed. And then we've got a whole host of function specific hands on training programs where we go and we say, okay, we're going to spend a lot of time just with our software engineers. Right? We're currently rolling out this AI coding essentials program. Um, and over the course of three months, we're going to put 2,200 of our swe's through hands on training to help them make better use of the various engineering AI tools that we've rolled out across the company. And then we're going to go do a version of that in sales and we're going to go do a version of that in finance and we're going to go do a version of that in marketing. And um, because like, even a horizontal tool that's sort of like generically useful, think about like a Gemini or ChatGPT or a glean or one of these things, right? It's useful for different people in different ways. And if we put the burden of figuring that out on our employees, it's just going to be a really slow process and we're trying to move fast.

Speaker A: I agree. And where have you seen the resistance? Is it resistance? Are people overwhelmed? Uh, are there just too many tools and it's like changing too much? Like, why aren't they all like leaning in and taking advantage of all this stuff?

Speaker B: Yeah, so I think some of it is, um, there are a lot of tools. And one of the blessings and the curses of working in this space right now is that there is a new cool thing every other week. And so folks do need support navigating the options. And, and we have tried as a company to take an approach that says we're not going to say you can't use something, but we are going to pick some things that we think are the highest quality enterprise bets. And we're going to provide support and training and paved paths around using those options because we've got our explorers, and our explorers want to go and touch everything and try all the stuff. And then we've got the people who just want to do their job and they're like, I just want you to tell me what to use to solve my problem. I want to learn how to use it and then I don't want to worry about it again. And so I think, I think a lot of it is actually is that sort of knowledge and accessibility. There's a, you know, a willingness to try a new tool that's going to make their job better, but they want you to be more directive about what the tool is. And then also it has to make their job better. And certainly, you know, if I go back a year when some of these tools were a lot newer, they were buggy and they hallucinated and people sort of Said, gosh, like, like, is this actually less work? Um, and so, you know, part of our strategy, which has been like a lot of small, controlled pilots to understand, you know, the edges of what the tools are capable of and where the highest ROI is, has also been validating. Like, is this a thing that's solid enough that rolling it out to the broader employee base is actually likely to be successful and contrary maybe to what sometimes people think? We have tried things and passed. We've gone through the early pilot with the vendor, we've looked at the data and we've said, gosh, we love the vision, we love the roadmap. We think this functionality is going to be really valuable for indeed when it's ready. But right now it's not ready and it's not worth the change management risk to roll out a subpar product to our employee base because it will erode some of that trust and some of that willingness to experiment. And we want to, want to like, hold on to those arrows until we can use them in the places that we think will be impactful.

Speaker A: Uh, I mean, that makes sense, I guess. On that note, you know, the AI market is evolving at a speed I've never seen before. There's always new tools, the models are changing, there's new capabilities, there's new acronyms like MCP and A, to A, whatever. It keeps changing so fast, you know, for you as you're building the roadmap for indeed. Like, how do you plan on a moving target, uh, where everything constantly is evolving. Can you plan or like, how do you approach it?

Speaker B: Yeah, yeah, it is, it is both really fun and really hard. One thing that we have really leaned in on, and I, um, have to credit a colleague of mine who leads our engineering and science for AI platform. His name's Chris Johnson because he coined the term the Lego block architecture. But effectively the strategy we have taken is to say, okay, we have a set of capabilities that we know we're going to need architecturally, we're going to need foundational models, we're going to need moderation, we're going to need evaluation and real time auditing, we're going to need memory, right? We're going to need systems for interoperating, um, between agents eventually, right? So many of the platforms that we build on and the platforms we're building involve agentic capabilities. We know eventually agents are going to have to talk to agents and that handoff and that communication is going to have to have a certain set of components around identity and security and information transfer. So we know the sort of like the pieces of things we're going to need and there are a growing number of options to fill any of those pieces. And so we sort of think of the world as like we're building an architecture with Lego blocks. And so you've got your moderation block, and right now we have one or multiple solutions, either first party or third party for moderation. But we're building the system so that if two years from now there's a much better way to do moderation, we can effectively pull that block out, stick a different block in. Right. So optimizing for modularity, optimizing for flexibility, under the assumption that we do want to be able to make investments, maybe not 10 year investments, but multi year investments in platform capabilities. Because we need to move fast, we need to move to scale. But also we may not make the choice that's the right choice today is almost certainly not going to be the choice in three years, that's for sure. And we all just need to be comfortable with that. Right? We are architecturally optimizing for long term flexibility. I think the tension though is that because things are changing so fast, there's also this deep temptation to hop to the next thing really quickly. And part of what we're starting to see is, uh, AI coding assistance is a great example. There are so many of them now, and every week someone comes out with a new one and says that it beats benchmarks in a certain set of ways. Um, my personal hypothesis is that most of these platforms, they're eventually going to get to parody. And so somebody being a little bit ahead now or a little bit ahead in a month, um, is like maybe from a long term perspective, not that important. And so picking a couple of partners who we think have the right philosophy, have good design principles, are really great to work with and saying, okay, your services may not always be at the very top of the benchmark, but we have a lot of confidence that over time you'll be at parity or better than any place else we could go. And so we're not going to do the thing where we hop from assistant to assistant and try to optimize every way is actually, I think, going to simplify our lives and help us to drive more impact over time. Because every time we bring out a new tool, you know, new onboarding, new integration, new training, new thinking about how it connects to data sets, new connection to the MCP server, and so there is, it's like you both need to be super flexible and you need to Be willing to make some bets and hold your ground for at least some period of time so that you're not just being sort of buffeted around by the winds of change.

Speaker A: I mean, that's actually really great advice. I've also seen the counter, uh, argument where some people feel paralyzed by the pace of change because, you know, they don't know if what they're going to pick is the right one, so they just do nothing. And I actually think that may be creating more pain for the long term.

Speaker B: 100%. Right. I, um, have told my team multiple times at this point, I need you to make a choice. We're going to make a choice. If it's not the best choice, that choice, like, that's on me. Right. But we're going to make more impact on the business. If we choose a good or even a great but not perfect solution now and we start driving towards execution and scale of implementation, then if we wait until we're absolutely sure, which of the 12 alternatives you're considering is the very, very best one. Right. And so that's like we sort of actively have that conversation all the time.

Speaker A: How much are, uh, the tool selections or the tool excitement coming, bottoms up organically from the team members versus you as a central body or central team, starting to set the guardrails and guidance for the groups?

Speaker B: Yeah, um, definitely some of both. I would say every week we have a bunch of slack channels that are devoted to AI exploration and innovation. And every week there's somebody in the AI channels with, oh my gosh, I tried out this really amazing thing and can we get it at. Indeed. Which is great. Right? You have this amazing source of ideas. I would say our team does a lot more of thinking about. Okay, somebody's come with this tool that they're really excited about. Let's understand, what's the use case that the tool is actually solving? What's the problem? And do we have something in the existing portfolio of solutions that solves that problem? Well, and if so, this is an interesting data point around training and enablement and helping people understand how to get the best use out of our platforms. Um, but if this is sort of net new, or if it's something that we can do somewhere else in the platform, but this is meaningfully, exponentially better, then there's a really interesting question around. Okay, how do we understand very quickly what the business case would be for bringing on a new tool? And how big a problem is it? Is it a problem that drives a measurable business outcome? How many people does it apply to what's the cost of the thing, all of that type of stuff. So it's definitely both directions. Um, and to be transparent, it's also the total opposite way, which is aggressively top down. Um, somebody at the board level or in the C suite was chit chatting with another friend of theirs who runs a company and they said, oh my gosh, we love this tool. And then you get the email from the CEO that's like, hey, when are we rolling this thing out? You're like, oh, okay, that's interesting. So I think part of what's really fun about the space is that everybody is exposed to this stuff and thinking about this stuff. And so, um, the ideas can come from anywhere.

Speaker A: By the way, that happens to me all the time. I get so many pings about new tools. It's fun, it's inspiring, but it also can be overwhelming. Have you found that as people are using these tools, it's inspiring them, uh, in different ways of building capabilities for Indeed's user base? If you think about the features they build, um, is there a correlation between what they use to build versus what they build?

Speaker B: Yeah, that's an interesting question. Uh, I mean, I do think so. We started our exploration of what to build for job seekers and employers probably a good year before we really started seriously thinking about what, what we wanted to build or buy or otherwise partner to develop internally. Um, and I don't know if that was the right order or not. Um, and so I think for a while the learning was moving in the other direction. We were like, oh, we've discovered that this sort of problem is really hard to solve for job seekers, which means it'll probably be really hard to solve for our employees too. Or, oh, we know that job seekers, as they interact with our chat agents, really like these sorts of formats. And what can we learn there from a UX perspective? Um, I think part of what we're seeing now as we roll things out internally is it is helping people sort of like as people are having the personal experience of how transformative it can be for workflows, it does sort of set off that spark of okay, just because there's always been this set of steps in the process of going through an interview. Like, like, we don't have to adhere to that, that sort of very linear process necessarily anymore. Right? They're like, oh, if I can have this experience where like, I, as the product manager and you know, like, like able to like, pull up and create like easy visual content prototypes and then use those prototypes and like, and sit in cursor with my engineer and we can, like, all, like, me and my ux, uh, lead and my tech lead can sit together and do live prototyping and cursor, where we're building something that we can all talk about and react to. Maybe that's an interesting mental model for how recruiters could work with hiring managers to think about a pool of talent for a role. Maybe it doesn't have to be. The recruiter goes out and writes a job Description and interviews 300 Candidates and screens them down and does all of this work and then comes to the end of that process and hands the hiring manager a folio with five resumes in it. Maybe that implied linearity and the real division between those roles actually isn't what we're going to see in the future. And maybe it still is, right? We don't know. But I do think it creates some really interesting thought experiments around the ways in which the world might change and that it's always easier to do those experiments when you've had that experience yourself of feeling the excitement and feeling the viability and realizing that maybe there's different things that could be better.

Speaker A: Um, I just got tingles. I got super excited about this. I want to lean in. Uh, can we call this vibe hiring or Vibe. Let's coin the term on this podcast right now.

Speaker B: Vibe hiring.

Speaker A: Yeah, vibe hiring. I love it. Yeah.

Speaker B: I don't know. My marketing team is like, probably having a, uh. They're like, oh, God. Oh, God. Um, um. Uh, yeah, I mean, I think, um, I think it'll be interesting to see, right? I mean, one of the things that, that we see across both our external products and our internal tools that's so interesting is the value of deep, rich context that lives in people's brains. And I think sometimes when you talk to people or you hear people get up and talk about how AI is going to transform work or jobs, they talk a lot about the sort of specific tasks or analytic pieces that AI, um, is great at. And then they look at job descriptions and they say, well, this job is mostly doing a bunch of these things. And so AI is going to come in and be really disruptive for this role. And it's not that AI can't be helpful for parts of the job, but so many jobs have this sort of deep contextual component. Um, and to pull an example from our product experience, um, there are probably more, but there are sort of three basic ways if you are looking for a job, um, that you, as a job seeker, might be invited to apply for a job on Indeed. And so one is we might send you, we might use AI to identify jobs that could be a good fit, uh, for you and send you an email and say, francois, we think you should apply for this job. And you get this email and you click on the link and you apply some of the time and you're a decently good candidate. Um, and so that works and it's super cheap and it's very automated and it doesn't involve humans and it's very, very efficient. If we were optimizing around efficiency, um, there's a second version of how you might get invited which is that a human recruiter so sits down at Indeed and they search through the millions of job seeker profiles on our database and they find people they think are interesting and they send them emails to invite them to apply for the job. Um, and uh, that is um, more effective, slightly more effective than the AI version, but it is very, very time and resource intense. Um, but you can understand why it would be better, right? The recruiter has a really good sense of who they want and so they're being more picky and all of those sorts of things. Um, and when you get an email directly from another human saying hey, apply to this job, that's more compelling than getting the email from a computer. We now have a product, uh, that blows both of those other things out of the water. Actual literal multiples of improvement in terms of the likelihood that a high quality candidate who will eventually be hired applies. Um, and that's that the AI does all of that work to pre screen and identify possible candidates and then they provide them to the recruiter with little AI generated blurbs about why the candidate could be interesting. The recruiter goes in and like clicks on the ones that they're interested in and sends the emails. And those people, right, the folks who have been invited by the AI and human working together are the very best, right? That's the best of the patterns. And I think it's a really lovely example of the fact that the AI can go out and do. It can, IT can scan 600 million profiles in the job database, right? It can do the matching and the comparison. It can think about what we know about the job seekers wants and needs and their profile and their experiences. But at the end of the day there is so much more in the recruiter's brain about what a great candidate for the role is going to look like than they have put in the job description that having that person who looks at everything and applies that additional context and that additional human judgment, um, is still something that we have no idea how to mimic technologically. Right. And I think increasingly we're going to find that many roles, even roles where we don't necessarily think about the amount of sort of human context and judgment and connection that are necessary, still have these very, very core components, um, where the AI can be super helpful, but only to a point. Um, so that's sort of like, I think we're seeing that in sort of both sides, both the internal and the external. And we're trying to figure out how do we pull all those insights together on both sides and make good choices for all of our stakeholders.

Speaker A: I mean, that's super compelling. It is another example of a human in loop, drives better outcomes. AI can remove some of the drudgery that's out there, but it's the human and AI connecting together, which I think is really important. So we've talked a lot about technology, and as technologists, we love technology. Um, but as leaders, building culture is extremely important. And it's another beast entirely when we talk about AI. Um, how do you think about leadership, change management, getting people on board with this massive transformation we're all getting into.

Speaker B: I was literally talking to a set of people last week and I was giving a talk and the headline message of the talk was technology is less than half the battle. Right. Um, because just having cool tech tools, like, I love a cool tech tool, uh, but that is not actually the thing that is going to drive meaningful business impact at scale. Um, so I think there's a couple of things that we have found have worked for us. Uh, one is like, you need the leadership buy in. Right? You need leadership to say, this is an important trend. It's coming, it's coming quickly. Um, we need to all collectively be focused on, on taking it forward. And we have as a company, every fiscal year a set of four or six top priorities. Um, AI in its external and internal implementation has been on that list for the last couple years. This year, it's two of the points on the list. And so we've said, okay, we as a company are committed to making this transition. But also, as you and I have talked about, execs, being really excited about something is not, is sort of necessary but not sufficient. Um, you also have to get everybody in the employee base excited about it. And I think some of the things that we have tried there that have been somewhat effective, um, are we are creating, trying to create spaces, safe spaces for experimentation, communities for folks who want to push the envelope and are trying stuff to connect with each other and to share best practices. We have the Spotlight series where we go find people who have implemented AI in interesting ways and parts of the business and we do little blasts on them, um, in newsletters and in Slack and in otherwise. Um, I think we've tried to be straight with people and transparent about the ways in which we imagine AI might change jobs and work. And we've leaned into the fact that some people are nervous. Right. And that there are concerns about what does this mean for me or my role. And we've really said, listen, there's a lot that's coming that we don't necessarily understand but we believe really strongly that every single one of you, if you're a software engineer or a customer support rep or you're in legal or whatever it happens to be like every one of you is going to need AI skills and AI comfort and familiarity and the ability to use AI tools successfully for your career to continue from here. Right. The world is changing. You're going to need to be ready. We're going to make you ready. Right? We're going to put every single one of you through function specific, role specific training. We're going to give you the best in breed tools that we can find. We are going to make sure that you are making this transition into this new AI powered economy with us. And we don't know what that's going to mean over time for how many people will work at indeed five years from now. Right. Um, can't control the future. What we can control is trying to implement technology in a way that is worker centric, that is empowering. Um, and as we make choices about which tools to roll out or any of those sorts of things to make sure that we're actually really listening to your feedback and that we're only rolling things out when we think they're actually going to be useful and helpful and accessible and all those things. Right? So that we're not going to have to convince you once we show you how it works, we won't have to convince you to use it. Um, you'll want to use it because it's helpful and it reduces that toil and that drudgery and all that stuff we talked about earlier, that makes sense.

Speaker A: So with great power comes great responsibility and with Spider man had to bring that reference in. Um, but you know, like with all great like technology advances there's amazing things you can do but there's also the negative side where some people are abusing it or there's other concerns that come out. Have you seen situations maybe you know, externally or internally, whatever that is, where people are playing the game and using their own AI against your AI or uh, finding different ways of, uh, bypassing the AI, uh, you've built?

Speaker B: Yeah, I mean, I think there is always, um, there are always people trying to game a system. Right. Um, and AI is just throwing more fuel on the fire around trust and safety and moderation and verification and all those sorts of pieces. Folks are like, well, are people using AI to beat resume screeners? And it's like, well, you know, there used to be like pre generative AI this practice that I didn't really discover until I got to. Indeed. Which tells you how up on things I am. Um, where people, you know, ATS systems historically have used keywords for more traditional machine learning matching to look for candidates. And so people would do this thing where they'd write their resume and then in white text at the bottom so you couldn't see it. They just fill in like dozens and dozens of words that they thought might be useful for TS matching. Yes.

Speaker A: Oh my God, that's amazing.

Speaker B: Um, so this has always been a thing, I guess is maybe the point. And yes, now you've got these situations where employers come to us and they say, well, doesn't every job seeker with a pulse have a great beautiful chatgpt resume? Or I've been interviewing folks for software engineering jobs and I think that they are using AI to beat their coding assessment tasks. Right. Um, and so it does create a new level of difficulty around going out and being able to really verify information. Um, it's going to create some, like, some, some like new patterns we're going to need to see in terms of, you know, like, how do, how do we assess talent, how do we collect information, how do we know people are who they are? We don't want people to like suddenly flood employers with 10 million AI generated job applications. Right. Like that's not going to be good for anybody. But I do think, you know, like, uh, at the end of the day, you know, we believe that we're going to be able to provide the best experience for job seekers and employers if we have that deep, rich context on what both of them are looking for. And we can use that context to suggest matches at the core of what indeed does for a matching marketplace. And one of the things I think is so cool about generative AI is that it gives us new modalities to collect more of that context and then utilize it Effectively. So if I'm a job seeker, let's say I, um, I've been working a bunch of retail, retail jobs. I've been a barista, I've worked at the Target, I've like, you know, been doing all of these sorts of things while I have in parallel been doing night school to get a degree in content marketing. Right. In the old world I put my resume onto. Indeed. And it reads all the stuff I've been doing and it just naturally is going to suggest a bunch of jobs in retail and service and all the. It's like, it's looking at my experience and it's like matching me with jobs, right? But like, I'm trying to make a shift into content marketing, right? And I want it to underweight all of those things and overweight the internship experience, the portfolio project that I've just done, right? And in a world of generative AI, I can tell it that and it can understand that and remember it, and it can rethink effectively, like, the relative weight it's giving to different parts of my profile. Um, or like, you know, humans change their minds all the time, right? Like, maybe I have been searching for content marketing jobs in New York, and, you know, I've suddenly found out that, like, I have a family member who, you know, is going to need support. And so even though for the last three weeks I've been doing nothing but searching for content marketing jobs in New York, I like, as of tomorrow, what I'm looking for is a job that pays a minimum of X number of dollars in Madison, Wisconsin. You know, traditional ML models are looking at my past history of behavior and they're like, you want a content marketing job in New York? And so they're going to keep trying to serve me content jarpening marks in New York, right. Even though my life and my needs have changed. And so I think there's this really wonderful opportunity with generative AI for us to have such a deeper understanding of skills and experience and motivations and which of those things matter to individual job seekers at any point in their life, which then allows us to serve them increasingly relevant, increasingly useful jobs. And so maybe the universe gets cluttered with a bunch of nonsense job seekers and nonsense jobs and nonsense applies, right? Um, and so you've got this. The problem of trying to find the needle in the haystack, right? But if you have really good data about the needle, finding the needle and finding the job that is the perfect job for the needle is actually very doable. And so we are doing a Ton of work to invest in trust and safety and identity verification and all of the things to keep abuse and fraud out of the platform. And also we think that, like, this ability to really delve more deeply into context and needs on both side of the marketplace will help us fight the sort of AI clutter in a different and in some ways, like, more productive way.

Speaker A: I'm just inspired. I love, I love the possibilities and I love the approaches that you guys are taking and just really thoughtful. Okay, Hannah, I have one last question. I ask all my guests this question, one of my favorites, um, which is really your oh, shit moment. You know, when you think about your career, all the things you've done, there's, uh, always highs and there's lows and there's incredible learning moments. So what is your oh shit moment?

Speaker B: Yes, um, there have probably been more of them than I am proud of, but, um, but the one that comes to mind. Okay, so a couple of jobs ago, I started a venture studio in New York. And for those who aren't familiar with the venture studio model, you basically try to create a place where you go out and find founders and help them start companies. And we were specifically focused on helping people start companies where their target consumer was low income households because we thought that was sort of an interesting gap in the existing capital market. And it's really hard for a bunch of reasons that are not worth going into now. Uh, but we had a founder, uh, who came to join the first cohort of our studio and he was spending a lot of time at the SNAP office. Snap, um, is the Supplemental Nutrition Assistance Program, better known as food stamps. Um, and he would go every day and talk to people for UXR purposes. And he was noticing that there were hundreds of people standing in line waiting to turn in their paper forms while they were playing Candy Crush on their smartphones. And, uh, he's the next Facebook LinkedIn guy. And he was like, well, this seems like, you know, there must be a better way. And for what it's worth, the company is actually, it's called Propel. They now, uh, do other things related to SNAP, but they, they help about 5 million families across the country, uh, you know, work towards financial stability. And they're cool and you should check them out. Um, but so, so they like, they were like, okay, we're going to create a way for you to submit a SNAP application on our phone in New York. And they started doing that and they'd been doing it for a couple of weeks and we got a cease and desist letter from the city of New York, which was, which was definitely an oh, shit moment, right? Um, having the government send you a legal document is never super awesome. We were probably six months into running the studio at this point. I still wasn't entirely sure I knew what I was doing and I was like, oh, oh, oh, okay. We have, we have made some mistakes. Um, and I think. So we had made two mistakes and I think they're probably both worth mentioning. The first was like, like, legitimately there was like a problem with the applications, which was that New York was a bit behind the times and had not yet approved electronic signatures for benefits documentation. And so the apps that were being submitted to the city, um, were invalid because they didn't have physical signatures attached. Um, and listen, to their credit, the team went out and found every person who'd submitted an application and like, drove to their house and had them sign a piece of paper and got this sorted. And, um, but it was a really sort of like, good reminder of the fact that, like, you know, to the extent that tech talks a lot about moving fast and break breaking things, right? Like, if what you're moving fast on is like people's government benefits that are going to allow them to, to buy groceries for the month for their kids, right? Like, you can't fuck that up. Um, so you got to figure out how to move fast and also like, protect the customer base. Like part one, we screwed up. They, like, legitimately, they couldn't take the signatures. We had to get that fixed. The second thing was just like, contrary to my maybe super overly optimistic view, the government was not super excited about some random crew of people in Brooklyn bringing technology to solve a problem that they owned. And they didn't control the app. They didn't love the optics of us writing pitch decks where we shone a light on how broken the system was. Nothing about this was exciting to the city of New York. And it was a good moment as well, because, uh, again, in sort of traditional tech VC land, if you're like the upstart and you're disrupting, maybe you can disrupt the incumbent out of business. And so of course they're not happy, but who cares, right? But in the government, the city of New York wasn't going anywhere, right? Like, disrupting the city of New York and having them be pissed at us for the rest of this company's history was a terrible idea. And I think about that a lot as I'm thinking about like, internal organizational change, right? Because what we're trying to do at indeed is make this really big, really fast technology shift and indeed still going to be there when I'm done, when I'm long gone. And so this notion that you're just going to push through with disruption and the righteousness of being correct about the way the world is going, it's not a helpful attitude. You have to make other people excited about the change. You have to get other teams at, uh, indeed to want to own the change and take on the change. And when you go to them and say you need to change your process, you need to change your tech stack, right? They need to believe that that's something that they want and you need to figure out how to do that while making them look good. Right? And so one of my team's core values and the work that we do is outcomes over ownership. And it's so important because our goal is to get indeed over this mountain, right? And like, honestly, I think the way we get there is actually by, like, making a bunch of other teams, like, into absolute rock stars. And if you can like, internalize, right, that like, you don't need the ownership and the ego, I actually think that, like, that is that like, really sets you up to, like, to succeed in a really different way. And so this was like, this was, this was rough. Um, it did not sink the studio. It obviously did not sync us or the company, but it was a really, like, lovely learning moment that I have found. I reflect on a lot of. Even though I'm now doing something really different.

Speaker A: That's amazing. Uh, and this is why you are where you are now. You're inspirational. You learn, you're a Next gen builder. I love it. Um, and thank you for all of your perspectives and ideas, uh, and views. We've gone from Lego Blocks to Spider man and Change all the way across. So, Hannah, thank you so much for joining us.

Speaker B: Thank you.

Speaker A: And thank you all for listening to Next Gen Builders. Look out for our next episode wherever you get your podcasts and please don't forget to subscribe.