Future Ready Lawyer · 2025-11-19 · 37 min
Key moments - from our scoring
Substance score
54 / 100
Five dimensions, 20 points each
This episode examines the tension between AI as incremental evolution versus transformative revolution in legal work and legal education. Speaker A from a university faculty leadership position argues that over the next few years, AI integration will likely be evolutionary rather than revolutionary - with hype cycles deflating before we see fundamental reshaping of the profession. The discussion centers on agentic AI (agents that complete tasks with minimal supervision like document review and case summarization) and real-world impacts: Thomson Reuters data shows 28% of legal firms currently using generative AI, yet 50% worry about job displacement. Real examples like Klarna's 700 customer service layoffs and emerging "AI-first" company strategies (prioritizing AI automation before hiring humans) illustrate the stakes. However, Speaker C counters that law remains fundamentally people-centric due to high-risk, complex matters requiring human judgment and client relationships. Notably, despite efficiency gains, law firms increased graduate hiring by 6% according to recent AFR reporting, suggesting AI may free junior lawyers from grunt work - document review, e-discovery - enabling mentorship and strategic thinking instead. Universities must pivot: building AI familiarity into curricula, strengthening human skills (client relationships, commercial thinking), and creating alternative pathways - clinics, project-based learning, industry partnerships - if firms hire fewer entry-level positions.
High-risk legal matters will always require human validation and judgment, so wholesale replacement is unlikely; however, efficiencies in high-volume, low-risk work like document review and e-discovery may reduce demand for entry-level paralegals and administrative roles.
No - according to recent AFR reporting, law firms increased graduate hiring by 6% despite rising AI adoption, likely because efficiency gains are being redirected toward better mentorship, client relationships, and strategic work rather than reducing headcount.
Universities should embed AI familiarity into workflows, strengthen human skills like client communication and commercial thinking, and create alternative training pathways (clinics, project-based learning, industry partnerships) if firms shift away from using grunt work as a training mechanism.
Major law firms are unlikely to adopt a pure AI-first model due to the people-centric nature of legal work and client relationships, though new specialist firms or high-volume, low-risk segments (like legal services companies) might explore more AI-heavy operating models.
Understanding AI tools and critically evaluating their outputs will become baseline expectations, so graduates will need to differentiate through deeper legal expertise, innovative thinking about where AI can improve efficiency, and strong client-facing and commercial skills.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode offers some substantive points about AI in legal practice - particularly the distinction between pre-thinking (expert lawyers structuring AI use) and post-thinking (juniors generating then validating), the productivity paradox from empirical research, and the shift from knowledge-keeping to judgment-curation. However, much of the discussion is exploratory and speculative rather than packed with novel claims. Several segments repeat the same themes (e.g., 'AI won't fully replace lawyers' and 'we still need humans in the loop') without adding new dimension or evidence.
AI requires a mindset shift from being a keeper of knowledge to a curator of judgment
pre thinking is what kind of the more experienced lawyers...feeding it into AI...whereas if you're a junior lawyer...you are post thinking, which is you get AI to generate the output, then you're evaluating that
The framing of pre-thinking vs. post-thinking is useful but not novel - it's standard cognitive science applied to AI use. The broader claims ('lawyers won't be fully replaced,' 'AI is just the next layer like the Internet') are widely circulated takes. The episode lacks contrarian or first-principles thinking; it largely validates conventional wisdom that AI will augment rather than replace, with universities needing to shift focus to judgment and conceptual knowledge. No genuinely counterintuitive arguments emerge.
we're no longer talking about evolution, we're talking about revolution
AI is that next layer and it doesn't completely shake things up, but it creates new job opportunities
Alex (Speaker C) appears to be a lawyer or legal professional with firm experience and relevant perspective on training graduates and law firm operations. Mark (Speaker A) is from university leadership involved in legal education strategy. Both have operational relevance to the topic. However, their credentials, seniority level, and specific track records are never clearly established in the transcript. They come across as thoughtful practitioners but lack the gravitas of someone who has scaled a legal AI product, built major AI infrastructure, or run a major law firm through transformation.
I think it is really interesting to see what people are saying about the agentic AI
as someone who works for a university and uh, who's sort of in the senior leadership team for the faculty
The episode cites a few specific data points: Thomson Reuters 2025 report showing 28% of law firms using generative AI and 50% worried about job displacement; AFR article showing 6% increase in graduate hiring; Klarna's 700 layoffs; mention of Salesforce (30-50% of tasks by AI) and Duolingo replacing workers. However, most claims lack hard numbers or timeline specificity. The productivity paradox cited from an empirical study (19% reduction despite perceived 25% gain) is mentioned but the study is not clearly named. No concrete examples of specific law firms implementing AI, revenue impacts, or specific legal tasks quantified.
the survey shows that in the legal firms or law firms around 28% of people already using generative AI...around 50% of people...said that they are worried about that there is a less need or work for lawyers
Klarna, which is a uh, buy now, pay later type of company that in 2024 they sacked around 700 of their support agents
Speaker B (host) does pose genuine follow-up questions and tries to challenge assumptions - e.g., asking whether efficiency gains might just mean fewer paralegals rather than job elimination, or probing whether universities might struggle if firms hire fewer graduates. However, the host rarely pushes back hard on speculative claims or asks speakers to substantiate assertions with data. When Alex makes broad claims about university strategy or the timeline of AI capability, the host largely accepts them. The conversation meanders and repeats themes without building cumulative depth or testing the guests' logic rigorously.
But how about it would increase efficiency to the point that we don't need like 10 paralegals, we will need like five of them
would that mean they will get paid less or not paid at all? And that becomes a norm. How do you see that?
Computed from the transcript - who did the talking, and the words that came up most.
This episode of "Future Ready Lawyer: AI and the Evolution of Legal Practice" explores the rapidly shifting landscape of AI in legal education and practice. The conversation covers the impact of generative and agentic AI on legal job roles, the prospect of "AI-first" companies, the productivity paradox of AI tools, the changing nature of graduate legal work, and the enduring need for deep expertise and human judgment in law. We review recent empirical research, legal market hiring trends, sector skepticism about full automation, and the new skills required as legal workflows evolve. Show note sources: Thomson Reuters, 2025 Generative AI in Professional Services Report Luis von Ahn (Duolingo), Duolingo "AI-first" company coverage (2025) Financial Review (AFR), LinkedIn, Law firms take more graduates even as AI does the grunt work Princeton researchers (Knight First Amendment Institute) AI as Normal Technology Gary Marcus Generative AI's most prominent skeptic METR: Becker, J., Rush, N., Barnes, B., & Rein, D.
Transcribed and scored by The B2B Podcast Index.
Speaker A: We're no longer talking about evolution, we're talking about revolution. In the university sector, I'm sort of more looking at in the next few years, I do think it's going to be much more iterative, evolutionary, and things might stay more the same than they change. I think there's going to be a lot of, um, the hype will go away, the bubble will deflate. You've got that classic graph of the hype and then it goes away. And then we get a more stable understanding of what this thing can do and how we're going to use it in our work. So in a way, I think the universities have, uh, a little bit of Runway if we think that the tech is not going to just turn into AGI in 2027, like some commentators are saying. And if it does, that's a revolution and who knows what will happen. But for the next few years, I would say universities need to think about what is changing that idea of having some AI familiarity, being able to leverage what existing capabilities it has and maybe look for ways of building it more into the workflow.
Speaker B: How do you stay up to date with AI? Uh, iTunes. Hello, and welcome to this episode of Future Ready Lawyer AI and the Evolution of Legal Practice. For this episode we're going to talk about future of jobs in the legal practice and how we think it's going to change, how it's going to impact the industry and what's the prospect in terms of legal education and how universities would react to impact of AI. The current AI models as we know their capabilities have been, you know, they're just getting better and better. But there is a type of AI that is trending and that's called like agentic AI or AI agents. And the meaning of agent AI is that have some form of agency in terms of they can complete tasks independently with minimal supervision. For instance, you may ask an agentic AI to access a database and find a specific case for you and summarize it and they would go and do it. So as the agentic AIs are getting better, they will be able to do more and more different aspects of any type of jobs, including a legal job and, um, in firms and what lawyers do or paralegals do or even judges do. At some point, there's a possibility that they can do a large portion of a person's job, that we may not need that person anymore because combination of AI and humans, things are happening so efficiently that we don't need as many people to work, for instance, in that firm. And when it comes like to statistics and whether like that's actual trend. When we look at the um, Thomson Reuters 2025 generative AI in professional service report which is recently released, the survey shows that in the legal firms or law firms around 28% of people already using generative AI, which is honestly a bigger number than I thought. And at the same time around 50% of people who participated said that they are worried about that there is a less need or work for lawyers in the future. They found that as a somewhat of a threat or major threat. So this is something that a lot of people are considering as a problem that AI may impact their job and displace them. And it's not that much of a futuristic things we have seen in some industries that companies have sacked people. Like an example is Klarna, which is a uh, buy now, pay later type of company that in 2024 they sacked around 700 of their support agents or people who work in customer service and replace them with an AI. There were some hiring back but a large number were kind of sacked. And another company, Salesforce, which is a cloud based software company, they're serious that around uh, between 30 to 50% of all tasks are being done by AI and there have been layoffs. But they claim that it's not because of AI. It's very confusing because I think CEOs are worried about the backlash. And another concept that been uh, emerging this year, it's becoming a trend is AI first companies. That is the companies that they place artificial intelligence at the core of their strategy. There are many different ways to define it but uh, simply put they say that if an AI can do something, we are not hiring any person for that position. So we have to first test whether AI can do it. And the priority is people who can use AI and collaborate with AI. And there are, there would be more expectations terms of efficiency because people are forced to use AI. In July 2025 Washington Post released a piece about a few companies that have announced or showed some kind of indication that they are becoming a first company. One of them is Duolingo. We all know it, we have the app, we never use it that they're replacing people with AI again due to backlash. CEOs have been, you know, change their wording or announcement that yeah, we are not replacing, we are just doing things more efficiently or we are hiring even more people. So a uh, little bit of conflicting in terms of what they say and what, what is happening. And these are all because AI had been being able in practice doing More things. So this is not that much of a future. Now with that introduction, the question would be if agentic AIs are actually able to do more and more legal tasks, how would it impact legal profession and how would it impact legal education? So Alex, I think you can tell us a bit about how do you think in general as an introduction. How will AI impact the legal profession and with the focus on new graduates, what would happen to them?
Speaker C: Yeah, I think it is really interesting to see what people are saying about the agentic AI and obviously there's heaps of capability that exists within that kind of AI. But ultimately I think because of the high risk matters, um, that most lawyers are dealing with and the level of complexity, I just don't foresee that there's ever going to be a world in which we can entirely outsource, um, that. I think obviously there is the capacity to collaborate and work um, with the AI agent, but there's always going to need to be a human in the loop to validate those outputs and ensure the appropriate sort of commercial considerations or client considerations are overlaid on top of that as well. And I just don't think that the legal industry will ever feel comfortable um, outsourcing those components completely. And it also begs the question as to whether or not there will ever be that capability. You know, will the AI ever get to that point, um, of being able to sort of produce outcomes that have, you know, say 99%, um, accuracy as well? I just, I think that there will be always place for lawyers, um, in my opinion, uh, I don't think AI is going to take their jobs anytime soon.
Speaker B: That's true, at least for now, that it's difficult to predict that AI will become that good to replace people. But how about it would increase efficiency to the point that we don't need like 10 paralegals, we will need like five of them. Because with collaboration with AI they can achieve more than 10 paralegal.
Speaker C: Yeah, I think that's probably where the risk lies in terms of that more sort of high volume, lower risk, more administrative leaning work within law firms. And I think, you know, it is natural that as you create efficiencies with those kind of works that you won't need as many positions. It's been really interesting though to see some of the commentary around graduates and people trying to determine if that means that we'll have less graduates, um, coming into law firms. There was an article recently in the AFR that was titled Law Firm Take more Graduates. Even as AI does the grunt work so the crux of that article was, you know, out of the firms that were interviewed for that, uh, their recent partnership survey, 6%, um, there was a 6% increase in the number of graduates that these law firms took on. So in the majority of cases there was actually an increase in the number of graduates despite the fact that AI is becoming more and more prolific in terms of, in law firms, all sort of being used by lawyers on a daily basis. Um, and I think that that sort of hopefully gives a little bit of comfort to some of our law students and graduates coming out of law school. Because I saw one of the examples and this is one that's actually, you know, from our firm in this particular article was talking about how, you know, with high volume document review and some of those E discovery tasks, AI is really useful in that context. But you ultimately still need a junior lawyer to be the one carrying out that sort of review using the AI tech. Like that's not going to be sort of, that first level review is not necessarily going to be undert by a very senior lawyer. So it's speeding up some of the tasks that they would already do like sort of the review and the summary and all those pieces. But you still ultimately need those graduates to be able to validate those outputs and sort of, you know, be the ones prompting and going through those document sets. So I think the fact that there has been an increase across most of the firms, um, yeah, sort of bodes well for there always being the place.
Speaker B: There's a possibility that there are other reasons for more hiring. Same as like there have been layups like computer industry that are not relevant to AI because they hire too many people during the pandemic, then huge number of people they had to sack. So sometimes that is referred to as an AI impact, but many say that it has nothing with AI. They genuinely, you know, hurt too many people. But Mark, with the considering that if we assume AI with collaboration with humans means the efficiencies is at such a high rate, we don't need as many people to do the same amount of work, we need like half of them, what would that mean for universities? What would that mean for graduates when applying for jobs? Or when we are looking at this in like 3 to 5 years period, how should we change our policies?
Speaker A: Yeah, it's uh, something I'm always thinking about. Um, as someone who works for a university and uh, who's sort of in the senior leadership team for the faculty, I think it really does matter. Uh, what is the timeline? Do we just have A, uh, linear increase and how steep or do we get that exponential increase in capability? Because, you know, in some ways I think we're definitely at the valley. We're still at the very takeoff point of this new technology and the perfection of it. Even just the basic trusted to some degree integration of that within our workflows. Seems to me that, um, we're just starting off there. Uh, we've got the Harveys and the co counsel and these sort of different tools and some of them have been around for much longer using natural language techniques. And obviously we've had just basic search previously. We're layering on this thing that we only really got pretty good as a chatbot with generative AI a few years ago. And we are in the teething phase. And so I'm not surprised that you would have an even an increase in hiring at the moment. I think the real question is
Speaker C: how
Speaker A: long do, how much Runway do we have before things really change? And when that happens, you know, we're no longer talking about evolution, we're talking about revolution in the university sector. Uh, I'm sort of more looking at in the next few years. I do think it's going to be much more iterative, evolutionary, and things might stay more the same than they change. I think there's going to be a lot of, um, the hype will go away, the bubble will deflate. You've got that classic graph of the hype and then it goes away and then we get a more stable understanding of what this thing can do and how we're going to use it in our work. So in a way, I think the universities have, uh, a little bit of Runway if we think that the tech is not going to just turn into AGI in 2027 like some commentators are, ah, saying. And if it does, that's a revolution and who knows what will happen. But for the next few years, I would say universities need to think about what is changing that idea of having some AI familiarity, being able to leverage what existing capabilities it has and maybe look for ways of building it more into the workflow. As a junior lawyer liaising with the partner and kind of thinking through what could we be doing here and how can we make it work. Um, perhaps if they do have some more expertise in the AI methods, so like equipping the students for that. While I think also being really clear about the other things we've always been trying to help students do is develop deep expertise and also develop those human skills that are, uh, I would say in the past we perhaps haven't focused on them so much as a university and students have developed them informally or through competitions. But we, I think we can do a lot better in bringing that into the formal education. So assuming it's an evolution rather than a revolution, those are the things I think we can sort of slowly develop. The only other thing I'd say is when we are looking at those places where you say we don't have to have so many graduates because AI can pick up some stuff that takes away that rung off the bottom of the ladder for more students, more graduates, um, some will still get onto that rung but others will miss that and we might have to think more about how do we create that rung through industry partnerships, through clinics, through ourselves as a university, giving those opportunities to do more of the actual project based, possibly even client based, um, learning on the job. I think we might have to step in there if in straight and economic times the firms are like, take fewer graduates and um, you know, people are, people are running out of options for developing the skills that firms want them to have.
Speaker B: Yeah. Anna, um, as you were talking Mark, and combining with what Alex said, the thing that comes to my mind that makes sense why we may have increasing graduates is that we always have had problem with access to justice and if AI is enabling us to do the same thing faster and cheaper, uh, then it would make sense to have more graduates to provide the same service at ah, a lower rate. And that means we need more people to come and do it. So I agree with you Alex, that could be the case.
Speaker C: I think as well it's kind of looking at the composition of the graduate workloads as well. So maybe if they're doing less of this sort of really admin heavy high volume document review, there's more time for them to attend meetings with clients or sort of have that mentorship from partners. Like I think there's also an acknowledgement that yes, we may be finding efficiencies and that may result in fewer roles being available. Maybe it's sort of, you know, law firms making the conscious decision that, you know, instead of this efficiency bringing an opportunity to reduce roles, it's actually increasing the opportunity to provide better training and sort of ensure that that commercial skill set, that more sort of client focused relationship skill set is something that's at the fore rather than relying on sort of some of the grunt work to do that training. So it's being a bit more strategic about understanding what are the best ways that we can equip our graduates Going forward, if they're not going to be able to just hand them sort of some of these really time consuming tasks that AI now enables them to do much faster.
Speaker A: So I find that really interesting because presumably there wasn't like a conscious decision we must have the graduates do all that grunt work. It was just there wasn't another way to do it really, because you needed to have a bit of legal understanding to do the discovery or whatever it was. So it does seem to, uh, be a real opportunity to rethink what you're going to, how you're going to train new lawyers, how you could deploy their skills and their fresh ideas perhaps within the firm when you don't have to have them doing the grunt work that no one else wants to do. So it feels like there's a bit of an opportunity to say, you know, for New Path aid, especially if, and I'm not sure how, you know, you're seeing things, Alex, but maybe there's a fault of the universities or to some degree the students themselves, if they're not bringing real good new ideas and capabilities with the use of AI, um, vis a vis, you know, the people who are within the firms at the moment. But it seems like a real opportunity for them to be like, look, look at me. I can, I can figure out where we can go with this and how we can build that evolutionary future of our firm. Um, instead of being in the basement in the grunt work.
Speaker C: Yeah, exactly. Months on end, more opportunities for them to kind of work closely with some of the more experienced lawyers and then, yeah, even if they may not have had exposure to as many matters or clients or things that the more senior lawyers have had over the years, they can really bring that innovative thinking. And if they have a proper understanding of the tools as well, it really does help you better identify where you can then, uh, apply AI or deploy AI to find some of those efficiencies. So I think it is a really good opportunity for them to stand out. I think it's also important to acknowledge that if everyone is sort of building those AI skills simultaneously, then it becomes more of an expectation versus something that will sort of make you stand out from the park. So then I think it would be important for students beyond that to then think, okay, what else is differentiating me? What, like, what else can I bring to this role? Because I think there's sort of that understanding of the technology. There's also, given the fact that we can use AI in the technology now, maybe there's less emphasis on holding all of the specific case law or legislative references and things in your head because you have access to sort of, you know, these research tools that leverage generative AI and things like that. Maybe that looks less impressive. It's like, what else can I focus on in order to build out my repertoire and be considered someone that's a bit of a standout amongst this graduate cohort to ensure that I, you know, that there is a role for me and that I can keep progressing.
Speaker B: Alex, do you think that the fact that the students would not do those basic tasks anymore? Because my assumption has been that students, usually new graduates who are employed, they do some of those admin or basic tasks and as they progress, they learn new things and they involve them somehow, you know, in the meetings and other things. But if AI is going to do those basic work, then new graduates would likely be more like a trainee. And early on they're just going to learn, you know, a lot of things so they're not contributing as much. And would that mean they will get paid less or not paid at all? And that becomes a norm. How do you see that?
Speaker C: I think with kind of it's, it's firms, um, being conscious about the fact that that traditional method of training is probably not going to work anymore. So it's making really sort of targeted decisions to be like, okay, what would we normally get the grads to do? This, you know, high volume document review, this discovery task. That's how they would learn what they should be looking for. Then that would get checked, you know, by a more senior lawyer to understand what they might have missed, what are the other nuances that they need to consider. So if they're not doing that task, it's how, you know, how are we supplementing those same skills that they would be building in that context and help them build them in a different way. And so I think it will still be, you know, there will still be a huge emphasis on being able to equip the more junior lawyers or the graduates with the right skills in order to move up through the ranks. Because eventually, like the senior lawyers will move on, the workforce will kind of, you know, there'll be cohort leaving. They need to have the people that have the right expertise and then do can, you know, critically evaluate an AI output and understand the legal nuances. And so it's like if we're not doing it in the way that we were before, and some of that may have just been, you know, we fall back on that because it's like, oh, the tasks need to get done this is going to train you up as well. It's how everyone cuts their teeth. It's what needs to happen. And that's just sort of been tradition for years and years. Whereas I think now firms have to be more strategic in how they are planning these training programs, understanding what's missing if they're not doing these tasks and then how they can supplement that and sort of better equip them to sort of, you know, use AI, evaluate the outputs, but then still learn those same skills that they were learning from doing the task in the first place.
Speaker B: That certainly makes sense. I really hope it goes that way.
Speaker C: Me too.
Speaker B: Because there's a lot of concern. I'd mentioned the AI first companies like the Olingo and a few other famous companies like Shopify mentioned they want to be or they are now AI first company. What would that mean for legal profession, like in legal firms, law firms? Can we imagine an AI first law firm?
Speaker C: I think we could in some aspects or for certain types of workflows, like I think there will be kind of that thinking around, you know, some of that high volume, low risk work. Is there a way that we can start to sort of automate that first or sort of build out an automated workflow over hiring the whole team to do that kind of work? I don't know if there would necessarily be that AI first thinking for all of the positions within a law firm, but I think as sort of firms become more comfortable with AI and understand what it can do, I, uh, do think that they'll be thinking about the way that they deliver services and the way that sort of, you know, some workflows are better suited to AI. So you would kind of maybe explore with specialist teams within firms or with external partners or something that sort of, you know, can we. We are seeing an influx in this work. Is there a way to better enable us to do that work? You know, with the number of people we have now, if there's been an increase in volume or, you know, with, with less people. And I think, I don't think that it would ever be branded as an AI first approach though. I think it would be very much just thinking about where AI can augment some of those workflows. But I think it will still ultimately be a people first business. I don't know, maybe I'm optimistic, but I just, I think because of the nature of, uh, you know, when people come to lawyers, they're often, you know, either in a real pickle or, you know, it's like a stressful time, it's an incredibly complex situation. There are so many nuances and sort of people are at the core of those problems. I don't think that there's sort of going to be a world in which, you know, the major law firms are ever kind of having that massive shift towards being, you know, primarily AI or agents instead of people. I think that there is room for sort of law firms, new firms maybe coming out of the woodwork to sort of build out their workforce based on AI and have less lawyers if they're sort of looking at a different operating model. Um, so I could probably see that happening, but in terms of the major law firms, I don't now kind of see that being um, you know, or them employing a bit of an AI service mentality. But I think there will be that mentality to be like, is there a way to kind of, you know, do this a bit sort of better and faster to better use the people we have or to reduce maybe the number of people that we do have to hire? If we were hiring 10, maybe we only hire five because we've got this AI enablement. So I, I don't think that it will be um, sort of to the same extent as some of those tech companies.
Speaker A: There's this big paper AI is a normal technology by, I think it's Princeton researchers and they're sort of saying like, yeah, it can do all these things, but we don't think it's going to be AGI soon and it's just going to be built into our ah, technology stack of a, uh, firm and we'll leverage those efficiencies. So it's kind of interesting to try and think about like maybe we'll just get to a new place and the kind of skills we will want will be slightly tweaked. Like what makes the best loyal be slightly tweaked? Compared to the book age versus the Internet only age. Now we've got the generative AI age and that gives us a little bit more efficiency. It changes the skill set a little bit more. We've got to integrate that into our workflows, but it'll be pretty much the same. I mean that's sort of how more and more, uh, um, I'm reading a bit of Gary Marcus, who's this great skeptic of the connectionist approach. He thinks we need to integrate the symbolic or the knowledge. You know, don't just go with prediction and pattern matching and all that kind of great stuff that generative AI has been based on and scaled to a very Impressive level. The next thing we might want to need to do before we get to the AGI or a proper agent lawyer, is to build in the symbolic knowledge, you know, the actual understanding, which might be very, very hard and very far away. And so in the meantime, I feel like maybe we're in that the new age of. It's like how the Internet came along and changed up, how we do pretty much everything. Client interfacing, sharing documents, et cetera, storing knowledge, retrieving knowledge. AI is that next layer and it doesn't completely shake things up, but it creates new job opportunities and it changes the skill set of the lawyer. And just one thing before, uh, in thought, um, I've been reading a few papers and they talk about, you know, how much does, how much does someone's job to stay the same because they're just using what has been created as quite a good product, either by a vendor or within the firm, by clever AI integrating people. And, uh, they just kind of have the same. The Rainmaker and the schmoozing with the clients. Wherever these schmooze on the golf course classically or over boozy lunches or I don't know what, I don't know anything about that great lifestyle, but they're doing their thing and they don't even notice it because someone else is just providing them with the analysis. Whether it's AI or a Clark, they, uh, don't care. But then there's a whole lot of stuff where it does change. And there are new legal roles and certain lawyers have to be developed new skill sets. I think it's sort of interesting. Maybe things after the bubble, maybe things will go back a bit more to normal. I don't know. How does that strike?
Speaker C: I think it's really interesting to think about it and I agree with what you're saying in terms of the augmentation. And I think we'll see more and more sort of these hybrid teams. And so it's kind of like, you know, you'll have your lawyers, but then you'll have your people that are quite specialized in legal workflows or sort of the code behind the legal workflow. So there still needs to be that understanding of the legal process and how you would achieve something. But maybe your role is more focused on what does that look like in terms of building out a logic flow to then inform, you know, automation or AI that's being used for that solution. You work closely with your legal, you know, your counterparts who are, uh, looking at the outputs and validating that they are correct against relevant laws like that, they are sort of incorporating relevant commercial considerations. Like I think that there will be a lot of that as well. So it's not all on lawyers to have to do sort of some of that technical back end kind of, you know, work, but it's working really closely together so that everyone's across kind of the sort of basic flow or function and how these different tools work. So then you can be thinking about the solutions, but then also understand what you need to look for in terms of the outputs and what might be missing based on your understanding of their limitations.
Speaker B: Yeah, on that note, uh, there was a kind of recent empirical study by this group, what was it? Uh, model evaluation and threat research. Uh, and they assessed like in terms of productivity of generative AI models because there was, I think a Stanford study that was saying that, oh, it would actually increase the productivity. But in this study, encoding specifically, they asked people like, how much do you think AI, uh, uh, would help you to increase your productivity? And on average people said like 25%. A small group of people, I think 16 people only. And then they use the AI coding, you know, help or tool. And once the task finished, they asked them again, how much do you think it helped you in terms of productivity? And they said, we think around 20%. And they recorded everybody's screen to see exactly what they do. And what they realized was that on average the productivity was reduced by 19%. And yet they were thinking that it is increasing their productivity. And the reason I was saying it goes back to what you were saying Alex, about you need to verify the output and for them when it comes to coding. And I think same for the legal work, when you have to verify, it takes them a long time to do that. And that really, that doesn't help with productivity. It may take more time. And in my own recent study, a lot of my students, in their reflection about the use of generative AI in their assignment, which I forced them to write one, they said their verification process was really time consuming to the point that they thought if they use traditional way of research, use the university digital libraries, they would do it much faster than using generative AI to write a report for them and they have to improve it and then verify all the content. They said that took ages and they think normally they would do better.
Speaker C: I think it's really interesting that observation around, you know, productivity actually being reduced initially with using AI, even if the perception is that we're going to be more productive. I think that's so true of uh, you know, particularly in the beginning stages of learning to use AI. And also if you don't have the right structures around it to be able to trust sort of the output, like if you're not using maybe sort of structures around, if you had a retrieval augmented generation tool. So it's like you have the AI appointed at a set of documents, they have been pre vetted, you know that all of the legislation and caseload is up to date. That's ultimately going to reduce your time that you would take to verify the output versus kind of starting from scratch. Or maybe your prompting isn't as specific, you haven't built in other levels of validation that you could, you know, use with AI as well to then reduce ultimately the time at the end to verify. But you will always need to verify. And so that will always be an impact on, you know, the productivity. And so if you're not getting enough efficiency from the way that you're prompting and the resources that you're pointing it at, then it does sort of beg the question like, should I have just drafted this myself? I know what's gone into it. I can validate that all of the sources are correct. But I think that that really does speak to the point around, you know, that judgment piece is becoming so critical and the critical thinking skills as well, where it's like you're having, you are having to evaluate what's coming out of this AI and making sure that whatever you're sort of putting forward or using these outputs you can validate. And I think it was really interesting. I went to a conference a couple of weeks ago, we were in an AI conference and one of the panelists said something that really resonated, which was AI requires a mindset shift from being a keeper of knowledge to a curator of judgment. And so it's going from, you know, knowing all of the ins and outs of this particular legislation or this case law, because you know, if you've got the right AI tool pointed at the right knowledge database, then that can draw on that for you. It's almost like the knowledge is kind of outsourced to that point, but then you're kind of needing to bring in your critical thinking skills when you're looking at the output. And don't get me wrong, there's obviously still a need to understand the nuances and sort of how some of those concepts are connected. You can never fully reduce the amount of knowledge you need to know. But maybe there's not as much emphasis now in being able to recall that specific section that might apply because the first pass that AI has done of compiling an advice or sort of drafting a contract has referenced those relevant points for you already. So then it's just sort of using that critical eye and that judgment to ensure that the output is, you know, contains what it needs to and it's not hallucinated or put in any sources that may not be relevant.
Speaker A: Yeah, I mean, that's a really interesting way of framing it around the judgment. I guess picking up on your acknowledgement about we need some knowledge. I think that's where what we're describing here is. Let's point the, the, the AI at some documents, some, some case law, some statutory materials, some precedents or, you know, the facts of a case in an appropriately compliant way, however that works. And let's get the first pass of a set of arguments. Let's use the AI to draft some stuff up and to try and get our theory of the case out and our arguments out. And I think what you need there, it seems to me, is you do need that a deep conceptual knowledge, a conceptual understanding of, I don't know, company law or property law or whatever area of law it is, because that's what allows you to get that efficiency. So a student can do that who knows nothing about it, Bang, they've got it, but they can't improve it. It's going to take them so long, as we said, to verify that. Whereas if it's an expert lawyer, they can again do all that thing. And then as they're reading through, they can just say, yep, yep, yep, yep. No, that's not how I'd put it. They can follow up on any more difficult points. Anything they're not, they don't really understand. They can verify probably more quickly. And I think the whole idea I would take from that is we still need that time, whether it's at university or there's this other AI powered educational institution or whatever it is, where you are really getting a hold of those conceptual structures in place. Uh, that means perhaps you could go and practice another jurisdiction without having all the detailed knowledge of the AIR law and apply the concepts to use AI to be more efficient in arguing a case in Singapore or London as someone trained in Australia or New Zealand. But I think, you know, I've seen people use it in this way. I, uh, had a friend who's an expert in the area. He was, had to do a presentation, he pointed it at his onenote and a few technical documents and said, help me explain this to a layperson audience. And he hadn't done this before, but he said it was really good. And you know, he knows and he knows how it's not quite so good. And he doesn't even have to verify so much. Cause he's got that knowledge. So I think when we're talking about the efficiencies and when it's going to be useful, you know, I would still, I'll always plug as an academic and as a someone whose, um, bills get paid by people coming to university. University is a place where you can get that, you know, the deep conceptual knowledge which I think is going to be necessary, um, if you're going to get the most efficiency out of these tools.
Speaker C: I completely agree. And at that same conference, that Women in AI conference on the same panel, um, there was sort of reference to the two different types of thinking. And so pre thinking is what kind of the more experienced lawyers, people that have been in the industry for a while, you're thinking about the structure and the concepts and all the elements and then feeding it into AI to almost get some of that verification or looking for any additional angles maybe that you've missed or sort of other pieces that you haven't lived in. Whereas if you're a junior lawyer or a graduate and you don't have that knowledge yet, you are, uh, post thinking, which is you get AI to generate the output, then you're evaluating that like there's not as much of that sort of structuring or pre thinking that is going into the output and therefore requiring way more verification. And as you say, Mark, by people that don't have, may not have the knowledge yet to be able to quickly go, yes, no, yes, or that sounds wrong, or you know, yes, that technically right, but it's not really valid here or it's making this contract a little bit longer. We don't need that clause that's not commercially relevant or, you know, whatever it might be. I think it's just really interesting to see people that uh, are coming through as kind of AI native, like as sort of, you know, we see the next cohorts come through and then the people that sort of have had AI applied or used it to augment the way that they work based on their level of expertise already. And I completely agree that sort of applying it to, you know, someone's workflow that already has that expertise is where the efficiency comes in. I don't know that there is that much efficiency if you don't have that level of knowledge to begin with.
Speaker A: Good to hear.
Speaker B: Thank you everyone for listening to this. Episode of the podcast. We hope the podcast, uh, gave you some ideas to bring these kind of discussions further in both legal practice and education sector in universities so we can all together navigate through this technology and get the best possible outcome. How do you stay up to date with AI news? Uh, iTunes?
Other episodes covering the same guests and topics, from across The B2B Podcast Index.