Future Ready Lawyer · 2025-11-03 · 36 min
Key moments - from our scoring
Substance score
48 / 100
Five dimensions, 20 points each
This episode examines the real state of AI adoption across legal firms, distinguishing between marketed expectations and on-the-ground reality. Speakers Mark, Alex, and Armin discuss survey data showing 70-80% of firms use AI for common tasks like legal research, document review, contract analysis, and memo drafting, yet far fewer have embedded sophisticated tools effectively. The conversation highlights a critical divide: large firms are piloting specialized legal AI platforms like Harvey and Co-Counsel alongside general tools like ChatGPT and Microsoft Copilot, while in-house legal teams face budget constraints and fewer resources. A major concern emerges around shadow AI use - lawyers deploying generic chatbots without firm governance, training, or guardrails, risking hallucinated case citations and confidential data exposure. The speakers stress that simply procuring technology doesn't guarantee adoption; without proper change management, training programs, and AI governance frameworks, disillusionment sets in quickly. The discussion covers the build-versus-buy decision, the cost-benefit analysis of enterprise tools versus custom solutions built on accessible platforms, and the ongoing challenge of helping legal teams develop AI literacy to safely evaluate outputs and understand compliance risks.
Survey data shows roughly 70-80% of firms are using generative AI, particularly for common tasks like legal research, document review, contract analysis, and memo drafting, with adoption trending toward high majority use.
Specialized legal AI platforms are built specifically for legal workflows with legal-trained models and governance built in, while generic tools like ChatGPT are more accessible and cheaper but lack legal-specific safeguards and may carry higher risks of hallucinations when used without firm oversight.
Without proper change management, training programs, governance frameworks, and ongoing support, lawyers become disillusioned if tools don't deliver expected outputs, leading to abandonment; the gap between procurement and actual daily usage is significant.
Lawyers risk submitting hallucinated case citations to courts, uploading confidential client data to unsecured platforms, and violating firm governance without proper AI literacy or understanding of compliance obligations.
The decision depends on firm size and resources: large firms justify specialized platforms for structural change and scalability, while smaller teams may gain similar value building custom workflows with accessible tools like Copilot, provided someone has the technical knowledge to do so.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode delivers moderate insight density with some novel observations about the gap between adoption announcements and actual day-to-day usage, and useful categorization of four AI tool types. However, substantial portions consist of repetitive discussion about generic vs. specialized tools, change management platitudes ('proper guardrails,' 'appropriate training'), and circular conversations without concrete resolution. The insights about lawyers using AI as 'Google' rather than as a research assistant, and the warning about misuse in court cases, are valuable but not sufficiently novel or dense to elevate beyond mid-range.
firms might be procuring this technology but are people using them um, on a day to day basis
There is a bit of a danger if they're not embedded correctly or there's not sort of the proper change management around it that people get excited at the start and they start to use it, but maybe it's not giving them the outputs that they had wanted or hoped
While the episode makes some valid contrarian points - notably that research/drafting may represent only a small portion of actual legal work, and that lawyers are defaulting to generic tools despite availability of specialized alternatives - these observations are not particularly novel or first-principles. The discussion largely recycles familiar frameworks (build vs. buy, governance and guardrails, adoption curves) without pushing beyond established tech adoption thinking. The research paper by Webb and Patterson appears to be the closest thing to original insight, but it is cited secondhand rather than deeply interrogated.
some of the things that generative AI is seen as sort of revolutionary for perhaps, and maybe not there yet, but in the future, things like legal research, figuring out what your arguments are, uh, drafting your briefs, just getting your theory of the case and getting your arguments out, that kind of thing might not take up the bulk of what a lawyer does every day
lawyers in practice are using it as many of them mentioned in cases that they use it as some form of Google
The episode features two academics (Mark Webb appears to be from University of Melbourne law school; Armin does not have clear credentials provided) and Alex, who claims exposure to law firm adoption trends but provides no concrete title, firm, or credentials. While the speakers have relevant expertise in legal AI, their authority is largely asserted through anecdotal observation ('from what I've been seeing') and secondhand survey data rather than demonstrated through executive track records, specific implementations they have led, or quantified impact. Mark explicitly states he is 'a career academic' who doesn't practice law, which limits practical authority on actual firm adoption challenges.
I'm a career academic, I don't practise law
from what I've been seeing, most of the law firms that I'm aware of have done that
The episode provides limited concrete evidence. While it references 350+ cases of AI misuse in courts (unverified, undated), mentions specific tool names (Harvey, LexisNexis, Copilot, Co-Counsel), and cites a Webb/Patterson paper, there are no specific metrics, timelines, dollar figures, or named companies adopting tools at scale. Claims about adoption percentages (70 - 80% of firms, 80% majority use) are attributed vaguely to 'surveys' without sources. No firm names, case examples, or measurable outcomes are provided. The discussion of cost remains abstract ('very expensive') without pricing data.
you know, there's all different surveys, different numbers, but it's really going up over that majority and getting into the sort of M high majority, sort of 80% of people and firms seem to be using this
out of these more than 350 cases of misuse of generative AI, I cannot remember a single case a lawyer says
The host (Speaker C) asks reasonable setup questions and occasionally pushes back on claims, but lacks systematic follow-up rigor. When Mark claims AI can make practice vastly more efficient, there are no hard questions about why adoption remains slow or why court misuse persists if the tool is so powerful. The host accepts anecdotes (the colleague's 'most intellectually satisfying week') without probing for metrics or scaling challenges. Speaker C does usefully challenge the efficiency narrative late in the episode, but this represents a single moment of productive pushback in a largely exploratory, rather than investigative, conversation. The opening meta-moment ('How do you stay up to date with AI news? I don't') is charming but not substantive.
Do you think that if they're not doing it or if they are planning to do, to uh, do it, do they, do they have the time to manage learning the whole new platforms and keeping up with the technology and updates?
But at the same time, like when it comes to courts and cases, again, out of these more than 350 cases of misuse of generative AI, I cannot remember a single case a lawyer says that, well, I used it to tell me what are the problems
Computed from the transcript - who did the talking, and the words that came up most.
In the second episode of 'Future Ready Lawyer: AI and the Evolution of Legal Practice,' the discussion revolves around the adoption and challenges of generative AI in legal practice. The episode touches on the current state of AI use in legal firms and specific applications of generative AI. Hosts discuss the varied adoption rates, the effectiveness of AI tools, and the impact of AI on legal efficiency. They point out issues related to cost and training that hinder widespread adoption. Show note sources: Damien Charlotin ‘AI Hallucination Cases’ Thomson Reuters ‘2025 Generative AI in Professional Services Report’ Dye & Durham ‘Lawyers face work-life paradox as AI adoption soars across the profession’ Clayton Utz ‘Clayton Utz adopts Harvey in next step of AI journey’ Webb, Julian and Paterson, Jeannie Marie, THE EVOLUTION OF LEGAL KNOWLEDGE WORK IN AN AGE OF BRILLIANT(?) TECHNOLOGIES: FROM ROBO-LAWYER TO DIGITAL LAW CLERK (April 02, 2025). Available at SSRN: or Open AI ‘How we’re responding to The New York Times’ data demands in order to protect user privacy’
Transcribed and scored by The B2B Podcast Index.
Speaker A: So, you know, firms might be procuring this technology, but are people using them, um, on a day to day basis? And what does that actually look like in terms of engagement and revisiting those tools? I think that there is a bit of a danger if they're not embedded correctly or there's not sort of the proper change management around it that people get excited at the start and they start to use it, but maybe it's not giving them the outputs that they had wanted or hoped. And so there begins to be a bit of disillusionment as well, um, as people begin to experiment.
Speaker B: How do you stay up to date with AI news?
Speaker C: I don't. All right, so welcome to the second episode of feature Ready Lawyer AI and the evolution of legal practice. In the last episode we talked about a few different topics briefly and we said in the future episodes, including today, we're going to, um, discuss those topics at a, uh, much deeper level and go into details. So today we want to talk about how generative AI is being used in practice. And to get there we will first talk about our assumptions and expectations of how, for instance, legal firms and lawyers have been adopting this technology, a little bit of history of technology in legal firms, and then move on to what actually is happening in legal firms and legal practice in courts and what are different uses of generative AI. So, Mark, how about you start first and tell us a little bit about what we have been seeing, you know, online and what are the expectations?
Speaker B: Okay. Um, yeah, so I'm really fascinated with what's actually going on in legal practice because I'm a career academic, I don't practise law, but I see people talking about this on LinkedIn. I look at some of the research coming out in terms of surveys, Thomson Reuters and other of the providers. How are lawyers actually using generative AI to get tasks done, to get efficiencies, to even improve the work product. So, so I've been having a look at some of the more recent surveys that have come out and it's really interesting to see that there seems to be, uh, a shift. Just like I see within academia, the law school and with students, you have a real trickle early on, before ChatGPT, you know, I was using OpenAI even before ChatGPT, but very few people were. ChatGPT turns up and you get a stream and that just keeps on getting more and more volume. And it looks now that we've gone over that threshold of majority use, most people are trying to use this thing. Most people think that it's just going to get bigger and bigger and more central to legal workflows. And so the statistics, you know, there's all different surveys, different numbers, but it's really going up over that majority and getting into the sort of M high majority, sort of 80% of people and firms seem to be using this. Now the other thing that I'm seeing is these big announcements. Australia, New Zealand, uk, those are the places that I'm seeing most of these announcements coming from. And it's about we're using Copilot and we're partnering with Microsoft and we're, we're doing stuff, we're getting on board with this, we've got our policies, we're putting our processes in place, we're experimenting. And then you've got other people using those products of the big legal publishers, Alexis Nexus, Thomson Reuters, even this new player Harvey, and saying look, we're investing in this, this is part of our uh, business going forward and we can see real gains in terms of time efficiencies and things that we can deliver for our client in a, in a better way. So you know, that's my outsider's view and I, you know, the thing that I see it being used for in these surveys are things like legal research. Obviously those legal research providers are sticking AI into their products and seeing what comes out of that document review and analysis, summarization of documents, stuff around contracts, drafting, memos, correspondence with clients. And then all that's what's happening now. Sort of 70, 80% of firms seem to be using it for those kinds of tasks. Maybe they're the low hanging fruit. And then there's more sophisticated uh, uses that I think probably aren't quite there yet in many cases. But things like document drafting and automation, um, case diagnosis, prediction in terms of what's happening in litigation and dispute resolution, things like due diligence for mergers and acquisitions or big real estate transactions, um, compliance and risk. So those are the things that aren't um, quite up there in the sort of mostly what this is being used for, but they're coming on stream and probably being developed at the moment.
Speaker C: Also AI avatars, I'm referring to the lawyer who used an avatar to present their case. And so Alex, going to provide us with some further information about how things are happening in law firms and what's the perspective and what's the direction because we also hear from some law firms that they're not, you know, adopting the technology or they've decided to delay things. And at the same time we hear A lot about that. They're trained to use it, you know, somehow to increase efficiency.
Speaker A: Yeah, very happy to. Um, and so largely what Mark has said around sort of the increase in adoption is definitely something that I've been seeing um, over the past year. Um, and it is really interesting to see though that even though there has been an increase in people using AI um and trying to embed it in their workflow, there are still a lot of people that are quite early on in their journey. I think the enthusiasm is definitely rising and sort of more people are wanting to experiment and get involved but there is still a bit of a gap between modes that have really run full force. Um, at AI had got out of the gate early in terms of their training and capability building and sort of developing out of their use cases versus others that might sort of, you know, still be at the, the starting blocks a little and sort of experimenting in their day to day tasks and things like that. But there's not necessarily that movement into considering the legal workflows yet. So I think it is interesting to watch uh, some of the players in the market sort of as you mentioned Mark, the Harveys, sort of co counsel, um, some of those really legal specific AI platforms, um, and seeing their uptake. There has been a lot from what I understand of sort of pilots and testing of those technologies within larger law firm. Um, and it is really interesting to start comparing the outputs and the capabilities of those tools with some of the more general AI tools that people might have been tapping into like a copilot or chatgpt court etc. Uh, so I think that that's really interesting to watch how people are starting to see the difference between the two and thinking about what a legal specific workflow does look like. But yeah, I will say that I uh, think few people are still, still a little bit early on in their journey compared to what some of the media says about the uptake in adoption. And I think there is also a difference between um, the tools that people are given within their firms and their team and then actually using those tools. So you know, firms might be procuring this technology but are people using them um, on a day to day basis and what does that actually look like in terms of engagement and revisiting those tools? I think that there is a bit of a danger if they're not embedded correctly or there's not sort of the proper change management around it that people get excited at the start and they start to use it, but maybe it's not giving them the outputs that they had wanted or hoped. And so there begins to be a bit of disillusionment as well, um, as people begin to experiment. So I think those programs that go around the tools, um, and sort of bringing them into different organizations and teams are really important to ensure that there is continued uptake and adoption of those exciting new tech tools that are coming in.
Speaker C: So you referred to Harvey also Mark, the and M you mentioned like co counsel that mean we have technologies that are specifically like generative AI technology that is specifically made for legal practice. And then we have general generative AI like Gemini and ChatGPT. So when it comes to adoption in practice like for law firms, then we would expect the law firms to adopt these tools that are made for legal contexts rather than trying to use generative AI, uh, models such as ChatGPT and Gemini and Quad that are generic, you know, models. Is it correct?
Speaker A: Yeah, that's right. So large law firms are leaning, from what I seen, towards having some of these legal specific tools. That being said, it doesn't take away the need for Copilot or something similar. You know, if you're operating on a Microsoft 365 environment, that's still something that gets used. And again you're sort of using that in the context of, you know, drafting emails and you know, meeting transcripts and all of those things. So I think there is still a place for that sort of more general AI tool. However. Yeah, from what I'm saying, most of the law firms are uh, looking at, you know, these legal specific models and seeing what they could do because you know, there are very specific workflows that they can be applied to that. And again it kind of has the additional layer of trust because it's been built with lawyers in mind. So I think that that's sort of where I'm seeing a lot of firms lean. Like there's been a few firms that have announced partnerships with sort of these legal specific, um, generative AI tools. And it's also interesting to see some existing legal tech. So some of the matter management systems, contract lifecycle management systems, et cetera that people have, um, within this is mostly sort of within in house legal teams. Some of those systems are then adding AI functionality and AI components. So it's been interesting as well to understand how that is being used within in house legal teams because you know, in some cases the functionality is really great and then in other cases you can kind of see that it's something that's just been tacked on at the end because everyone's talking About AI wanting to use AI. Uh, so it would be really important as you know, an in house lawyer or somebody looking to engage with this tech to really interrogate what the capability is and how it sort of works in the background and how that will actually work in practice. I would really advise, you know, teams or sort of recommend that teams look at, you know, really specific workflow and also look to do a proof of concept or some piloting with these different tools to really understand the output that it is going to give you, um, based on some of your own kind of, you know, methodology or you know, a contract or something like that to just really see what its capability um, is because I think in a lot of cases some, some of that capability can be overstated.
Speaker B: So what I'm really interested in around these specialized legal tools is that you know, you see a lot of talk about in these surveys, lawyers are using the AI and then it's a real big deal. At least in New Zealand when a big law firm signs up and says, yeah, we're using whichever these big providers specialize to, it doesn't seem like it's ubiquitous. It doesn't seem like everyone's doing it. I uh, understand that they're very expensive. I've also heard sort of rumors and whispers that maybe they don't work as well as perhaps they should or we might imagine them to do. I have seen a lot more happening in, overseas in Australia, England, us. What do you reckon the state of play is? It's like every law firm basically going to be subscribing to one or the other of these specialised tools like now or in a year or so. What do you think?
Speaker C: I just want to add something to that question also. Do you think that if they're not doing it or if they are planning to do, to uh, do it, do they, do they have the time to manage learning the whole new platforms and keeping up with the technology and updates?
Speaker A: Yeah, I think it's a really good question. I will be interested to see how it plays out because I think at the moment we're still in the phase of some extensive piloting. I know there's a few firms that have sort of piloting two tools at the same time to really understand the comparison. Um, and also, you know, to your point, there is a massive investment required like uh, you know, these tools are not cheap and so I think it's really important for firms to validate that there is going to be ROI on whatever tool is chosen. So I feel like, you know, while a lot of firms are really interested in testing these larger scale or sort of enterprise level tools. You know, whether or not the investment is made in one of those tools I think remains to be seen. Like you were saying, Mark, there have been a few announcements of that already, but it will be interesting to see if sort of all firms follow suit. You know, there's always the discussion around sort of buy versus build as well, which is, you know, maybe some firms have the ability to build their own, um, you know, AI tools in house. And so I think that that kind of plays into it as well. I think firms are very much in the scanning mode at the moment and it kind of, you know, it's looking at those bigger players, but it's also looking at some of the newer uh, sort of, sort of startup players and trialing them with specific groups and sort of workflows to really get a good picture of what is going to be worth the investment. Because I think it is a, you know, it's a big decision. And to your point, Armin as well, kind of, you know, if you're going to make the call to have that tech and to have everyone expected to use it, how are you enabling that to happen? Like it will be so important to have you know, actual supportive structures around it to a. Yeah, give people the time, ensure that they really understand the tools, make sure all the checks and balances are in place, um, in terms of, you know, any, any governance frameworks and safety structures around that tool. Um, make sure that clients are comfortable with the use of that tool as well for their specific work. Um, and it'll just, there'll be a lot of things that kind of need to be in place before a firm I think is willing to make that, that big investment in sort of a bigger player. Um, and I think it's interesting to contrast that with in house legal teams who are often dealing with much smaller budget. It's kind of looking at, well, what makes sense from their perspective, how do we again ensure that they have the time because they're also incredibly time poor. Um, we are often dealing with a smaller team. We may not necessarily want something that's kind of at, you know, the enterprise level because again enterprise level also brings with it more cost. And so it's sort of, I think they're probably looking at maybe some of the smaller, um, you know, players out there or the sort of players that have integrated AI into their existing tech solution. But you know, as I mentioned before, it'll, it'll be sort of really important to interrogate what level of AI capability that is and what that means. Because I think as well there's a difference between saying, oh, this tool has AI when it's sort of, you know, automation or the type of AI we've seen for years versus generative AI, or the ability to sort of redline contracts and all of those as well. But yeah, I'm interested to see how it all plays out. And I think as well, you know, it is a bit of an overwhelming tech landscape. I think it always has been in legal tech. There's always been many, many players. Um, but I think now more so than ever, understanding which technology provider to put investment into will be a big call. Um, given all of the infrastructure that has to go around it. In addition to sort of a financial.
Speaker B: I'm really fascinated with this idea of, you know, build or buy, because one thing that I picked up, there's a really new paper from Professors Webb and Patterson from the University of Melbourne just come out recently and it's looking at what is gen, what can generative AI do at the moment. Um, it's a really good read. Should just be able to find that on Google or maybe we put that on the show notes. But one of the things they identify is that actually some of the things that generative AI is seen as sort of revolutionary for perhaps, and maybe not there yet, but in the future, things like legal research, figuring out what your arguments are, uh, drafting your briefs, just getting your theory of the case and getting your arguments out, that kind of thing might not take up the bulk of what a lawyer does every day. It might be a sort of tiny minority for different lawyers. And amongst all of the other kinds of work they are doing with clients and negotiating and persuading and just getting on with business of a firm. Um, so it's sort of interesting. If these tools are really expensive, maybe you do go for the build. And I've had a lot of joy just building with Gemini and Notebook LM and ChatGPT, knowing, you know, I know the cases that I want to look to talk about. I can, I could just do a normal search on, uh, Lexis or whatever free database and find the key cases. But then I can just build a little custom GPT or a little Notebook and chat with the chatbot to my heart's content to figure out what's the best argument here. What am I missing with arguments that I'm putting forward? Here's my draft brief. Can you red team it? Can you come up with the counter arguments? And I'm Wondering whether these big players, they're trying to get it right and I'm sure they will. But are they going to be able to sell it for this inflated cost when smart lawyers who are willing to play with the tools might be generating some, a lot of that value already? It's just a thought.
Speaker A: Yeah, it is interesting to think about. I think the reason that it becomes appealing to kind of buy would be, you know, what we talked about before, which is the sort of the, the time. And I think while there are some people that are really enthusiastic about interrogating this and building their own points, and obviously Mark is an interest of yours and sort of all of us on this call are really interested. But I would say for the most part you're kind of dealing with maybe a level of apathy in some cases, but you're also just dealing with, I don't really have time to engage with that mentally. Like I think there's an, there's an openness or a willingness to be able to use AI tools. But I think when it comes to spending the time to interrogate and to learn to the level that you would need to in order to build some of your own supporting, um, you know, whether it be agents or sort of training your own, um, model, I think that's where the sort of challenge lies. And so in such large organizations, if you're trying to embed sort of that structural change around using AI every day, then I think the, you know, that's where the appeal comes in for these sort of ready made, you know, really good in terms of the level of interrogation and thinking that has been done around these different legal specific models, I do think that there's an appeal there. But in terms of the build, you're right for the smaller teams as well. Even looking at something, you know, within the Microsoft 365 environment, that's something all teams involved, not all teams, but a lot of teams have access to. Um, and so within that comes copilot and sort of depending on your license, the ability to use Power Automate. So could you build in some workflows or use some agents within Copilot as well, you know, within your team. But again it relies on somebody having the knowledge and the capability to do that and that often is not lying within the legal team. And then even if there are people within the broader organization that may have an understanding about how to train those chat bots or sort of build those agents, they don't have the legal specific context. So it still would require a lot of downloading and collaboration with the legal team in order to get that right. But I do think it's still a really good option for teams that may not have access to the budget for maybe some of the more enterprise level and therefore a bit more expensive.
Speaker C: So we are talking about four categories of generative AI on the chatbots. One is the generic ones that look chatgpt and everybody, uh, I can access them these days. The second job, that you can build some kind of an app, not a chatbot, an app yourself. And that's what we call wipe coding, which means you just prompted, you say, I want to create an app that look like this and I can upload documents and it summarizes it in this way and that way for me. So anyone can do that. And I don't think lawyers have that kind of knowledge yet, at least as of July, August 2025, to do it on their own. So third type is a more complicated apps for lawyers. That is, there is an application like LexisNexis. There's a software or platform that we implement generative AI as part of it. So, uh, for instance, when it comes to search, there would be an option to use generative AI to search, among other things that were already there. And finally there's a category four that is building a new language model that is specified or specialized for legal context, like Harvey. That's what they did working with OpenAI. And they spent I think 2 million US dollar around 2 years ago to create some kind of, I wouldn't say, I don't want to say a completely new language model, but they use a large amount of legal data, uh, to train the model, you know, from kind of scratch. So these are four different categories. And my understanding is, from the research we have been doing with all my colleagues, Michael lake and Vicki McNamara, the only thing that has been used by lawyers so far is the generic one. I heard maybe two cases so far that lawyers have used some kind of specialized chatbot or AI, uh, implemented, you know, platforms. Otherwise, majority are using the generic ones. And that's what worries me because it may mean that law firms are not. They haven't adopted any type of, you know, specialized legal generative AI, and lawyers are working on their own and they're just using these generic chatbots because they're so accessible. And that's how a lot of them get in trouble, because it's not part of, uh, more, it's not part of a system, it's not the firm's, you know, policy. There was no training or anything. And that may suggest that law firms are not doing enough, even at a very basic level to inform or train their lawyers to, you know, how to use this technology and how to benefit from it and not, you know, take submit hallucinated cases and material to the court.
Speaker A: Yeah, I think that it's sort of, it's really highlighting the importance of having an AI strategy and an appropriate governance framework that underpins that to make sure that sort of the guardrails are in place for safe experimentation and use within an organization. And I think, I don't know, maybe I'm living within a little bit of a bubble, but I feel like most of the firms that I'm aware of have done that and they've really run quite hard at sort of having the guardrails in place, having appropriate training and capability programs. Well, um, because they're trying to minimize that shadow, you know, AI that, that sort of using of the systems without people having the appropriate under of the risks. Um, and also, you know, the, the, the actual training to, to have AI literacy to be able to really evaluate those outputs, um, and understand sort of, you know, what they need to be doing in order to ensure that they're not, you know, uploading a confidential client data, et cetera, and things like that. So I think the firms that, that sort of, I'm aware of have all done a really great job really early on of putting those guardrails in place. Obviously it sounds like there's not all firms are ah, in that position, but I, um, think, you know, because as you say, I mean, the sort of general AI tools are so accessible. It was really important upfront to be really clear on what they should and should not be used for. And I think, you know, there are some organizations, and I'm talking more broadly than law firms now, but I think there are some organizations that have said, no, you cannot use it, do, um, not use it. But it, you know, that is, it is a problem within itself because people are still going to use it. There's always going to be people that want to use those tools. They're there, they're accessible. So I think it is so important to err towards the side of having the guardrails and some sort of safe, you know, space that people can operate within to, to test and use these tools because they're going to do it regardless. So let's equip them with the right information and the right framework to make sure, you know, the risks are minimized wherever we can. Um, yeah, it is just sort of really Interesting to, to think about, you know, what that looks like. But I think with the legal specific tools, I think that's what is kind of front of mind now. And as more and more have come out and there is an understanding of what's in the market, more firms are trialing though uh, to use again they may be with specific pilot groups and not adopted more broadly across the organization yet. Um, but I think that those are really front of mind in terms of trying to understand which one is going to be worth the investment to then support the firm. Because you know, as we talked about before, there are limitations in terms of the extent of support that those general AI tools can provide for legal specific workflow.
Speaker C: So Mark, I'm going to talk about something and ask you a question. So uh, I said the way we see it in practice, a lot of law firms are not doing much about proper user putting guardrails about use of generative AI. So that's why the number of misuses of generative AI in courts, the number of cases you're finding are increasing instead of decreasing. And the one problem it seems from what Alex been saying, that when it comes to adopting more specialized generative AI, all those law firms that have to pay more and they have to try it and see whether you have the capability, now it comes the question, my question is about the capability from many of these generative AI models that you have tried and you play around. Do you see them as kind of capable technology that can actually make your life easier and make things more efficient? And asking it, like with respect to specific legal research, as a lawyer, when the uh, a lawyer may use this kind of technology because to me a lot of the advertisement out there is a hype about this technology and what it can deliver. And in many cases it is not. It doesn't make things more efficient. If you do the traditional approaches, use traditional approaches that go manually, you know, do your search, you will be way faster than. The reason it's more efficient in some cases is that people are not verifying the outputs. And uh, I'm not saying in general it's not more efficient. There are many use cases, but at the same time there are very common use cases that it doesn't seem to be as efficient. What do you think?
Speaker B: Yeah, I mean, look again, I'm a, I'm an academic lawyer. I haven't been doing so much research recently, original research. I've been teaching and I've been discussing contract interpretation, implied terms, um, this kind of thing, complicated concepts, talking to students about it. Sitting exams. Um, and what I can see is that if I train the AI on the cases, on my PowerPoints and on my transcripts of my lectures, it can get an A in the exam. You know, it knows what I'm going to say. You know, a student could argue against what I'm going to say, but, but it can perfectly predict my answer, uh, my A plus answer to a question. The other thing I can do with it is chat with it and, and just interrogate ideas and remind myself of, well, what did they say about this? In this case, it's kind of like an advanced search that is conceptual and can help you with those concepts. And of course sometimes it gets it wrong. But you know, 90% of the time I, I look at it and say, yeah, that's right. You know, I wouldn't say anything different. Now I think when I get around to doing some more research, I can really see with the sort of perplexity style or using one of the other products that kind of AI powered search, um, the deep research, I'm, I'm, I'm seeing the savings of time. You know, you've got to wade through that. You've still got to read, be able to read 5,000 words, 10,000 words. But it is like uh, uh, a, uh, web search on steroids. The other thing I can see happening is when you get a whole lot of key articles, academic articles, key cases together, uh, put that into a notebook, notebook limb or one of the other tools, get them together and just chat with them and you know, an expert like myself or any of us, you know, academics, lawyers who've done the work, they are going to see a huge amount of efficiency of being able to just get ideas out there. The AI is like a, uh, just a idea generation machine. And then experts like ourselves, we can tick them off. Yeah, that sounds right. No, I'd tweak that. I'm going to merge that with another, an idea in a particular way. But it's just that another set of hands or another brain that you can riff off a bit. Like a colleague, um, one of my colleagues said, I've just had the, I told her to use AI. She was sort of reluctant and then she went and did it and she said, look, I've just had the most intellectually satisfying and exhilarating week of my career because I got through a month of work in a week because I could just dialogue with this thing and use it as a research assistant. So that's from an academic perspective. And uh, I definitely think in practice you're going to get similar results for that kind of work. Like, that's definitely not most of what legal practice is. That's a small aspect of legal practice, I think that kind of academic style or the research and the argument, from what I understand, you're probably going to get all sorts of efficiencies elsewhere with the AI that I just can't even dream of because I don't even know what those tasks are. But yes, certainly once people play around with it and become a little bit of a super user, if you just try it and see what it can do, you will blow your mind. That would be, you know, it's not perfect, it's not everything. You know, you still have to know your stuff. But definitely people got to figure it out how they can leverage it and just play with it.
Speaker C: So I have two points. One is the fact that it can make things more efficient and it blow your mind. I've been, you know, I've been many workshops and teaching people, you know, how to use generative AI and the simplest thing that I asked them to, you know, have a hands on, you know, practice with generative AI, I, when I asked them to try it, I can see their faces, they smile like they, they're uh. It is just so surprising to me that you have never tried this simple thing and it seems many people didn't and they're happy with the outcome. So I agree that it can make things more efficient. But at the same time, like when it comes to courts and cases, again, out of these more than 350 cases of misuse of generative AI, I cannot remember a single case a lawyer says that, well, I used it to tell me what are the problems, you know, in my argument. Some of them, they say I use it to improve by writing or things like that, but not as intellectual as an assistant the way that you were just referring to. So there are many uses of generative AI that can make you more efficient. But what it seems at least majority, uh, of lawyers in practice are using it as many of them mentioned in cases that they use it as some form of Google. So they just prompt it or put their question in and they expect to get the correct answer while it requires verification. And, and that's the type of use of generative AI that in my opinion it's not efficient, that it generates a whole bunch of things and you have to verify every bit of it for millions of hours. It really, if you want to properly do it, takes a long time and doesn't worth the Time.
Speaker A: That's a very good point. And I think that's why sort of having the setup done right is so important. You know, to, to your point. I mean, it takes so long to verify something. And if you are just using sort of an unstructured prompt or asking it generally and it's not pointed at a curated set of, you know, legislation or case law or relevant documents that it needs to look at, then you know, you are having to go through and manually check everything anyway. So at that point it's much faster to do it yourself, I think. So I think that's why it's important that you know, when you are thinking about use cases or certain technology to use, it's very specific things. Like, for example, you have a document or a set of documents that you want to interrogate. Here is a set of documents. Here are the questions that I'm asking about it. Maybe in the course of sort of an investigation or something that's requiring you to put together things like chronologies. And again, it's sort of knowing what the, the AI is good at and then having those structures in place to then make it easy to verify or audit the output as well. Because as I say, if it's generating a whole bunch of stuff that's not going to be faster. Let's be really targeted and be really specific with what we're trying to get from this output. Here is the specific use case. I understand the information that I'm pointing it out. I understand my sources. So again, that sort of reduces the risk a little bit more. But then it's also still checking that what it pulled out is within the right context or it's been applied in the right way, or it's interpreted sort of a particular sentence or something that's within that document source correctly. There's still that level of verification. But I agree, you kind of have to do as much as you can to sort of point it in the right direction, be very targeted with what you're asking and what you're doing in order to actually see some of those efficiency gains. You know, for other things that are not legal work, you know, for example, you know, doing a little bit of general research about something or wanting to write an article or something for a website. Lawyers often have articles on sort of new legislation and case law and things that are coming out. So stuff like that. It's quite helpful in the more general sense. But yeah, if you were using it for a specific legal purpose, you need to be incredibly targeted and there needs to be some strategic thinking about the setup beforehand. Otherwise, the efficiencies just aren't there.
Speaker C: So this is the end of this episode. And we talked about, um, how in practice, generative AI being adopted specifically by law firms. And we talked about some obstacles. We talked about that we have generic chatbots that are not good for legal practice. And we talked about specialized legal generative AI tools, but they are problems with their capabilities, whether or not good enough for the purposes of legal practice and the cost of it. And how much time does it take to train lawyers and the staff to use these tools. And the cost of specialized legal AI chatbots because they're way more expensive. So these are all together, um, there are some of the reasons that why generative AI is not being adopted as much as we were expecting or it's been talked about.
Speaker B: How do you stay up to date with AI news?
Speaker C: Uh, I don't.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.