S2|E1: The Verification-Value Paradox - With Joshua Yuvaraj

Speaker A: How do you stay up to date with AI news?

Speaker B: I don't.

Speaker C: Welcome back to Future Ready Lawyer AI and the evolution of Legal Practice. Martin hi, I'm Mark Bennett and I'm joined again by co hosts Alex Vost and Amin Ali Madani. And in today's episode we are going to discuss a hot button issue, verification of AI outputs in legal practice. This links back to everything we've noted in previous episodes about case law hallucinations and how to avoid them. But today we're going to ask a, uh, fundamental question. Does the fact that we can't trust generative AI 100% mean that the costs of verifying what it tells us outweigh the efficiency gains that we can get from the fact that it can give us a super quick and confident response for this conversation. We are so lucky to be joined by Dr. Josh Uvaraj. A very warm welcome to the Future Ready Lawyer post podcast. Joshua uh, we're really looking forward to this conversation.

Speaker A: Thanks Mark. Really happy to be on and I'm very grateful to have this conversation. I think it's a really important one.

Speaker C: Wonderful. So, just to give listeners a, uh, brief introduction, Josh is a senior lecturer at Auckland Law School where he teaches courses on contract law, privacy and intellectual property. He's a Trans Tasman person, having completed his undergraduate and doctoral education at Monash Law School. He's also a co director of the New Zealand Centre for Intellectual Property at Auckland and has teaching and research affiliations with Melbourne Law School and the National University of Singapore. We wanted to get Josh on because he's a real rising star in this space and is really contributing to the debate in all sorts of areas. You can learn more from his institutional webpage and his LinkedIn profile which we'll post in the show notes. So again, we're delighted that Josh agreed to join us to discuss his recent paper, the Verification Value Paradox, a Normative Critique of Genai in Legal practice. And this paper has been making a big splash for reasons we'll get into and we'll link to that in the show notes as well. So just to start us off just Josh, could you please give our listeners a brief overview of the central thesis of your paper. Um, particularly that verification value paradox we hear in the title, uh, as it relates to AI and legal practice.

Speaker A: So in this paper I wanted to make two main contributions. Uh, one of them is to give lawyers a framework to help them assess is an AI use something that they'd want to, uh, to undertake. And that's quite a simple calculation or Formula, it's that the net value of an AI use is determined by the efficiency gain less the verification cost. The efficiency gain is simply how much time or money or other expense that you would save. Staffing cost, the time it would take, uh, that an AI could streamline. And the verification cost is how much time and cost, et cetera, it would take for you to uh, satisfy yourself as to the accuracy of that output and to the degree that you need it to be accurate. So for example, if you have a hundred page set of documents that you have to review for a client, and uh, integrating AI into that workflow would save you 10 hours. Now if you uh, it would only take you about two hours to verify and satisfy yourself to the standard that you'd be satisfied with, that the content was accurate, well then that's a net gain of 8 hours. Right? So that would be the time that and the cost that would be saved in terms of using the AI. So that's the first contribution. And I say the first contribution because oftentimes this debate is uh, often hit by the polarization. We have undue skepticism on one hand and we have, uh, hype on the other hand. And what we need is nuance. We need nuance to say that, look, there are many, many uses of artificial intelligence. Uh, there are many, many use cases. And, and so the question then becomes not so much do people agree with my conclusions necessarily, but, um, have I given them a framework to think through? And I think that's the first contribution. So even if people disagree with the conclusions, at, uh, least here's a framework to think about it. That's the first contribution. The second contribution of this paper, uh, is to say, well, actually, uh, the, the standards of legal practice are very, very high. Lawyers are subject to overriding duties of integrity, fidelity, and to the court and to the administration of justice. Those duties are sacrosanct. They're central, they're paramount, they're very high. And when you marry those duties with the fact that, uh, the more you give to AI to do in legal practice, the more you trust it. And the more you trust it, the more important the task is. And the more important the task is, the more, uh, the more the consequences are for getting things wrong. What you have, I contend, is a hypothesis that every increase in efficiency is going to be met by a greater increase in the need or the cost of food verification. And so in the paper I have a diagram that divides AI use in legal practice into four quadrants. And uh, I make the point that what AI is often marketed as assisting lawyers with legal research, drafting, etc, is going to fall into the, the quadrant of high efficiency gain and a high verification cost. It's going to undoubtedly save a lot of time potentially. But my argument is that's likely to offset in a lot of situations or render negligible the benefit, uh, by the verification cost, uh, because of the importance of verification, because of the standards of verification and my evidence or grounding for that. It really is the comments uh, of judges that have been made, uh, in the context of lawyers who have submitted AI generated materials to the courts where that has proven to be false or hallucination. And those judges have taken a very dim view generally. They've said things like, you've got to verify everything that the AI produces. You are responsible for the accuracy of all content. And in some cases they actually require lawyers to declare when and why you have used, uh, artificial intelligence. So as much as possible, I've tried not to uh, just say things that I'm thinking. I've tried to ground them in really what judges have set the standards and the rules are. But I recognize that this is a hotly debated issue. And in fact when I published the paper on LinkedIn, I had a, uh, lot of responses, far more than I thought I was going to have and a lot of um, uh, a lot of support actually, and a lot of lawyers saying, you know what, Actually this really mirrors my experience with uh, with AI use. Some were saying, well actually, you know, you should think about this. And I think it was meant to inspire debate and to bring some nuance to the discussion. So really that's the focus of the contribution of the paper. One, the framework to think about it. And two, my hypothesis about the correspondingly greater imperative of verification, the more the efficiency, uh, gain increases.

Speaker B: George, thank you so much for summarizing the thesis, your, your paper. Uh, I just want to have a quick ask this question just to clarify something. When you say the productivity paradox, and you are considering how long does it take for generative AI to find the, you know, answer or generate the answer, then how much does it take to verify it? And then that would be net. What was it? That net value? Yeah. Are you considering in this formula how long does it take to do the test traditionally? Like if the task takes 20 hours traditionally, and with generative AI it takes like an hour and a half and then verification takes 5 hours, we are still being more efficient.

Speaker A: Yeah, so that's a, uh, that's a good question. So that would fit within the concept of the Efficiency gain. Right. So that would be. Your efficiency gain is if you got your junior lawyer to do, uh, the research task, that would take them, let's say, six hours. But you could put the same question into the generative AI chatbot and get a legal research, uh, response in, I don't know, let's say 30 seconds or a minute or something like that. Right. So your efficiency gain is, I don't know, let's say five hours. And the question then would just become how, how long it takes you to verify, uh, that to the standard that you need to, um, that that would be the framework.

Speaker B: So given practitioners also verified junior lawyers, whatever they provide, wouldn't that be kind of a similar process? And in my opinion it is not. But I just want to say, what, what do you think?

Speaker A: No, I, I agree entirely. Uh, I think it's a valid question to ask and I think more broadly, I'm an intellectual property scholar as well, and I, I'm doing a broader project on copyright and creativity. And I only mentioned that by analogy because I think the AI stuff or discourse can, uh, often make us miss the forest for the trees. If everything you talk about is AI, then you miss the broader context of the disciplines and the economies and the structures that we work in and a chance to actually make those better. So the junior lawyer point is a great example. Right. Because oftentimes, and I'm not, this is, uh, not necessarily, um, making an empirical point here, but my sense is that, um, uh, the senior partner who asks for a research memo and enters into a letter of advice or whatever may sometimes only give that a cursory look and think that that's perfectly fine. Or they might just slap their name on that and put the letterhead on that, and that's also fine. So, um, in that context, if you were to say that there are similarities. Absolutely. Uh, and I agree with you, there are differences, but I think it opens the door for a broader conversation. I mean, do we want that to be the case? Do we want those practices to continue?

Speaker C: So with that basic idea of the verification value paradox, we've got one of the key concepts that you've put forward in the paper, but I thought it was a really useful exposition also of some features of generative AI that all lawyers should understand because they inform why you think there's always going to be that verification problem or verification task that's got to be there if you're using generative AI in a, ah, legitimate and robust way. So there was the transparency problem and the reality problem. So I wondered if you'd just briefly walk us through some of those concepts and what role they play in your argument.

Speaker A: Thanks. Yeah, I definitely should explain those. So first thing to note is that there are different types of artificial intelligence, right? And you guys have probably covered this on prior shows. I'm talking about generative artificial intelligence, where you ask the AI to output some sort of content, right? And we can see this in uh, video content or music. And in lawyers, uh, in the sense of lawyers, it's largely going to be text based content, right? So that's the first clarification. There are two flaws, as you said, the reality flaw and the transparency flaw. And I tried to do this as much as possible to be structural rather than developmental, because one of the dangers of AI discourse is that it gets uh, outdated pretty quickly, right? GPT 5 may be better than 4 and 3.5, et cetera. Uh, and so I really wanted to sort of dig down on, well, the actual nature of this technology independent of model development, what is it? And so that's where I developed, or I got from the literature, these two ideas. The reality flow is this idea that a large language model, which is the technology on which AI chat bots are built, is not cognizant or aware of the accuracy or truth of what it is producing, right? It's trained on billions, perhaps trillions of data points of images, words, et cetera, from which it derives statistical patterns. Now the complexity of that process may well increase, but at the heart of it, it's statistical, uh, determinants. And therefore AI model were only trained on a thousand images of in fact black cats, but they were told this is a white dog. So then if you ask that image, that AI model for a picture of a white dog, it's going to give you the picture of the black cat. Because statistically that's accurate, right? But it's not actually in reality that. So, uh, that is a flaw that is endemic to AI models unless and until there's a massive paradigmatic shift and AI becomes aware, which then there are a lot more questions than legal practice when that, that occurs. And it's important to note that uh, what this means, it means that AI can hallucinate, it can end up with mistakes that sound plausible but are, uh, in fact wrong. And in legal practice this manifests not just in the fake citation problem with cases that don't exist, but more insidiously, as I think you pointed out In a recent LinkedIn post, Ma, the inaccuracy of summarizing or uh, uh, expositing the law. Right. So it might say case A said X, case A may exist, but it might have said Y, not X. And there may be degrees of variation that are, you know, there's a, there's a kernel of truth in there, but it's not quite there. And accuracy matters for lawyers. So that's the reality flaw. This non tethering to reality, the transparency flaw kind of flows from that. Because if we can't trust an AI to produce work that is tethered to reality, just work that sounds good. Right. Then we need uh, it's beneficial to see how it reached its conclusion, to show it's working. But the problem is most or many AI models will work as black boxes. You can't reliably open up the model and see how it reached a particular decision or decision, uh, or the uh, process it went through to output that particular thing. Now some who have used say more agentic or thinking based models, right. Would counter and say, well those models actually say, I went to this website, I found this data, I did X, Y and Z. And that working absolutely is possible. However, there are concerns that even in doing so the model may actually be misrepresenting its process. In other words, we don't have a reliable indicator that what it says it does is actually what it does. Right. Uh, and so those two flaws ground the need to verify content uh, by humans because uh, we simply cannot trust it to produce uh, accurate content in the first instance. Now I hear the counter from many lawyers or many lawyers will probably say this. Well yeah, but it actually in fact does produce stuff that's accurate. And so I would say there is a Venn diagram. Absolutely. And an overlap between what the AI produces and what's accurate. Right. My problem is you can't predict with any certainty what that overlap is going to be for your next task. It might produce something absolutely accurate, it might not. And so are you willing to live with that uncertainty is the question that I wanted to put forward. So those are the two uh, flaws.

Speaker C: Yeah, it's so fascinating to kind of lay that out. And it's what anyone who is wanting to use AI has to think about. It really prompts me a couple of ideas for me. Number one, yeah, we're always going to have to verify. And the way you verify might depend on how much you've played with these systems and checked, you know, how much verification you've done in the past for certain kinds of tasks. You might be able to get to a level where you say, well I Trust what? Quite a lot. And then my verification takes a particular form and maybe it's taking less time in terms, of course, your formula. The other thing this makes me think of is if functionally you can verify that this has got you a really good result. Does the reality flaw and the, uh, transparency floor really matter? It's like a black box. But it's got you the thing that is going to save you however much time, because you're going to go verify that. And it's just this amazing thing. And I've seen a lot of lawyers talking about it in that way. But I think it's really important to kind of think about what these flaws mean. My final observation before I let others jump in is that we kind of really need the verification task to still be a valuable aspect of legal work, or else all of that work goes away. You know, it's, it's, in a way, it's, it's quite good. If the verification task is necessary and for all the reasons you outline in your paper around duties to the court and civic responsibility and a, uh, dedication to the truth, it is necessary. If AI gets too powerful, that's when we start saying, well, for certain kinds of legal tasks, you don't need the person. And I don't think many people like that idea, at least not lawyers or law professors.

Speaker D: I agree. Mark. I was just thinking Ben around, I don't know that sort of, uh, most people or, you know, the majority of people in the legal profession would want a world in which there is no verification. I think, you know, obviously, as you say, it goes to the root of the profession and, you know, it. A lot of judgments are based on sort of ethics and the devotion to truth and all of the things that you mentioned. So I think outsourcing that entirely to AI is a really sort of concerning, um, concept to most lawyers, and I would say most people even outside the legal profession. Um, I think that point around the, you know, the transparency and reality flaws, they're good ones and they're things that are really important to keep in mind. As you say, Mark, if the output still gets you something accurate, sort of how, you know, impactful are they? Um, especially if. And I think this is where it becomes really important for the lawyer or the person verifying the output to have the subject matter expert expertise, which is where it kind of goes hand in hand. It's really tricky when maybe you've got a junior reviewing an output that doesn't have those years of experience to say, oh, this is not particularly applicable. In this context, you know, maybe they're able to validate now that it's become more publicized around the hallucination and the false citations. But again, is this the right way, you know, to apply this case law or to apply this legislation? Um, you know, so it still becomes really important for, you know, the AI literacy and the ability to use the tool to go hand in hand with that experience and sort of developing out, you know, that subject matter expertise.

Speaker A: I was just going to say, I totally agree. And I think the difficulty is you can produce things that you need. Absolutely. Um, in fact, uh, the diagram in the paper I used as a case example, because that diagram, I generated the code for that using Microsoft Copilot. But I knew how to code, uh, from some prior experience. And because of that I verified that the code would produce what I wanted to. And I still had to obviously manipulate it quite a lot, actually ended up taking a lot of time. So that was the illustration for me, uh, to get it to where I wanted it to be. Um, but yeah, I do agree that there are a lot of variables in the use. One point I would add, which I think is material, because a lot of people would say, well, he can replace the junior lawyer. To which I would say there's probably a few, um, differences. Right. Number one, the junior lawyer is or will be admitted to the profession in their own right. So they are a moral agent in their own right. They have duties to the court independently, their duties to the client independently. Uh, they're not just, uh, a blind tool. Okay? So, so there's something there that, at least for them, if they're presenting work to you, uh, there's some notional sense in which you can rely on them to have produced work that will be of a reasonable quality. Granted, a lot of Junior's work would not be up to the standard. Uh, but there is that idea. The second is the relational aspect, right? Because the junior lawyer is employed by the firm or the judge's clerk is employed by the chambers or in whatever capacity. And that relationship is one. Uh, there's a, there's probably a. There is a contractual relationship absolutely, with obligations. Uh, arguably this, this, those obligations, then, um, they import some level, some level of trust. Right. Uh, whereas I think there's more to be said on whether or not the trust one places in a tool can commensurate to that, that level of trust. Because Mark, you made the good point. You said that, well, um, you know, if after experience, then I actually trust that copilot or GPT5 or whatever can, you know, produce that. And that may well be, that may well be accurate. I do think that there may be a difference with the junior lawyer. That means we need to be careful with the compare, comparing the two. Um, and so that's probably what I add. But absolutely all your points are incredibly

Speaker D: valid I think as well in relation to junior lawyers, given the fact we will always need verification, whether that be because the tools sort of aren't accurate or again just from an ethical sort of devotion to the profession standpoint that will always be required. So we will always need lawyers. And so there has got to be some way from, for junior lawyers to then reach the level of expertise that is expected of um, a more senior lawyer that may be verifying the outputs instead of verifying the output of a junior lawyer perhaps. So it's, there's that real discussion around how do we make sure that that experience is still built and that they are still given the appropriate training and support and development. And so, you know, as you said, there is an argument that AI could replace some of the junior lawyers or maybe there's not as many grads, et cetera. But the thing is that will ultimately have impact down the line around sort of people that are appropriately skilled to continue the profession and continue providing advice to clients. So you wouldn't want to do, and I don't think it's possible to, but you wouldn't want to do a one to one replacement because therein lies kind of the end of the profession if there's not people coming through and continuing to build that practice.

Speaker C: Can I just follow up on that? Because in the law we have this philosophy of the bad man. Oliver Wendell Holmes talked about the bad man doesn't care about all this fancy stuff around morality or um, dignity or inherent value. They just want to know are they going to win this case? What are their legal rights? And a bad man in this situation might be thinking, well, can I get this result for 10 times less cost? And I just mentioned that in relation to junior lawyers because there was an interesting experiment recently, Channel 4 in the UK. They pitted a junior lawyer, uh, against Garfield AI, which is basically a small claims automation generative AI system which will give you the thing you need a file to make a, to collect small debts in English courts and judged by the junior lawyer's supervisor, Garfield AI produced something which maybe wasn't quite as good as what the junior lawyer produced, but at a tenth of the cost and presumably with the same kind of time expedited production of that work. Product. And there's that idea of a good enough, if it's good enough to do the legal task you want to do, maybe you're fine with that.

Speaker D: I, I think it is an interesting concept to explore, particularly if there are certain clients that wouldn't be able to afford perhaps a human to do the entirety of, you know, that analysis or that case. I've heard examples of kind of clients saying I need to do this document review, um, and I need to check for XYZ clause, but I don't have the time internally to do it, but I don't have the money to pay you to do it manually yourself. It's kind of like do they accept the risk of a, uh, largely AI enabled or modified review, noting that there could potentially be, you know, flaws in that, or maybe it's not as good as if a lawyer had gone through meticulously each and every contract to identify some of those things. There's kind of that interesting discussion around in order to reduce cost, do you accept risk? And you know, in some areas that's unacceptable. You wouldn't be able to kind of accept that risk because of the ramifications flowing from that. But maybe there are areas of law, again, if the client is comfortable and fully aware of what they're going into, could that potentially provide access to certain services that they wouldn't have? Um, you know, in other circumstances?

Speaker A: Yeah, I think there's a broader discussion there about access to justice, um, and the difficulty people have in accessing sort of legal representation and advice. I would say probably two things. Right. I think there's a, there's, as you said, uh, Alex, there's always going to be the risk of mistakes and so the client is just going to have to assess do I bear the potential consequences of that, which may not be that material in quite small disputes, but if you have sort of uh, small to medium enterprises that are still worth quite a lot, or sort of large trusts or independently, uh, wealthy people and they're relying on um, solely AI generated thing which could cost them quite a lot of money. And as lawyers we know that one word in a contract can actually end up in, you know, a lot of money being paid to one side. So I guess there's that, that risk calculus that every client needs to make.

Speaker B: I think just you made m very good point. And I want to refer back to what Mark was saying about the documentary and UK TV that I wonder if they ran the, the AI like 10 times. We did every 10 time get it right. And if it gets it wrong once. We should take that as answering that question or legal task incorrectly if it's a part of a, uh, you know, benchmark. I think that's very important to consider that the AI models, sometimes the output may differ if you try it several times. And when it comes to humans, it makes it less likely because we usually try to follow the same logic. But I don't say it's necessarily the same. And I think we think humans are really good at various tasks. We are very overrated. We constantly say, oh, AI should be accurate while we are not. One important question here is, when it comes to various tasks, what is the human accuracy? What is AI accuracy? And if we run, let's say, each task 10 times, considering that, and then we may say it's sufficiently good because it's. Humans are accurate like 97%. This AI is like 97% as well, or 98%. And we don't need 100% because nobody's 100%. I think that's one part of the discussion. But another thing is I want to look at it from a broader perspective. I totally agree with Josh that, uh, when it comes to many tasks, it is inefficient to use your AI. We already discussed this previously in this podcast, and in my last study, empirical study with the students, a lot of students in their reflection said that if I did it traditionally, it would have been way faster than using generative AI because I had to verify everything. And it took me, uh, a lot of time. And I think part of this problem goes back to the. After November 2022, when ChatGPT was released, early on, there were a lot of people who didn't know what they were talking about. And they just said stuff about the technology that they didn't know exactly how to work. And they said, oh, it can, you know, help with, uh, legal tasks in this way and that way. While in my opinion, the technology is shifting every day, it is not good for many tasks because it's not made for those tasks. So from very beginning, we got it wrong and we are now saying, okay, for this task, it is, uh, inefficient. While we weren't supposed to use it for this task, you know, for other things can be helpful.

Speaker D: It's such a good point because I think as well, when there's sort of, there is so much hype, as you said, it's like, oh, it's gonna, can do everything. It's like, just get on there, you know, like, start using it. And I think, yeah, there has Been a bit of a lack of thinking about which workflows or which work types within legal are best suited to this kind of technology and then how you would work in with it. Like for example, obviously asking AI to just produce a draft advice based on a set of circumstances, pointing it to some case law and legislation and things like that, it would come out, but again, you're having to read through, re edit the whole thing anyway because it's such a large output, again this sort of text and there's not as many parameters or structures around it to help move it through a process that then ultimately starts reducing some of those verification elements versus kind of having something that maybe, you know, the firm has agreed precedents or templates in that area. We've had similar circumstances that we've pointed it to within word. I'm assessing it within word and I'm seeing markups in real time. So I'm able to verify because there are some of those legal specific like AI tools now that kind of have the word plugin. And so again, you're not sort of staring at this big chunk of text that's been outputted by the AI with very minimal prompting or minimal guidance. You're starting to see something that kind of leveraging the structures around it or some of the pre work and the pre thinking that's been done to then have it in front of you and you can validate in real time as you're going through. So I think it's kind of the way that we're taught to use these tools and again, the circumstances that they're being applied to. Because to your point, I mean, in some circumstances there's, you know, I think, real opportunity for efficiency gains, but then in others, you know, it goes to the heart of the paradox where it's kind of, you know, you are spending so much more time verifying the outputs that it was not worth putting it through, you know, an AI tool in the first place. So I think there is nuance to that and there are certain things like when sort of law firms or in house legal teams are planning how they go and use AI. It's so important to think, you know, is this task the right task or is this workflow the right workflow? Considering all of these different elements?

Speaker B: I just want to add something very quickly and following what you said, Alex, I totally agree. And I think that's what makes Josh's paper important because it reminds us that maybe we should rethink artificial intelligence, at least for some specific tasks, or maybe we should rethink where it is efficient. Because Josh very openly says, this is my, you know, theory. I'm saying this. I don't know, we gotta, you know, test various things. But looking at the facts, this makes sense. So then we should test Josh's hypothesis. And that makes us rethink that, whether it's actually helpful in this area. And I think it's a very good way of rethinking about where generative AI is useful and where it's not. And again, Josh also mentioned in the paper, and right now also he did, uh, many M judges say that, oh, you have to verify the output. There is nothing about efficiency. Many courts, both in, in Australia and the US we have discussed in the other paper with Vicky and Michael that they constantly, uh, try to encourage people permanently to use generative AI, but they suggest, verify it while maybe you're not supposed to use it for those purposes, because if you want to verify it, it would take way more time. So they're encouraging people without knowing that it is not really helping them.

Speaker A: I think there's also, um. A good point, Mark. You, you shared something on LinkedIn. You're a great follower on LinkedIn M. But there was one, uh, judge where the lawyer got reprimanded for saying, oh, I don't usually read the cases. Right. And again, I like this idea that AI makes us think more broadly about the profession and more broadly by how we do things. I think that then it's useful. Right? Just makes us rethink things. Uh, I wonder if there's, uh, a shift in workflow then, in terms of, um, uh, the sort of engagement that's needed more with content that we don't just skim things. And yeah, I thought that was really, really useful because it's analogous to actually if we were to adopt a Socratic teaching method at law school, uh, which I don't. Um, but if you were to say, okay, students, have you read this case? What did this case say? Well, imagine a judge asking you to tell them what the case said. It's like, I didn't read the case. Um, so again, I'm not sure there's a right or wrong answer. Then it's more. More a question of if that's how judges are operating. The other thing that I think has gone un. Under, uh, explored in the discourse is the question of liability. And a colleague, Lucy Holloway, brought this point to my, um, attention. Well, what do professional identity insurers think, right? Like, what's the likelihood of coverage in the event of a negligence lawsuit brought against a, uh, firm for AI generated content. In a piece I published a newsroom in New Zealand this morning, I made the point about, well, Deloitte got in trouble for that. Right. For submitting a report to the government that had AI generated apparently hallucinated content. Okay. Now the only liability there, or the ramifications are Deloitte has to repay the government a fee. Right. The part of the fee and obviously the public shaming that has occurred. Right. Those are the only, uh, liability issues. Now, uh, it's not hard to imagine that given the widespread uptake of AI use, particularly in Australia and the legal profession, that been the first negligence lawsuits. I'm not too far away for. Oh, you advised me of this and actually there was a mistake in this lengthy advice or lengthy review of documents or whatever. And as a result I lost millions of dollars. Right. And in that case, I think it'll be a very interesting experiment to see, well, will the professional indemnity insurers cover that, uh, and will they start requiring, you know, that kind of disclosure? And how will insurers be involved in regulating or, um, influencing the application of AI use in the profession? I say I think that's under explored as well. Nothing about the sort of moral, ethical motivations, purely from a consequential point of view. I think that's a really interesting question

Speaker B: following what Mark was saying. And I totally agree that sometimes it's not about efficiency, it's about quality. You want to reach a quality that even if you spend hundred more hours, you won't get it. But if you collaborate with AI, there's a possibility you would get that. And for many of us, when I, uh, put academic work out there, that quality is very important. And, uh, there are many different ways to collaborate with AI to increase the quality. It doesn't need to do the whole thing. And you verify it can be the other way around. You do the whole thing and it verifies, and if it finds an issue, an error in your work, it would save you 20 hours, you know, later, because once you submit it, they say there's a problem and you have to go back. There's a whole process. Maybe you have to pay more. So, uh, yeah, there are many other things involved that efficiency maybe can have different meanings. So sometimes efficiency is not a priority. It's quality.

Speaker D: And I think things are changing so fast, right? It's like, how do we even know what the profession is going to look like in five to 10 years in terms of how we work and the way that we deliver Work now and sort of apply the law and all of those things. It's kind of, you know, what is that going to look like? So how do you best equip them for a future that we don't really know? Um, and I think one of the things that I valued most about my law degree was that it was almost, you know, teaching that skill, as you've all mentioned before, is the way of thinking and the critical thought, the logical reasoning and how you would sort of move through these different, um, you know, sort of thought processes. And I think that that's valuable across any, um, role or profession. And it's something that I've really valued and I think that there is an element of the way. And again, I'm not saying these are exactly on par, but when you are working with AI or having to use an AI tool as part of legal work, you really do have to go back to breaking down that thinking process in order to prompt it appropriately. Put the right structures in place, use the right inputs. It's sort of, you can't be on autopilot anymore in order to get a quality output. Not that you could really be on autopilot before, but I think with the AI, it sort of helps you think through things in a bit of a different way and you're having to brief more things in and it's almost highlighting some of the things that maybe were a challenge as a senior person briefing to a junior person where you're like, oh, that wasn't actually a very clear instruction, of course. That's why I got an output that probably wasn't aligned with what I was wanting. I think working with AI really highlights some of those things as well. So teaching the concepts about breaking it down and how you would step through this and then sort of, again, it's building up that technology skill that you may not have been, you know, building previously. I just think it's a really interesting way of, of having to think and structure and again, thinking about how would I verify this, what are some additional prompts or some additional things I would build into this tool in order to make it easier for me to verify. And it's kind of that higher level thinking skill set that sort of maybe sits outside of law, but will still be valuable across, um, any profession, particularly given most of them will have some element of AI integrated, I suppose.

Speaker C: Okay, so thanks again so much, Josh. A real pleasure for all of us to have you here today.

Speaker A: First, thank you for having me. I think it was a really stimulating, uh, discussion. Exactly the kind of discussion that I wanted to spur by writing, writing this. Uh, and, yeah, just, uh, an encouragement for people to keep a critical mind. Benefits, risks. I think there's a real danger in going one way or the other, hype or skepticism. And we just need to keep a really open, uh, open mind and ensure that we are still. If we're lawyers, we're still fulfilling our duties to the court and the client, which won't go away.

Speaker C: We'll give the listeners the link to the paper and to your LinkedIn profile, and they can join the debate. Um, and, yeah, we look forward to the next paper and having, uh, you on again. So, yeah, thanks. Thanks a lot, Josh.

Speaker A: Cool. Thank you.

Speaker B: Thank you.