Misuse of Generative AI in Courts - With Vicki McNamara

Speaker A: How do you stay up to date with AI news?

Speaker B: I don't. Welcome to this episode of Future Ready Lawyer AI and the Evolution of Legal Practice. For this episode we have a special guest, Vicky McNamara from UNSW. I'm just going to give Vicky a, uh, quick intro. Vicky is the Senior Research associate since early 2024 at the center for Future of the Legal Profession, Faculty of Law and Justice, University of New South Wales. Vicky has over 30 years of experience across the legal profession and is admitted to practice law in New South Wales. Her key areas of professional expertise include knowledge management, legal tech and innovation. Vicky and also Michael Lake are ah, my co authors in our recent paper on the promise and peril of the user generative AI in Litigation, which will be published in UNSW Law Journal in December. The paper is in general about how generative AI has been misusing courts across various jurisdictions, possible reasons behind it, and some suggestions on how to fix this mess. And today's podcast is mainly on this topic. So, Vicky, let me just start with a question to open the discussion. How many cases do you know of globally that genai has been misused in courts?

Speaker C: The cases that we've identified in our research, which is focused on common law and hybrid jurisdictions, are currently totalling over 470 examples. And in that case data set, we're mainly looking at cases where there are references to the use or misuse of generative AI, uh, during proceedings or in the lead up to proceedings. Um, we know that there's actually probably a lot more cases out there. Not everything makes it through to that reported stage. Some things are probably picked up in the case management stages and therefore the judge or um, member or whoever's deciding a matter doesn't actually need to reference the use of AI. But um, we also know that uh, there's, there's similar things happening in other jurisdictions that we're not studying within our research project. So there's a lot of it happening up there basically. And 470 is probably the tip of the iceberg.

Speaker B: So it's way more than what is being reported in the media. Because I usually see in the media they say, oh, there are like 10 more cases, uh, you know, yeah, lawyers or other people in the court, they somehow misuse generative AI. So the problem is way bigger than what we see in the media. And what jurisdictions are these cases from?

Speaker C: So in our research dataset we started with Australia, we then added the U.S. uh, and then cases started to appear in New Zealand, um, obviously the uk, Canada. Uh, I did have a look at some material in India, we could see a bit of activity there. At one stage Ireland did pop into the mix, um, because the UK is more than just England and Wales. And uh, then we also started to see some activity in the Southeast Asia and Asia region. So most recently um, I picked up some activity in Hong Kong and just this week Singapore.

Speaker B: And how do you find the trend so far? Do you feel like there's an increasing or is decreasing?

Speaker C: There's been a huge jump in 2025 so we started looking at for cases basically from when ChatGPT was released at the end of 2022. Our Ah, research actually started 12 months after that. But we did backtrack and see what had been going on in the courts during the period since ChatGPT was released. But at the end of last year when I did some presentations, I had a data set of around 90 cases. So you can see how enormously that has jumped. And it's not that we've added a whole lot of new jurisdictions, we've been looking at all of those jurisdictions in that period. But the activity has definitely spiked in 2025 to the point where I'm now struggling to keep up with the volume of cases, particularly those cases coming out

Speaker A: of the U.S. so Vicky, um, what kind of characteristic AI misuse are you seeing coming up in the cases? Is it sort of one or two main ways that this makes its way into a reported case?

Speaker C: Look, the thing that we see most often is that uh, the idea of um, hallucinations or fake law where people have used AI in preparation, they've used it for research and it has invented the law, it has invented cases, it's invented references to legislation that may be repealed or doesn't exist or sections of existing legislation. Um, so that's the big one that everybody talks about. And certainly the majority of the cases in our data set have that characteristic. But there are some other features where we think AI use is also causing people issues. So for example, if somebody uses AI to draft document and they're supposed to be complying with a court or tribunal commission's procedure, it can often steer them in the wrong direction and they end up with documents that actually aren't procedurally correct. And the other thing that we've seen, particularly where litigants tend to be on the vexatious or perhaps sovereign citizen side of um, the scale, we do see AI being used to put completely spurious legal arguments that support their political position, if you like, or whatever argument it is that they're trying to push through the court, and also volumes and, you know, just to create volumes of documents that somebody has to read. Um, and, you know, just to see if there's some kernel of a legitimate claim. Uh, so, yeah, it's not just the hallucination issue that AI is facilitating.

Speaker A: Yeah, it does bring me back to the kind of stuff that I deal with as an academic. Um, I think at every university people are finding that, um, some students are using AI and really the hallucinations is the easiest way to spot it. But as soon as you see that, it casts suspicion on the whole paper and you've got to start working out, well, if it's hallucinated, this reference, how much of the paper has it hallucinated? And that's just so tricky to understand. It really has a, uh, wider effect on trust in the thinking professions.

Speaker C: I think it's manifesting. That issue of trust is manifesting not just in the litigation context, in the legal profession. Some of the work that the Centre does on the side is, um, training, um, firms who want us to come in and give them generative AI training to help them with their adoption process, to make sure that AI is being adopted responsibly. It's always interesting to talk to those who are being trained because they're telling new stories from the coalface. So a few weeks ago I had a, uh, couple of sessions where somebody shared with me the experience of where they had received documents to review in a transaction, uh, from another law firm, all of whom will remain nameless. And they started to look at the documents and they looked fine on their face. But then they started to realize that it had probably been AI generated and a document that they might have otherwise been prepared to trust is largely correct and maybe just dispute, um, particularly particular commercial issues or risk issues. They then found themselves having to go through line by line and they said basically it turned out to be a document that was nonsense at the end of the day, wasting everybody's time, but also undermining the trust between the two practitioners.

Speaker B: I think that's the. The time is a very good point that you mentioned. Given the number of cases identified globally. And we know it's way more than that, we can imagine how much time is being wasted by parties and the court, uh, every time they have to go through the documents and look for cases. And it's as we know, it's not like if you don't find the case in a database, it means it doesn't exist. You have to check multiple databases to make sure if you're accusing Someone that you have submitted a case that doesn't exist, you're sure that it doesn't exist. So the amount of time that is being wasted is extraordinary.

Speaker C: I assume we haven't done a study of the time wastage, which you can see coming through in the comments from courts. And that is definitely one of the court's primary considerations, apart from, um, you know, the fact that evidence is being infected, if you like, by AI, um, generated content. You see hearing dates being vacated. You hear stories of judges, associates running around looking for cases, not being able to find them. And then obviously you've got that administrative process where you've got the communications going back and forth between parties trying to work out what has happened. And this is all in addition to what was ordinarily, um, probably a very administrative load. Heavy, heavy administrative load and procedural load for people to follow. So that misuse of AI is actually adding to everybody's workloads and the touted saving of time and the efficiencies that AI is supposed to be producing are being eroded by these sorts of behaviors, particularly when it comes to legal proceedings. And these are often in um, courts that have already have very heavy lists. And I did see, uh, like quite a pointed, but I won't call it humorous comment from one of the judges that said, yes, you are entitled to have your day in court. What you're not entitled to, and I am paraphrasing very heavily here, what you're not entitled to is to have somebody else's day in court. And that is what is happening with AI use at the moment.

Speaker B: That's assuming it's true for the courts, it's a lot of waste of time. But for lawyers, when generative AI is working, it saved them a lot of time. But we don't have that information. We don't know how frequently it's working and how frequently they get caught in court because of hallucination. Yeah. But, uh, from the look of it, courts are being involved way more than they should be. I hope we had some more information in terms of like how much time is being saved for lawyers.

Speaker D: Yeah, I think the whole sort of time saving piece is really interesting to think about because you're right, if you're kind of having to verify things in more detail, or if you're going through higher volumes of documentation, you're not getting there's sort of time saving, um, elements. And I think the measurement of the productivity or the time saving is also an interesting one. While law firms still operate on the billable hour as Well, I do think law firms find it hard to track the level of, you know, time saving that comes with using AI in these particular cases. You know, you know, to Vicki's point before, if you're submitting something that's already very like, there's high, high volumes of documentation, it's already very time consuming process. If you're either, you know, not having a true understanding of what the productivity or efficiency gain is within a firm, and then you're also adding time to the way that the court is able to review these cases and um, you know, review these different claims, then it's sort of, you do start to question what's the real benefit here? Like how do we really clarify what the value is coming from sort of incorporating AI into these workflows. If at the moment it just seems that there are quite a few instances where, if it creating more challenges. Um, I know that there is obviously benefits and I've seen some of them, but I think it's really hard to convince people that may be more skeptical of AI's benefits and AI's use when you are seeing examples like this. Yeah.

Speaker B: And we also know that some courses are not dealing with the matter of fabricated citations. So we know some courses, they just orally deal with it or in uh, uh, Vicky, tell me what you think. But in majority of cases I've read maybe more than 80% of them, the courts don't bother talking about fabricated content. They just mentioned that it seems they use, you know, generative AI. This case is, you know, fabricated or the content doesn't exist and they just move on. They never go back to that issue. And that's uh, seems like they have basically enough time. They don't want to, you know, spend any more time on this and just skip it.

Speaker C: I think I'd agree. Arm and there's some cases which have m, particularly the more recent ones. But then very early cases like Matta and Avianca in the US where the judges spent quite a lot of time dissecting what's gone on, uh, using the case as a potential, um, vehicle to educate the profession and other court users about the risks in using AI poorly. But then in other instances you do see, particularly if the AI use has not really affected the substantive issues being decided by the court, the court will go, yes, it appears that AI has been used and so it's not always confirmed. Sometimes it's an inference, usually based on the fact that there's a fictitious case or nonsense arguments, um, have been put up, uh, and the court will move on to the substantive issues probably for expedients as much as anything. But then you will have cases like Mayan Kostros which recently came through the Court of Appeal in New South Wales where both the uh, Chief justice and um, Chief Justice Bell and Justice Payne I think it was, um, both spent a bit of time commenting on AI use and again that's, that's using that case as a vehicle for educating in this instance self ah represented litigants in that forum as well as other New South Wales courts. So um, what I am seeing in the cases I'm looking at, you know through 2025 there is increasingly more commentary and I think part of that also comes from the courts realising that this is not something that's a passing fad. It's not going away that some of the initial guidance that's being released probably isn't quite hitting the mark and it needs to be backed up with other forms of education and guidance for people. It's not just enough to release a practice note and a press release related to the practice note and hope that that will have the desired effect. It actually needs to come through judicial commentary as well.

Speaker B: We will definitely go back to the topic of education um, and just before we go there because our discussion is as if uh, it's only lawyers are doing it. But again in my database around 80% if not more are self represented litigants and not lawyers who've misused generative AI. What do you think?

Speaker C: Oh, completely agree. And that's the pattern across most of the jurisdictions we're studying, apart from those that only have one or two cases where that skews the percentages. But certainly in Australia it's more than 80% of the case examples in our set are self represented litigants. We do see the occasional expert witness or third party helping a self represented litigant um, get into hot water as well. Most of the media attention and commentary focuses on lawyers using AI in poor ways and that's obviously it cuts across lawyers role as officers of the court and their professional duties in particular not to mislead the court and to assist in the administration of justice. But in terms of quantity it's definitely self represented litigants who are uh, leading the march on AI misuse. And that says to me and to others who are looking at this data that more needs to be done to educate people who are in that position who often don't know their way through court processes anyway and come across this magical thing called generative AI, uh, who will answer their legal questions almost instantaneously at low cost and usually give them the answers that they want in language that they understand that sounds convincing and that's how it often ends up in their pleadings and submissions to courts without being checked.

Speaker D: Yeah, it's kind of a real question of access to justice. Right. And how do you sort of enable people to be able to self serve in instances where they maybe can't afford legal representation but you know, structured and sort of, you know, sound way that's not going to impede the time it takes, you know, to move a case through the courts. Like how do you. Do you have any thoughts on how AI might be used to better enable self represented litigants? Or is that kind of, I don't know, a bit of a nebulous concept at the moment given there's so many different pathways you could go down?

Speaker C: Uh, I don't think it is a nebulous concept. I think it's something that the profession and the judiciary needs to grapple with as well as the broader legal legal services community. We know that self represented people are going to use these tools. It doesn't matter what guidance you put out there. So what it really comes back to is that point about educating people and giving them the right guidance. Um, and when I was preparing for another interview earlier this week I had a look at a few of the sources online and that are targeting that cohort. And really what I see is a lot of material, whilst it's well intentioned, has probably been drafted by lawyers for lawyers and so it lacks that plain language and consumer um, educative approach if you like. Uh, some exceptions were some um, material that I saw on the Fair Work Commission's website. I saw some material that I think is spot on and is pitched at the right level. Not just at self represented litigants using AI, but other procedural issues as well. Um, preparing for here when preparing for hearings and I did notice that the Federal Court of Australia has a pro, I think it's litigants in person and pro bono project, um, underway. There's a consultation period that's just opened up and I believe that we will probably see coming out of that some materials that target that cohort. Um, but you know, specifically with tools that, and resources that are geared to, you know, the different, the very disparate types of litigants that you get um, help try to self help themselves through court procedures.

Speaker A: Yeah, it's interesting that um, the New Zealand courts have provided generative AI guidelines and they're for lawyers. They're also for uh, judges and people working within the courts. But they've also got some guidelines for non lawyers as well. And they are written in quite um, everyday terms with a sort of basic headline idea, um, like number one, uh, understand generative AI and its limitations. Before using generative AI, chatbots ensure you've got a basic understanding of the capabilities and limitations and then it sort of lists out, you know, this is what it could do, these are some of the capabilities and then it goes through about ten limitations. Um, and there's a few other quite good guidelines for those people, litigants in person, non lawyers engaging with the legal system. You wonder whether people are searching that up or getting that presented to them or whether they bother reading that. But um, there's definitely a sense that we need to not just give education and information to the lawyers and judges but also to other people who might be the people who really turn to this the most.

Speaker C: And look, New Zealand courts are to be commended for putting out those guidelines very early and taking that very educative approach. However, we are still seeing cases coming through New Zealand courts, maybe just not at the pace that we're seeing in jurisdictions in Australia and elsewhere. And uh, whilst those guidelines in themselves are great, I think putting that knowledge management hat on, that history of looking at how people access information, how they engage with it, how they consume it and how they reuse it, if you've still got cases coming through, I think you probably need to look at how that information is being made available to people. Has there been a process of user centered design if you like, um, engaging with users of those materials or potential users of those materials to understand how they consume information. Is the information that's being provided actually hitting the mark? And this is a broader question I suppose about we are in a period of transition. To me this is not so much a technology issue as a changing communications issue. Yes, it's a governance piece around it as well. But it's like we are in the world's largest computer science experience, uh, experiment, as one of my um, contacts said to me some time ago. And I tend to agree we are still in the middle of that. We don't necessarily have all of the answers as to how to find a way through it, but we can draw upon previous experience from other technology and other types of change in organizations, maybe take some of the gold out of there, uh, and apply it to this scenario and that um, for example Alex, that would be one of your primary focus is within the law firm context I would have thought. I think we can apply that more broadly to the use of AI in the courts by more than just lawyers and judges.

Speaker D: Absolutely. And I think it's that phrase where it's something along the lines of you have to tell someone something seven times before it's going to kind of, you know, they're going to remember and it's going to be something that they can

Speaker C: sort of, you know, refer to so much like AI.

Speaker D: Exactly. Like. No, but you're right. It's one thing to kind of have all the, you know, the information and the resources there. But to your point, Vicki, how are people accessing that information? How can we tell them in several different channels or several different ways, how do they want to digest and consume that? Um, because otherwise, you know, you could have the best guidance in the world. But if it's not reaching the people that kind of need to hear about it, then, you know, it really does impede that sort of adoption and that ability for people to engage with the technology in the right way. And I think, you know, even just looking from, you know, the law firm perspective, like within Minters, there are some really great training programs. There's a really big push for sort of learning about this AI literacy and understanding, sort of, you know, the limitations and all those pieces. But you'll still come across people that are like, oh, I didn't know that, or like, oh, I hadn't heard that we were doing things about this. Like you can have the most, you know, comprehensive programs, but there's still always going to be gaps. So it's kind of thinking about where you can meet those people, where they are if they're not sort of, you know, fitting into that um, you know, standard structure or accessing those training programs and things like that. Completely agree. It's a big people communication and change piece.

Speaker B: Yeah. I think that one of the main problems I usually see with these guidelines, especially for self represented litigants, is that whether they ever see those guidelines, like when do they see them, like at some, at which point that even if they know like generative AI has limitations, like access to legal information we know is very difficult and understanding is difficult. And even in one of the US cases which we also discuss in the paper, Court says there are uh, you know, public libraries that give you access to legal databases that usually you need to pay and even I emailed them to check that whether it's true and it's true, but they give you only one or two hours a day access to use, uh, those databases and check the cases. And even if they do, even if they have the time and they want to go that far, whether they actually understand what they're doing. And another thing is in some cases, again, I don't want to, don't want to generalize, but in some cases, some, some self represented litigants, they know exactly that, uh, generative AI, you know, hallucinates. And when they are being told, these cases, you submitted 10 of them, 15 of them are hallucinated, they come back two weeks later with more hallucinated cases. This problem of access to justice has become way more clear to me how desperate people are to say, this is my problem. I really want the court, you know, hear what I'm saying. And the expression about not understanding the law. And now there is a way, at least I get the foot inside the court. I think that's one of the reasons a lot of people are still using it despite the fact. Knowing about the limitations.

Speaker D: Yeah, I think the reference to kind of the desperation or kind of, you know, it's acknowledging that it's probably one of the most stressful and emotional times of their life as well. Being involved in something like this and you're not necessarily thinking straight or having the time to go through everything, you are kind of clinging to whatever, you know, support that you could find. So it would be really hard if you're in the midst of that really intense time to be thinking logically about those things. And you know, to your point, I mean, maybe they can't verify some of those cases because they don't have access to relevant databases and things like that. Or there's the piece that I know is in the paper as well, which is just sort of, again, still a lack of understanding about the limitations around these things or a lack of understanding that if they were to write into, you know, ChatGPT or another LLM in a really, you know, kind of way like, oh, provide all the arguments that proves that X, Y, Z happened. It's automatically biased in the way that you're prompting it and the way that things are coming out. But you maybe, you know, wouldn't know that if you hadn't been trained. So I think it is really hard when you're dealing with people that, that are going through such tricky times, um, to kind of be able to, to sort of have them work through it in a logical way and seek out sources and guidelines for how to do it rather than just run straight to, to kind of, you know, these tools and yeah, as you say, try and get their Foot in the door with the courts.

Speaker C: There's a couple of points I just wanted to address in those comments. Firstly, we're assuming that people can actually read English really well and you do often see litigants who, you know, English is obviously not their first language. They may not be very well educated. And I've often thought, you know, to myself sometimes when I'm trying to go through a particular official process myself, if I didn't have the benefit of a good education and actually being able to understand English really well, let alone having a law degree as well, would I. How would I cope with the materials that are being presented to me to consume, um, let alone respond to them in an appropriate way? So there's that on top of the stress piece because yes, usually this is what, you know, people are going through some of the worst times of their lives. That's how they've ended up in a dispute resolution process. And sometimes it's not through any choice of theirs either and they're having to respond to a process that's been dropped on them by another party. The other thing I would say though, that's slightly more positive. One of the pleasant surprises I've had and uh, I mean I've been working in law firms with knowledge management resources for a long time, so very used to those big expensive data sets as well as all of the sort of rich intellectual property of a law firm, particularly a larger one. And doing the research actually did bring home to me the fact that we have pretty good free legal resources in this country. Um, you know, we don't have those barriers to accessing legal information that you do in the us for example. So, you know, Austli Jade, they're terrific sources but also you've got some great secondary sources when you go looking on the websites for courts sometimes. I mean there's some good materials, particularly for commonly disputed types of areas. Uh, but so, um, the legal aid website's often really good sources of plain language information as well. They will guide people to go and find legal resources in the right locations that are reputable and as are, um, the community legal centres have also invested quite a bit of time on providing resources where people can, uh, legal self help resources is what I would probably call them. So I think all of those organisations should be commended for the efforts that they have put in. And particularly in this country there is a lot of effort to try and help people help themselves before they hit the court registry with their queries. Uh, but the other other place is our public libraries and I Do sound like a knowledge manager when I say this but there actually are some really good free legal resources available from um, the state libraries both in Victoria and New South Wales. They've gone to some effort to organise legal information so it's easy for people to consume. But that said you still have those barriers to actually you know accessing and understanding the information and working out actually how to navigate some of those bigger services that lawyers require quite a few years of training, um, including quite a few years at law school before they actually hit the profession to find their way around. So we're expecting a lot of self represented litigants if we're expecting, expecting them to use all of these services to verify all of their um, content before they put it before a court. And you can understand why AI provides this very beguiling solution or apparent solution and I think it might be a really interesting point.

Speaker A: So this reference to the free legal materials in New Zealand, I think AUSTLI generously helps our nzly our free legal uh, case law and all sorts of legal materials actually. So we've got that, we've also got our legislation obviously online and I just see that in August the Parliamentary Council Office released um, some materials about a study they were doing into providing a chatbot for the New Zealand legislation website.

Speaker C: Ah yes.

Speaker A: And it makes me wonder whether um, we're in a sort of definitely a transitional phase. I don't think there's any question that we're in a transitional phase around AI and launch. But are uh, we in a transitional phase where we're not going to have a hallucination anymore because we will have easily accessible. We don't want to use train, but the AI is pointed at a certain set of legitimate data sources that's necessarily going to get rid of a lot of the hallucinations. I would have thought at least in my um, use of NotebookLM for example with the set of cases and materials I don't see many hallucinations. But then that brings me to a wider question which is maybe this phase is quite good because we actually see the AI use because of the hallucinations and maybe it'll be worse when we don't see the AI use. You might get reference to actually existing cases but maybe the AI doesn't quite understand the nuances of them, doesn't quite get right what the law is or provides an argument which doesn't quite meet what we would expect from a human and doesn't lead to the development of the law in a useful way because it's all just AI generated and that's relied on. So I wonder whether we'll get past the hallucination phase and it'll be a worse kind of phase of the development of AI and law. It's just a big, big idea and haven't quite thought it through. But uh, maybe people have ideas around it.

Speaker B: Unfortunately, I don't think we're going to pass hallucination phase. I think the idea of that hallucination would go away. We should think uh, about it in a different way. We should as Mark put it, like we can reduce the hallucination by providing context. It goes to the point that I think that content could be reliable enough that uh, if a lawyer looks at it for a self representative against before they submit it, they don't need to spend too much time to go through and check everything and they can give them some pieces of advice before they submit them to the court and that that's how I imagine the direction we will go. But I think the hype in the market is ridiculous. I've seen, I don't want to name any of these legal platforms or organizations or companies that the way they phrase their AI products that um, are extremely accurate things like that, they are just to me so bizarre and unfair to many lawyers who end up using these. And because of all the you know, advertisement they take it as true that this is accurate as they are saying it. While we don't know even if they have any kind of benchmark, like if you don't have benchmark, how do you know it's extremely accurate? And uh, they are misleading somehow people, despite the fact that they always say oh you got to check the M material when you say it is extremely accurate, people over rely on it.

Speaker D: There's kind of a two pronged danger there. Right. Which is one of the things around sort of in the paper, the verification drift where people stop you know, scrutinizing the output as much or there's sort of not as much critical thinking associated with, with the outputs. But then there's sort of the other piece which is like you've been promised, you know, all of this amazing, um, you know, there's all this hype around what it can do and how accurate it is and all of these things. But the moment you pick up something wrong in the outputs or something that's not sort of quite right, that does also erode trust. So people that were open to using you know, those tools and you're looking to get them to adopt those tools, when they do see the Limitations because they've kind of been brushed over what those limitations are. Then there's that drop off and they're like, I'm not going to use this because I can't trust it. And so if you are trying to get people to adopt it in certain settings, if there's too much hype at the outset and not enough acknowledgment of those limitations, then I think, you know, that presents a danger as well. Then on the flip side, people get complacent, um, you know, and they, and they trust it too much. So I think that there's kind of issues on both ends of the spectrum.

Speaker C: I think going back to points that both you just made, then Alex and Mark made earlier, there is a risk. And we know, I mean, there are other studies that have shown this averaging effect that AI use has on, um, groups of professional knowledge workers, if you like, where it benefits people who don't have sort of a high level of expertise or were sort of the lower expert at the lower expertise end of the scale, brings them up towards sort of this middle level, if you like, but it also brings down the performance of your high performers. I'm just wondering, uh, if in that process of relying more heavily on AI to prepare for court, um, and to do legal work, are we losing some of the richness and the variation in human input that's not assisted by AI because we're relying on tools that are giving us something that's pretty good, even if it's not quite as intricate or thought through as what we might have had previously without the assistance of the technology. And uh, you're probably seeing this with your students, Mark and Armen, um, as they're preparing for assessments, you see this lack of, I guess of richness or lack of variation probably, and averaging out of what's coming through, um, the greater use of AI in that context. And if you start to lose, it's a bit like when you go to the gym. If you don't keep working out and lifting the weights, you're not going to build the muscle. There's actually some work involved. You can't just, um, have a look at the exercises on the gym app and hope that's actually going to give you the muscle that you're after. So I don't know if that's an analogy, um, that resonates with people. But, uh, yeah, it's that, it's that averaging out, that tendency towards mediocrity potentially, that's what concerns me in some respects more than this period that we're going through even though hallucination is a serious issue. But uh, are we losing those opportunities to build expertise, to build knowledge and have that richness flow through into the services we provide? And the legal thinking that comes through, um, doing things the hard way, does that make sense?

Speaker B: Absolutely. Um, I think, I think there are two things to consider what you said. One is that we usually say AI going to help us to get rid of those tasks that don't need high level cognition so we can spend more time on those difficult and challenging tasks and be more creative. But I think it has had a reverse effect on a lot of people. And because XFL into AI, how to do the simple but repetitive task is way more difficult and requires usually clicking on different things. Current AR models are very limited in that regard or very expensive if you want to use it. And another is I've seen in my students when I allow them to use generative AI, they just stick with the, you know, a draft that AI has generated and the first rep looks good, but it's not good enough if they work on it, they get stuck with the direction that the AI has provided. So as you said, that reduces the creativity. And um, because AI models, and I know it's controversial but majority of them, they are not creative, at least text models and they cannot address new, uh, the problems that no one has ever solved before, they usually can address problems that we already solve and it's somehow in their you know, training data set. So yeah, there's kind of convergence in terms of ideas and we will have less creativity unless we can create that next level of AI models that are creative as well.

Speaker A: Yeah. And I think that links to a, uh, question I have. When I look at judges saying, look, AI is quite useful, I'm going to use it. I'm a little bit more optimistic about whether AI can be used in a creative way if you have a deep expertise and if you use it as sort of a sparring partner and a, uh, sort of advanced search tool and to kind of as a brainstorming exercise. Because I find that, you know, if you just put in uh, the, the legal submissions of each side and just said write my judgment, well, that's obviously going to lead to that kind of mediocre, flattening, averaging effect. But if you are more creative and if you start to asking, you know, say well, I know this case here and I think there's this really interesting line of argument in this article and start uh, brainstorming your own ideas about where this might go and ask the AO how does it relate to the pre existing case law, to the submissions that have come in. And so long as you then think yourself and apply your expertise, that's fine. But the problem with the whole hallucination thing and with um, you know, uh, lay litigants and with law students who haven't developed, haven't graduated law school yet and developed a uh, deep legal knowledge, it's very hard to do that yet. You sort of have to wait till you've done some of that fundamental learning.

Speaker B: I think it's way more difficult than it seems. And we have only a few minutes. I just want to go back to what Alex mentioned because we didn't define verification drift. The idea of verification drift that we discussed in the paper is that unlike what a lot of people used to claim, that people don't have AI literacy and that's the reason they are, you know, submitting unverified document. Verification drift is about people know they are supposed to verify but due to various reasons they don't. Those reasons include that the content is so authoritative and credible looking that people don't think they need to verify. And we usually assume lawyers or self represented litigants who submit these kind of documents, unverified documents, they use it as a one off thing. But I think even if you verify the AI content and you use it frequently at some point, again because it looks so credible, sometimes you're like, okay, it doesn't make sense if I want to, you know, verify this thing as well. I've uh, been verifying that everything is correct or it's, you know, rarely ever, you know, make mistakes in this area, in these areas. And of course we also know about the time constraints that um, like I don't have much time, this looks good. And we see sometimes lawyers specifically say that yeah, this seemed, it read well, something like that, that they didn't find it necessary to verify it. And I think the, the fact that if people using it frequently, um, not very verifying becomes so encouraging because they know how time consuming is to verify, um, it becomes kind of more common thing. And that's what I found so difficult to teach students to verify the content despite the fact that I taught them, despite the fact that we practice, despite the fact that I showed them closest that how easily it hallucinates. And in my current study, students who wrote a reflection on their use of generative AI, a lot of them said that I found using AI for legal research inefficient because the time we have spent to verify the Content takes way more than doing the traditional research. And, uh, to me, that makes sense. I never do, uh, research regenerative AI, because verifying every source, every bit of the content that is generated by generative AI, that I'm not 100% sure or correct, takes extreme amount of time. And that is usually left out. And that is usually. A lot of people we talk about AI, they say, oh, go and use it for research, but you got to verify. But people won't because they realize how time consuming it is.

Speaker D: Sorry, the conversation's moved on a little bit since we were talking about this, but I just wanted to add to the points before. Vicky, to your point, around sort of, you know, what. What might we be losing in terms of, you know, some of that critical thinking or sort of that creativity or individualized, you know, opinions about things. I thought it was really interesting. I was at a conference a few weeks ago, and there was that reference to kind of those that sort of, you know, have the expertise. To your point, Mark, you kind of are, uh, sparring with the AI and you're sort of, you know, bringing your lens and your experience and all of those things to what the outputs are. And then how sort of some of the younger, more digital natives, uh, you know, tend to sort of go to the AI first, which is to your point, I mean, around sort of. Once they, you know, go and generate an argument or generate sort of, you know, a structure for an essay or sort of putting that together, they're kind of tied to that because they. They haven't done the independent thinking around sort of. What do I think about this? What is my opinion in this particular context? Um, but I thought the way that they framed it, the conference between those that use the AI to refine and test and brainstorm, sort of, those were the people that had the experience and had been in the industry for years versus those that were sort of quite new. And again, it's kind of telling you what to think, in a sense, also, you're going with what it's been generated. But I think the key danger there is how do we, you know, if people are going to AI first to help shape what they're thinking and what their opinions are and how they're going to sort of, you know, argue something, how do we then ensure that they end up people that have years and years of experience and expertise to test it later down the track? It's like, what is the training and development piece here? Because if they're not that ability to kind of think first and reason and use the logic and then sort of, you know, there's plenty of amazing ways you can use AI to do that testing. But if people are repeatedly falling back on that, what kind of habit does that develop and what kind of risk does that pose in terms of, you know, maybe some lazy thinking or people not being as critical about what the outputs are. And I think it is hard when you don't have the experience to look at it when it does have such an authoritative and you know, very sort of um, you know, impressive tone. Sometimes it's kind of hard as a, you know, someone without experience to go, oh yeah, it sounds good but it is lacking the depth in this particular area or that's not arguing this point or you know, I think when you've had a few years under your belt you can read it and say, oh this is really missing something. What is this actually trying to tell me? It sounds good but there's no real substance behind this. How do we ensure that those new lawyers are developing the skills that they need to become those expert professionals?

Speaker A: I'll just say that um, just comes back to the idea that hallucinations are at least a good warning at the moment. You can't trust it. If we lose hallucinations people will have more of a, automation, a ah, verification drift, automation prejudice. So maybe they're doing something good at the moment.

Speaker B: I mean quickly I'll add something about what Mark says that as hallucination uh, rate reduces the problem of verification drift increases significantly because it becomes so trustworthy that a lot of us would trust it. This is interestingly something that was raised back in 2023 by OpenAI when they released GPT4, the model after uh, ChatGPT, the first version of it in their technical report it specifically says that we worry, we are worried as the models get better over reliance on these models would be more and we believe that's going to be even more significant of a problem. So yeah, Mark, what you're saying, I think it's make so much sense that it could be a very good warning for us that how we should learn to deal with this problem because it's so much a big pain point that we want to address it before it becomes less significant in our eyes because it doesn't um, obviously that much go on Biti.

Speaker C: Sorry, I was going to say yes, we need to be having active dialogue around how we use AI to enhance our skills and build our skills rather than focusing on, as a lot of the dialogue does at the moment, on efficiency and cost savings. I think the debate is skewed in the wrong direction. If we want to retain the richness of knowledge that we have and the systems that we've spent many, many decades, if not generations developing so that we have that sort of layer of creativity that we can bring and that richness of thought processes that we can bring to legal practice. So, uh, the hallucinations are a wake up call. But yes, the dialogue needs to shift, I think, so that we're actually debating things not just from that efficiency and cost saving perspective and making investment decisions accordingly and changing our processes around and our workforces and our society. So.

Speaker D: Agreed. It's like, what's the ultimate goal here? It's like, yeah, you're kind of, you know, pushing for these productivity gains or efficiency gains, but what are you losing in the process? And the practice of law is an innately human practice and it requires ethics and kind of an understanding of human behavior and it's kind of what, what are we gaining from outsourcing some of this thinking and, you know, some of this reasoning to AI? You know, it is, is it worth it? Is that productivity gain worth it? Um, and yeah, how come to your point, Vicky, how can we reframe it to make sure that we're using AI for the right aspects and sort of, you know, having the messaging around, you know, the real benefit, rather than just focusing on that efficiency piece? Because I don't know if the scales necessarily line up, if some of that efficiency gain is worth losing some of the, you know, the critical thinking, knowledge and expertise that exists within the profession. So keen to make sure that there's some structures and sort of governance in place to make sure that we're not losing that richness.

Speaker B: As we are getting towards the end of the, um, podcast and we are all enjoying this conversation, we should finish it here. Um, Ricky, would you tell us, like, what, what are you doing these days and what the center is doing and upcoming plans?

Speaker C: Okay, so in the next, um, month or so, what we're planning on doing is finishing off a report, uh, on the data set that we've prepared for the Australian Academy of Law, which will be made available on their website as well as most likely through the Center's website, either, uh, late this year or early next year. So if anybody's interested in having a look at the, some extracts from the cases all pulled together and some of the statistics that we've extracted out of those and the patterns that we found, uh, in compiling this data set, uh, then that will be made publicly available soon.

Speaker B: That would be very interesting report to read.

Speaker C: Thank you.

Speaker B: And. All right, I think that's it. Uh, thank you so much, everybody. I'm just going to wrap it up here. And that's the end of the podcast.

Speaker C: Thank you.

Speaker B: I do a horrible job every single time. I don't know how to do it.