Theodore Christakis: chatbot privacy dreams and the health AI agent rush

Speaker A: M foreign.

Speaker B: Okay, today we have Theodore Christakis here again and once again we'll talk about the intersection of AI and privacy. But this time we're putting aside the training data challenge and mostly focusing on the use of personal data at inference level and the manner in which such status or conversations with chatbots can be stored, exploited, retained, abused and more. So today we're going to be, we're going to do something very ambitious. We're going to look at our guests recent research work which is very thorough and it's a lot and we're going to try and summarize it. The first or the main piece um, of research is titled you trust your chatbot with everything, should you? And it has two parts. Part one was released in March and included thorough uh, analysis of how the main and comparison tables and how the main AI labs use conversational data. And the second part was released last week and it is about the consequences of this if you look at the outside world, so through law enforcement, data breaches and more, the consequences that go beyond what the AI labs are doing. And in fact the title for the subtitle for this second uh, part is Government courts and the battle over your chatbot conversations. In total these two, you know, pieces of research amount to some 175 pages which is not bad. So bear in mind we're going to summarize that or try and that's not even all because we have another

Speaker A: document,

Speaker B: um, and another you know, piece that Theodore uh, released in between those two parts which is called the Health AI Agent Rush. So yet another document that we're going to be discussing and this one deals with something we did touch on during our last interview on the Spanish channel which is the fight for the interoperability of health records, the manner in which AI can leverage such records to make our lives easier and in general a, ah, more efficient use of health data both in our personal sphere and as a social value, um, and social service. If last week we covered the European health data space with Mike El Recuero as a sort of top down approach to what I've just said and ended up wondering how much that approach would could resist the push from the consumer tech and individual empowerment side or a bottom up approach. In this study, uh, Theodore has been looking at developments in the US and realizing that we need to find a way to make them somehow compatible with each other if Europeans are going to enjoy the safeguards they expect. In any case, we'll add links to all of his works on airregulation.com. that's where you'll find them. But again, we'll have the links. And as for our guest, Theodore Christakis is a Professor of International European Law at Universitie Grenoble Alps in France, Director of the center for International Security and European Law, Director of Research for Europe with the Cross Border Data Forum, Senior Fellow with the Future of Privacy Forum, and a former Distinguished Visiting Fellow at the New York University Cybersecurity Center. He is also Chair of the Legal and Regulatory Implications of AI with the Multidisciplinary Institute on AI and has been a member of the French National Digital Council and today serves as a member of the French National Committee on Digital, uh, Ethics, as well as a member of the International Leader Transfers Experts Council of the UK Government. So there's a lot coming up. Let's go for it. Theodore, thanks for joining me again.

Speaker A: Well, it's a pleasure to be, uh, here, Sergio. Thanks for having me.

Speaker B: My pleasure. And the last time we spoke, we went through the good, the bad and the ugly, um, in the EDPB opinion, um, on LLMs ingesting personal data or, uh, using personal data. And now you have. I don't know how you do it. You've written so much to bring this to the next level. I'll say. So this is a larger project on the privacy of our exchanges with AI and AI chatbots and so on. And the first thing I need to ask you to at least please satisfy my curiosity is the COVID right? You clearly love Magritte because, you know, you use it a lot. So there's a piece of, uh, derivative work. We'll say where you are using the man in the bowler hat, uh, but his face is a legal folder with, you know, the Exhibit 8, a, um, sort of sticker. Why would you do this to us, please.

Speaker A: Yes. Thanks, Sergio. And it's good to start with a little bit of art. And, uh, you know that all the studies that, uh, have been published on the confidentiality of chatbots that have been published on our, uh, website, AI, uh-regulation.com um, have a cover which is, uh, inspired by, uh, Magritte. And, um, uh, of course I love him, uh, a lot. This one, the part two, is his famous son of man. But, uh, uh, as you said, instead of having the famous green apple in front of the face of the man, you have, uh, there above legal folder tied with a red ribbon stamped Exhibit A, uh, with, uh, chat transcripts, uh, spilling out of the top. Somehow I have the feeling with using chatbots, you know, have all become that man, our private selves, uh, into a chatbot. And uh, the intimate face we think we are showing can uh, quietly become someone's, uh, else. Evidence. And this is what part two, uh, is about. And this is why I use this analogy.

Speaker B: Okay.

Speaker A: I hope that Magritte will uh, forgive this small theft.

Speaker B: Yeah, well, or he's a statement inspiration. Yeah, yeah, that's for another day as well. I don't think we've. No, not at all. 70 years have passed and we'll see about that one. But look, let's move on. So then this is part two of a larger project and we. And you, you have a part one. So what's. What was part one about?

Speaker A: Yes, the, the whole project, uh, asks a, uh, simple question. You know, we are, I don't know about you and about uh, the auditors. I guess this is the case. But around me I have plenty of persons, including starting with me, who use uh, chatbots, uh, all the time for very uh, confidential uh, things, uh, including health issues. For instance, uh, earlier we have uh, chatgpt tell us that, um, OpenAI tell us that, uh, there are 230 million users who ask every week questions about health and well being, uh, and all these confidential things. And you know, I asked myself how all these interactions with the chatbots are protected. So the question is, uh, uh, at the beginning, uh, every time of part one and part two, you trust your chatbot with everything, should you. And part one looked at, uh, what I call the internal boundary, what providers themselves OpenAI, anthropic, uh, Google with Gemini, but also Deep Seq or uh, X, uh, et cetera. What do they do with your conversations, uh, behind their own walls. And then part two turns to the external boundary, which means, uh, what happens to our interactions, uh, when they get out of the building somehow into the four places, uh, they can uh, pass out for control. And we'll be able to discuss this uh, today. And what is very interesting is that exactly there is some convergence, which means that the same futures that make a chatbot very useful, that it remembers you, that it um, keeps uh, your history, personalization, uh, plugs eventually to your mails and your files as we're moving towards agentic AI, all these futures are exactly the same that make uh, uh, the resulting record preservable, searchable, discoverable and even, we're going to talk about it, stealable. So the same stored conversation is uh, at one and the same point, you know, uh, a candidate for, or a police referral or a target for Government demand or an evidence in a lawsuit or an asset in a data uh uh breach and uh, as far as I could find working on this project for crazy project, totally crazy project for uh one year now there was no study which was uh examining uh uh this uh in a comparative way. And uh this is why I undertook this project and uh, just uh uh uh a few days ago I published the uh second and last part of this series.

Speaker B: Yes, yes. Okay so before we go into that one into the outside um let's look at the insight. So at part one which was about the providers and what they do with the chats. And it's true that last time we spoke we were all speculating and now we know more plus you've digged a little or a lot and build even comparative tables and so on. So what did you find specifically in terms of what happens with the data that we share with our chats?

Speaker A: Yes, the last time we talked uh Sergio it was mostly about um uh web scrapping about how all these generative AI models are trained and to have had a great uh. We had a great discussion also about the DPPB opinion and all these things now. And I think that a lot of the regulators keep working on this which is an important question but we should not neglect the elephant in the room today. All these models have been trained and uh, uh we use them every day. There are just uh OpenAI has 900 million users uh anthropic and CLAUDE are uh also um developing um in an exponential way very very quickly Gemini all these models not to talk about the Chinese mod and uh uh basically part one asked uh how these companies which provide how do they use our interactions which we feel are very confidential but how do they use them? And I found interesting things. Let me limit uh myself to eventually to three uh that uh first that uh they all train by default on our conversations on our interactions so they use our interactions for improving the model. And of course people like you, me many of our auditors who are privacy professionals will go and will search and will opt out from training because they don't want uh for any reason. But uh the problem is that uh uh the burden on switching that off uh uh sits with you uh and uh we privacy professionals will do it but ordinary people might not do it. The second issue that I found is that there is uh what uh I call the feedback trap which means that you uh know when you try to beat into Claude and all the other they have a thumbs up, thumbs down. I was always very curious why they're doing this, you know, because we're not in a social media here to do like. And some people have that tendency from social media, you know, when they're very happy with their response to do thumbs up or where they're very frustrated and agree with the chatbot to do thumbs down. When you do this, this will automatically uh override your privacy preferences. Which means that uh uh despite the fact that you have opted out, it will keep not only the specific interaction but the whole thread. Imagine you have a thread for two months about a health issue and suddenly you did thumbs up. And it will in some models, not all of them in two of the models that I have uh examined. So I think it's a kind of ah, dark pattern. Another interesting finding goes for instance that uh what I call the price of convenience uh which means that um, I don't know about uh our uh auditors and you but personally uh when I use these models I very often I want to be able to come back and continue the conversation and um, so to have the history preserved. Otherwise I will use the cognito model or whatever. But when it comes, for instance you have a health problem and you want continuous uh advice or how to interpret the new test that you have. You want to keep the history, you don't want to reinvent the wheel, uh start a new discussion all the time. You're happy that the model uh is following this. And the problem is that uh for one of the models that ah I examined uh in order to be able to do this uh, you have to accept that it uses your conversations for training otherwise you cannot keep the history and find this extremely frustrating. And uh, uh I don't know why this specific uh model uh is doing this. The other things that I discuss a lot in part one is uh the issue of what I call the monetization uh problem. Because uh using the chatbots requires uh a lot of uh energy costs, a lot of money. And so there is of course a strong pressure uh you know that uh, uh we will have very soon the ipo. So we just had on Friday, last Friday the IPO of SpaceX and we uh have uh also the IPOs of uh OpenAI and uh anthropic coming very soon. And uh, there is uh also uh, there will be a business and investors pressure to monetize. To monetize. So uh, of course you monetize through subscriptions, through enterprise offering and business and things like that. But it's not enough subscriptions. For instance uh, I read somewhere that for OpenAI, it's only 5%. So if you want to offer, uh, also ChatGPT to the remaining, uh, 95% of people around the world, uh, how exactly do you do that? Want to monetize. And I respect this. And for instance, uh, OpenAI has launched, uh, advertisements in advertising in the United States. And they want to expand this. Let's see how it turns out. If you imagine that there is a lot of issues that I have discussed in my study, the others have not, uh, followed this path for the timing. And the last thing is open operational sharing. There is a lot, of course, all these companies sell. We're not going to sell your data. This is very, very good. Of course. But not selling does not mean not sharing. And there is a lot of operational sharing which creates, uh, some privacy issues that I will not discuss here, of course.

Speaker B: Okay. And then there's the expectation that we have. So, you know, recently the California Bar issued this warning to everyone, to, you know, all the lawyers, that under the rules of professional conduct, for example, people were dumping all of these sort of confidential information, privileged information, uh, on their chatbots. And they were saying, be careful with this thing because you think it is covered by privilege, by client attorney privilege. And if you are not looking at a very specific environment that you protect and is isolated, then any generic chatbot is not going to be. Whatever you throw in there is not going to be covered by client attorney privilege. So be very careful because you cannot promise this. And as it happens again in the practice of law, if you extend it to everybody, if a lawyer can be deceived as to the expectations of privilege, imagine everyone else. So how wrong is that perception that we have that this is privileged?

Speaker A: I think let's start by the fact, before talking about lawyers and, um, attorney client privilege and work product doctrine, let's start by something else more general about consumer AI. And let's start by, um, saying that people speak to the systems in the register. We often reserve for, uh, a doctor, a therapist, a lawyer, as you said, can be a priest, or no one at all. You know, the things that you wouldn't even say to your best buddy that you keep on your head. But somehow as the interfaces feel very, um, uh, private, it's just you and the interface. There is no human judgment. And, um, the interface are built to invite exactly this. And the chatbot, some chatbots, are very emphatic and they want to help you and do things for you. So sometimes you say even things that you will not say to a friend and uh, what is interesting is that if you see the positions of the companies, uh, they acknowledge this. OpenAI, for instance, will talk about, uh, the big New York Times litigation in the United States. And there they use the argument, and they said that, uh, as a matter of fact, this is among the most sensitive information in your digital life, what you're doing with ChatGPT. So the company shares you this intuition, but, uh, the law does not, uh, in. In law, chatbot conversation is an ordinary business record held by a third party. And, uh, it carries none of the protections that attaches to. To the very same words in the consulting room, room of a, uh, therapist, for instance, or in a lawyer's office, as you said. So there is no AI privilege. Some Altman, Sam Altman, the CEO of OpenAI, called, uh, for one, and he has said that talking to AI ought to be confidential about these issues in the same way as with a doctor or a lawyer. But, uh, uh, as a matter of fact, uh, wishing does not make a privilege. And, uh, it is very clear that there is no AI privilege as such, neither in US Law nor, uh, in European. Uh, but if we go in reality to the question you raised at the very beginning of this discussion for professionals, uh, the big issue is that you are talking about, uh, uh, attorney client privilege, which exists, of course, in law. This exists in all legal systems. But if you use an AI tool, what happens there? Are you going to forfeit the protection of your own preparation, uh, that you would normally enjoy if you hadn't used, uh, such an AI? And for these, there are, uh, important, uh. There is important case law, very recent case law you have heard, without doubt about last, uh, February, just, uh, three months ago, uh, the Hepner decision in New York. What happened in this case, it's very interesting. Mr. Hepner was indicted for securities fraud, and he had hired, uh, lawyers. He was working with his lawyers, but then at his own initiative, he started using Claude in order to improve his defense. And he was saying very confidential things to Claude, and then he was sharing with the lawyers who were putting into the filings, etc. But when the FBI searched his home, uh, on his, uh, arrest, it saves the computer, it found these interactions with Claude, and it used, uh, them against him. So, uh, the big question there was, is this covered by what we call the work product, uh, doctrine, which means, is it protected, uh, and, uh, preventing the FBI of using it against Mr. Hepner? And the judge said no. Why no? Because the chatbot is not a lawyer. The exchanges were not Confidential given the fact that the provider anthropic says in their privacy policy that uh, humans can access it and it can be disclosed uh to law enforcement and all these things. And finally that the work had not been done at the direction of the council. This is interesting this exception by the way, because you imagine that if somebody, a rich guy has clever lawyers who tell him, who instruct him in an email, use for instance uh Claude or ChatGPT, uh in parallel with us for your defense, then it might be protected. But poor people having lawyers, uh who do not think about this, uh, they will not cover uh Anyway, uh, there is another decision Warner the same month, uh where uh, a different federal court reached the opposite conclusion. And I uh found this reasoning by the way more convincing, saying that an AI system is a tool, not a person. And uh, so waiver requires disclosure to an adversary and using a chatbot to think no, uh, more waves your protection than using your own word processor or search engine. And uh, I uh think that uh, uh, this is um, uh more uh credible. We also have what I discuss in my study, the cloud storage analogy, which means that when clients use Google Docs to draft a document or Gmail to send an email or icloud to store materials, nobody ever said, I think that they tried at the beginning but it never worked. And nobody uh, no court has held that drafting a privileged memorandum in Google Docs destroys the privilege simply because Google terms of privacy, for instance permit data retention or disclosure to law enforcement. So uh, I think that uh, there is a lot uh to wait and see and uh, until that moment I think that people should be very careful and uh, uh, for instance um, address the head and have your lawyer send you an instruction to use a chatbot. And at the same time you can always save the interaction in a word document and uh, delete, don't do data retention. You know, you, you should do things like that that will preserve the work product doctrine while waiting for more clarity from the case law in this field.

Speaker B: Very good. Because now you've connected also part one with part two this. So in uh, the meantime, um, how do people protect or what stands between people's conversations and these risks?

Speaker A: Yes, exactly. I think that uh, for all the things that we discussed today we're gonna back, come back later I hope to technical solutions also and encryption etc. But while waiting for ordinary people using the light to data GPT and all these things, I think that uh, we should be careful with uh, data retention. So you should delete uh, or using cognito mode for instance. And so um, the habits, it's a matter of the habits of the users but also it's things that, that the providers themselves can do with limited data retention and minimization. When I delete a specific conversation in ChatGPT, I have the expectation that it will be deleted, not retained, you know, for, for m. A long time. So I think that there are a lot, I make a lot of recommendations about this uh, in my studies and I examine the data retention of each one of these providers.

Speaker B: Yes, yes, now that's super interesting. Very, very interesting. Yeah, totally. Now moving on a bit farther into the outside. So there's these four doors that you talk about and the first one which could be uh, surprising is that the very provider that you're using, so an anthropic, an OpenAI, they decide to call the police. How is that, how does that happen?

Speaker A: Yes, absolutely. Well this is very similar to what happens with social media moderation. So uh, companies, both social media companies but also all these companies have created safety departments, um, with some, um, exactly. With some very good uh uh, uh people working on this, uh, a lot of experience working uh, um, on issues of uh, safety with digital platforms. And uh, they try, it's a very good thing, they try to, to find out if there is any abuse or is there any danger created by the use of their tools. And so companies, what they do is that they scan for alarming content and uh, in defined situations they can refer a user to law enforcement. So the mechanisms exist for things that are uh, uh, I could say horrors like uh, for instance threat to others. You know, if you are trying to use uh, I don't know, ChatGPT or Cloud Tool to make a bomb or Gemini to ask uh, how to get a gun to kill people. You know, uh, if this is detected automatically then it's scaled to humans for review and uh, the companies can decide uh, to refer this to law enforcement or in the same way for suicide. Suicide at times, you know teenagers have heard a lot of stories, um uh, if, if they detect a danger of self harm this time these uh, chatbots can on the one hand provide uh tools uh, for, to this person in order to, to get help but also eventually uh, uh, take action in order to prevent a very dangerous situation. And in my study concerning harm to others, for instance, I refer to three very recent cases which are extremely interesting and important. Um, the first is the Tumblr Ridge case in Canada, the Florida State University shooting and uh, much more recent uh um, uh, armed police Intervention in Strasbourg. I will not detail them here because I don't have the time, but they're extremely interesting. Uh, especially, for instance, it's very interesting that in the first case, OpenAI detected a threat. They discuss it. There was human escalation. They discussed it, and they even suspended the account, blocked the account of the user, but they did not refer it to law enforcement. And now they have. They are accused of not preventing what became a mass shooting in Canada. And, uh, they have, um, uh, they will face lawsuits of, uh, billions of dollars. Uh, while in the third instance in Strasbourg, it's interesting because they're having also, uh, put the bar lower. Uh, you know, they detected someone who asked ChatGPT, uh, how can I get a gun in order to kill a Mossad agent or a CIA agent or a, uh, uh, French, uh, agent. And, uh, they detected this. And just a few hours later, the guy had, uh, read the antiterrorist brigade, which, uh, came to his house and, uh, arrested him. Finally, there were no charges. And he said, I was trying to test the system. He had psychological problems. You see, what we have here is the legal plumbing of what we call, uh, something classic, the emergency disclosure framework, often used in social media moderation in similar way. But this framework was not really designed, I'm sorry, for a system that holds, uh, a rolling and intimate and frequently ambiguous monologue. Ah, Judged at scale by automated classifiers with a human looking only when something, uh, escalates. So this is, uh, the issue. Uh, the thing is that, you know, if you come back, uh, if you want to post on social media something to call for violence, of course, this is very clear. But, uh, if you go to the chatbot and you say, uh, after a hard day's work, you know, I want to kill my boss, like, you will say this to your spooz when you come, you know, your spoons will tell you, okay, you know, have a drink, calm down, you know, uh, and, uh, she will not refer you to the police. You know, it's not. You're not serious. You're just venting. So this is, uh. I think there is a big difference, uh, between the two. And, um, so somehow we need to think much harder on these issues that we are already doing.

Speaker B: Yeah, this is not scalable and it's not sustainable because if you picture now a local model and you chat into your tool, it's. It's the same distinction you were making earlier with the court cases, where one thing is, you're assuming that someone is listening all the time. And There's a third party. Another one is that you accept this is a tool that eventually could be used within your personal sphere. You could be chatting to your Google Doc, in fact, and just dumping illustrations on a document. And that doesn't turn it into your psyched, your confidant that uh, has to report everything you say to a Google Doc and you're going to have local models where you will, again, you will just express your frustration. I don't know. So, yeah, it feels like finding the balance because now we rely on their criteria, on their judgment. Uh, I mean, how can you rely, or expect to rely on the judgment of OpenAI to decide where is the threshold for reporting?

Speaker A: These companies are really trying to do and to improve and to do their work. Uh, and they uh, have, uh, very good people. But, uh, it's a hard thing and um, uh, precisely. Um, and let's also say from the outset that it's not just these companies. I mean, it's normal for OpenAI when you, after all, when they have somebody who say, how can I get a gun to kill, uh, uh, an agent of the state to refer, but then the police must do their work. If somebody comes in a bar and he says, you know, I heard two people speaking and the one was saying, how can I get a gun to kill my boss? Or whatever, a Mossad agent, uh, then the police will do their work. They will not arrest him immediately. So there is one thing is a, uh, referral by the companies, which might be perfectly legitimate in some cases. And the other thing is, uh, uh, the fact that the police must really do the work. But going back to the provider, my, my concern is that the entire mechanism operates out of public view. We do not know the criteria, we do not know the volumes, we do not know the error rate, the false positives, the false negatives. I don't know how often, uh, the classifier flags, uh, the novelist, the person venting, or someone asking a dark question for an innocent reason. So in the field of, of course, in the field of social media moderation, there have been plenty of discussions and there have been also some, uh, big steps. The oversight body of Meta, for instance, you know, we have nothing equivalent for chatbots. So you cannot fully support the mechanism and still insist on being, uh, able to, to sit. So you need somehow, and this is why what I suggest urgently is more transparency reporting, publish how often this happens, under what standards, with what outcomes. So a safety mechanism we cannot see, it's hard to trust, gives the impression that there might be surveillance all the time, and, um, it's impossible to improve. So only transparency can help us, I think, find solutions to this.

Speaker B: Okay, so then the second door, which is, you know, the warrant, um, you know, the who asks the for this, which sounds very creepy as well. So how is this, please, door to you that you describe?

Speaker A: Yes. So the second, the first door was that the provider himself will refer dangerous situations to law enforcement. The second is different is law enforcement, uh, doing what we call compelled access. Exactly. The classical thing where police will come and, uh, use a subpoena, court order, a warrant, as you said, uh, in order to get, uh, uh, data, uh, from, um, either metadata or contact data, real interactions from chatbot providers. And so I discussed this extensively in the second chapter. I will only give this highlight about what you said is the risk of what we call the reverse prompt warrant. So a normal warrant is tell us what the specific suspect. I have a suspect. Tell us what he said. You know, we try to establish that he made conversations in relation with a criminal enterprise. Uh, a reverse prompt demand is the opposite, which means that, uh, uh, uh, for instance, tell us who typed this particular prompt, who typed, uh, a prompt about, uh, um, anything that could be of interest either to law enforcement or to intelligence agencies. And you see, as a matter of fact, this is no longer a thought experiment because the study documents the first known federal reverse prompt warrant served on an AI company. In this case, it was specific, which means that another cover agent knew exactly what, uh, a suspect, uh, asked, uh, this, uh, chatbot, and they asked the company who. So they had the prompter said, who is the suspect? Please tell us. And they found the suspect. But you can imagine what is the risk, which means that if we start, uh, asking AI companies, uh, to do some kind of bulk research, who asked questions about this, you know, uh, uh, and, uh, we can ask, uh, things that could even, you know, in some authoritarian states, you know, who asked, or you can ask or who asked questions in order to evade, uh, sanctions, you know, by the United States or. I don't know, uh, you can imagine that this could be very far, uh, reaching. And this is the big risk has not materialized in the field of chatbots. But, uh, uh, we are expecting right now, as we speak, all this. Of course, it's not new. It's new in the chatbot context. But we already had a. Have a huge discussion, as you know, uh, Sergio, in the United States, about what we call geofence warrants, uh, which is, um, ask a provider to identify everyone whose phone was near a crime Scene or also reverse keyword warrants in relation with, for instance, Google search to identify everyone who searched for a particular term. And now we have a very big case that uh, everybody knows in the United States, all previous professionals, which is called, uh, chatri. Uh, it's like chatgpt. Chatgpt. No, it's not about chatbots. Uh, uh, it's precisely about these other cases. But the Brennan center warned the Supreme Court of exactly this, that uh, uh, the warrants might authorize. If they accept this practice about geofence warrants, then we will authorize eventually to identify everyone who issued a prompt to an AI chatbot. And uh, uh, from this point of view, there is a real risk because, you know, prompts and interactions with ChatGPT are far more detailed and revealing than for instance, a search query. In search query, you put only a term. In the chatbot, you put the whole story. And uh, uh, from this point of view there is a real risk and not to talk. Another layer of concern is memory. Or these chatbots, in order to be useful, have to know arms better than ourselves, you know, and they are doing our memory and respond to us. Yes, Sergio, you should do this, uh, like you did, uh, uh, in this case, uh, before, you know, they know everything about you. So could, uh, government authorities access this memory? And what about the agentic futures? Because they will start connecting with everything in your computer, your emails, everything. And from this point of view, this creates another big risk of disclosure. So somehow we need to think proactively. There is little that the providers can do about this. It's not, uh, contrary to other places, there is little that the providers can do about this because this concerns more, um, uh, you know, the law of, uh, uh, compelled access. But I think that we should definitely start thinking also about this new dimension. Taking into consideration are often very private interactions with, uh, these chatbots.

Speaker B: Yes. I found it so creepy and dystopian. You know, this is terrible. So let's say I hope we can really exactly do it. Yeah. Okay, so then there's door three. And door three is, uh, you've called this thing, I mean, based on the number of people in that lawsuit, so 108 million strangers in a lawsuit. And this is the one thing that I know has prompted Sam Altman to become such a defender of privacy. Uh, it sort of helps, uh, him in this case. But could you please explain this? Door three?

Speaker A: Yes, it was this case. The New York Times, uh, were trying to show that, uh, uh, chatgpt was, uh, eventually, uh, accessing New York Times or other editors articles, uh, and giving it to users, uh, despite the payroll. And for this uh, they wanted to get uh, as much evidence as possible, especially using existing interaction with chatbots. And uh, uh, there was a long litigation which still going on and at the beginning a court first order a sample of 20 million ChatGPT conversations to be handed to the other side's lawyers. And then um, just uh, uh, three months ago, ah, this uh, has grown to 108 million conversations in total. And uh, so this is uh, interesting because none of the probably our conversations are among them. You know, none of these conversations belong to the New York times or to OpenAI. They belong to users, to ordinary people who are total strangers to the lawsuit and who never were never asked and uh, will never be told that their uh, interactions have been uh, accessed. So you see, uh, you type your symptom, your financial problems, your secrets into the box and these words might end up uh, uh, in a litigation database where I read that they have more than 45 uh, lawyers that will access them. And OpenAI tried to fight this, as you said, uh, they used the privacy argument, uh, and loudly in public and they asked for a privilege, etc. But uh, it didn't work for them, this privacy argument, uh, defense.

Speaker B: Yeah, yeah, I remember. Yeah, but still, I mean the, the. It's not public, right?

Speaker A: Yes, yes, of course it's not, uh, what they're not going to publish on the Internet, you know, the 108 million conversations. There is a protective order. The conversations are de identified, they sit under a protective order which restricts who may see them, uh, what, uh, they may uh, be used for. Uh, so this is very clear. So there are a lot of, lot of protections and uh, OpenAI is trying to introduce even more. But the study spent some time on why protective orders reduce the risk, of course, but without removing it entirely. Um, uh, modern litigation runs on many hands and many systems. You know, law firms, experts, discovery vendors, repositories, all of them can fail. Law firms and ediscovery providers have themselves been the victims of major breaches. So I document some uh, cases and especially as we know, we Europeans will have a huge debate about the difference between pseudonymization and anonymization. And as a matter of fact, de identification that happens there is most often pseudonymization. Uh, and uh, you know, we have some interactions where the simple fact to remove your names, for instance, it's not enough because if you are giving details about when you went to the hospital and where you live, etcetera it can be re identified. So yes, there is protection. But uh, as a matter of fact, uh, in this protective, or don't forget the other states, uh, that uh, we're talking now here about the big discovery cases in the United States. And some Europeans will say yes, this is not happening in Europe. We don't have similar mechanism. I cannot imagine a judge in Europe who will ask OpenAI or Anthropic to disclose uh, uh, the conversations of millions of users. But don't forget, and this is important for the auditors that, that in Europe also and uh, uh, we already have cases both in the United States and uh, in Europe, uh, your conversations can be demanded in your own litigation. You can have for instance, don't know in a case of divorce to use it against you or in business relationships. You know, it's very, very important in business we already have cases where um, uh, businesses have been asked to produce and the other party is using this against them. So in your employment dispute, you know, so uh, people should also be aware of this. It's not just about the big discovery cases, it's also about disclosures in your own uh, specific uh, cases.

Speaker B: Yeah, okay. Okay, well, okay, we're getting worried. Let's go for the fourth one now which is maybe even more worrying. Okay. And growing and growing mean, uh, you know, at speed. So that's the breach. So what would you say about this? So breaches. And um, how are breaches going to work? How would you expect them to affect these conversations that are being piled. Because some of them, I mean. Yeah, please let you elaborate on them.

Speaker A: Yes, I, I think I, I will be brief because I spent a lot of time on the other three doors. Let's say, uh, the big thing is that when you have all these conversations, intimate conversations in a single high value target, you need huge cyber security. And although we don't have any big case for them, the public record already offers a round of warning shots like misconfigured databases left open, sharing futures that push chats into search results, credential leaks and even there was a recent uh, study by some researchers that I examined in my um, uh, own study which have documented um, without any breach at all. It was just trackers embedded in chatbot interfaces transmitting conversation identifiers to for instance advertising and analytic networks.

Speaker B: And by the way, the, the Spanish.

Speaker A: Yeah, exactly, exactly. And the Spanish dpa as you know, asked the ADPB very recently to take over this uh, finding. So I think that there is a uh, big issue there, especially when you Add the agent AI which will access everything in your computer. You don't want it to start leaking things in uh, an accidental way or to be hacked. So there is also all this huge fourth door to think uh, about and uh, to have uh, protection from uh, such disclosures to third parties.

Speaker B: So let's move into the health Agent Rush, because you wrote another paper. This is really happening very very fast and you've called it the health AI Agent Rush. And I loved how you presented the trade offs between all the safeguards that ah, we've been building in Europe and how now those very safeguards are leaving Europe out of the additional effort that seems to even be aligned with your own ideas for how conversations should be protected. But I'll let you elaborate on that please. What you wrote about the health.

Speaker A: Thanks a lot. I think that when we're talking about uh, these health agents, uh, there are two dimensions, the privacy dimension and the governance uh dimension. The privacy dimension is the following. Uh, what happened first of all is that five major. As soon as I published the first part of my study, you know I was calling for what I call a sealed mode.

Speaker B: Yes.

Speaker A: Which means uh, this was my main recommendation, one of my 10 recommendations which is protect that in some conversations, like health conversations, nature and has protections. And some people told me but this is science fiction, it will never happen. And uh, in reality I discovered just few weeks after that it was already happening. It was already happening. And five during this period since the start of the year, five major companies launched health specific AI products, agents if you want, inviting also users to connect their medical records, their wearables, their wellness apps, etc. So uh, what I found is that in all these five companies uh, they were moving towards, I was calling sealed mode which means for instance uh, in ChatGPT you will have a tag has not yet arrived in Europe, but it's experiment in the United States where uh, in the interface of ChatGPT you will have ChatGPT Health and you will enter there and a lot of recommendations that are made for Silk Mode you will already find them, which means that you for instance uh, no advertising, no training. They will not use your health interactions for training, separate memories or siloed personalization. What happens in the health space remains in the health space and other limited data retention, limited human review and uh, as a matter of fact this is something very uh, positive from a privacy uh point of view. The problem nonetheless is that instead of they could, with a turn of the button they could have launched this also in Europe to have this uh, tag and all Europeans would have uploaded. Uh, but instead of this they have launched a kind of separate product because their idea and we can understand them that uh, if you really want to get good advice and this is what health professionals say also the ChatGPT must say know you, you know, or the other chatbot. So it makes a lot of sense from their point of view and to avoid also liability problems to know they're not going to say to problem to person with uh, heart condition to do more exercise. You know, the chatbot needs to know you. So their idea is that you have to connect you. It will be much better if you wish to connect your uh, health data, your medical records, your results of a test and everything. And from this point of view it will become much more effective. The problem with this is that in Europe you cannot do it that easily because we have Article 9 of the GDPR concern sensitive data. You have the medical device regulation, uh, which uh, creates uh, problems it considers as a medical device. You have the AI act high risk rules that might uh, um, be applied here. So, so we have some problems which means that uh, uh, unfortunately we're not able to benefit from this privacy by design futures that they have built in order also to create trust. And this is what I say in my study. It's a real pity. They could have launched first the privacy by design futures and then after resolving the regulatory issues permit people to connect their. Many people do it at doc. By the way, you know, when you come back, uh, from a uh, test and you have not yet seen your doctor, you go to ChatGPT and you upload the result and you say what, what does this mean? Explain to me and what should I say to my doctor? They do it already Anyway. This is the previous dimension, the governance dimension is uh, what you said, which means that uh, if this really works, and we can hope it can work, you know, all my studies are entirely pro innovation and uh, health professionals are saying that in some cases, not in all cases, but in some cases this advice, it would be fantastic in overwhelmed health systems, uh, around Europe for instance, you know, or uh, it's Friday, you have a condition and the doctor is not there. It's good to have your private doctor, you know, in your telephone, uh, if it can give you very good advice, can be fantastic. But uh, we must not forget. So it's something that Europe should not block. On the contrary, uh, I'm calling for uh, somehow a proactive work in order to do this right. It could be fantastic to benefit from These evolutions. But we must do it uh, right through. For instance, I don't know, could imagine public private partnerships launch these with hospitals, with uh, I don't know, uh, cooperation with the European health data special. Whatever the issue is at the same time the governance issue is that whereabout, you know, you're talking about the European data space and we adopted finally this regulation and we are trying to use uh, uh to uh, uh favor the secondary use of health data uh uh for research in order to invent good things. This is very good but at the same time you see uh, it takes a lot of time, it's very burdensome, a lot of, of bureaucracy getting access, uh uh, it's going to take uh, some time for researchers. So all these safeguards that you have and all this time consuming mechanism, you might not have them for these private companies who are about to create probably the biggest health data, private health data hubs in the world. If you have 900 million users of uh, OpenAI, for instance, uh, as we speak, it must be more, much more. Uh and uh, from all over the world, from cities, from villages, from the rural area and the urban, from different countries, you have a fantastic sample and you can only imagine how useful this could be in order to detect side effects of drugs, in order to target people in a very, very precise way for clinical tests, in order to invent new uh, treatments or to find new uh, uh pathologies. So there is a big governance question. I think that it makes sense for Europe to act a little bit proactively and cooperate with these companies and to uh, do things right. And the companies could be very open to these public private partnerships because uh, precisely. They want, they know that there might be liability problems behind when you touch the health sector. So they might be much more happy if it is doctors and hospitals who use their technologies first and who you know, uh, uh find frameworks would provide all the safeguards and protects them from liability. So I think it could be a win, win situation and uh, uh this is why I wrote this study. But I also did uh, uh an op ed in Le Monde talking asking for proactive action and interest over Europe on these issues.

Speaker B: Yes, and you've argued about that balance and uh, I think that's very interesting where you're saying that you can have, you can minimize data you collect but still you can work around memory because memory makes it useful. And you were saying yes, you can have some memory so within a sealed room and yet have all of these things and this idea of confidentiality by Design and confidentiality by architecture. Um, yeah, as you were saying, it did happen in the end. We saw it happening um, with uh, Meta.

Speaker A: Right, with meta, but not only with Meta because just uh, uh, between the invitation, participate your program Sergio, and uh, the recording that we are doing today, Apple also announced uh, uh, something very similar with Siri AI. Yes, this is very interesting. Let's explain first to our auditors what exactly happened. And we wrote uh, a paper with Peter Zweier discussing this very important evolution, uh, which is the first mass market example of confidentiality enforced by architecture, rather by uh, promise. What happened is that when you use WhatsApp and I send you a message, we have end to end encryption. Everybody knows this and it is protected. And you know that for years the law enforcement were asking for backdoors and there is very strong resistance both by the companies, by civil society, by cybersecurity agencies. And we're very happy to have these tools where we can communicate uh, and privacy is uh, really uh, protected. Ah, and meta went all the way uh, to the end of this logic, uh, because you know they have introduced uh, when you use WhatsApp you have a little circle and you can open their chatbot and start discussing with their chatbot, uh, meta, uh, AI in the same way you discussed with ChatGPT. And this was not encrypted. So you had an environment where there is an expectation of big privacy. But when you were interacting with your chatbot, all the things that you have said until now apply and meta, uh, could access. And so what they did is that they announced that they are launching now the encrypted chatbots with meta AI, you know, incognito, they call it incognito mode. Uh, it's a little bit like uh, the incognito chats that you already have with ChatGPT with Claude, you know you go up, right and you click the little phantom or I don't know what and you can have a conversation. But uh, the conversations that already exist, the cognito modes are not encrypted, are not decrypted, which means uh, that even though you, you go out of the conversation, very sensitive conversation, there is a ah, possibility ah, of human review. Humans can access it. There is a data retention for 30 days. It can be produced to law enforcement. They can have access to this, everything. It's not used for training of course, but there are uh, uh, limited protections and the companies can access it. Meta. What they did was that they have created what they called um, a trusted execution uh environment cell which creates exactly similar effects like. Which means that Meta cannot access your conversation with uh, meta AI and they have also asked uh some uh, big companies to audit this. The fact that they cannot access the conversation, which means that you can um, according to their claims you will be able to communicate with the chatbot entirely in a confidential way when of course there is no history. You cannot conserve the history. You can only do copy paste and save it in your computer if you want. But uh, um M. When you leave the platform and you close uh this the conversation will be destroyed and will not be retained and will not be visible my meta, which means that you cannot have all the things you remember the four doors we're discussing before you can have none of them. Them, none of them. It cannot be disclosed to uh, law enforcement, it cannot be disclosed to discovery, it cannot be breached. And part one also, and this is the catch probably Sergio, is that there cannot be a moderation, you know and there cannot be uh, the referral to oh they can be a moderation but Meta cannot see it. It must be done in an automatic way. And I think this is, is eventually very interesting and uh, Apple just to add this, that Apple rebuilt Siri around the same uh logic their private cloud compute. And uh, uh uh also they made a lot, a lot of promises about privacy by design and uh architecture in their announcements just uh 2 uh 3 days ago. And uh, uh we will see to what extent other companies will follow this example uh and to what extent. It's not necessarily appropriate for all, but we will see who will uh, be uh, who will follow this example, who will prefer uh, the classical example that we have here eventually introducing more protections like uh, you know, sealed mode, like the health uh tags that uh we discussed earlier.

Speaker B: I mean if you have a very powerful phone and they'll become more and more powerful, you can do things locally perfectly. Nothing has to go anywhere. If you have to go. Yeah. And then they have this uh, idea, this private cloud compute where they have their own silicon and the Google models, right. Or their sort of uh, co developed models on the cloud. And then you do need additional safeguards because they're in the middle. But at that point we have this connecting it back to what you were saying in terms of uh, moderating. I'm sure some jurisdictions are going to be willing to get ah, if you look at the UK and the Online Safety act for example. Exactly, you were saying this and there is a difference between moderating when there is a predator and there's a human Challenge, right. And a threat. And when it's just a system.

Speaker A: The thing is the following, you know, uh, what we raised with uh, Peter Zweir. We say uh, is this going to reopen the going dark, dark, going dark debate about encryption. And uh, uh, why what happens? The thing is that meta, um, cannot access your conversation with their chatbot, which means also that they don't know, for instance, you know, if you, you cannot accuse meta, uh, why you did not refer to the police. The discussion that somebody had about this, this could be a threat to the others or self harm. So, and people could say no, we will not accept this because we want uh, moderation and uh, people might press against this solution. But I think that there are a lot, a lot of things to respond. And you were mentioning this, this was a discussion that I had after the publication of these studies with uh, uh, Rihanna Peffer con of uh, Stanford, which said something very interesting. Interesting to me that I have not thought about is she made the point that um, you know, for all these years the case against encryption in WhatsApp has rested on child safety. You know, a human predator, as you said, that could use the protected channel to reach a victim. And this does not map into chatbots because there is no human predator on the end of the line. The chatbot cannot groom a child. And the chatbot will be trained as much as possible, as much as possible to protect the child. So somehow you see that um, some of the arguments that have used in order to say let's introduce backdoors in order to protect child, for instance, you know, in WhatsApp, some of these arguments will not uh, be uh, relevant and transposable. And I think that she made a point. On the other hand, she also worried about incentives running uh, the opposite way that the lawsuits that we have seen in cases like the Tumble Ridge or um, FSU might um, discourage providers of taking this stance, um, and push them instead towards reading more, m surveying more and reporting more. I think that she makes a point. But liability cuts both ways, which means an accusation of having seen warning signs and done nothing cannot attach to providers that cannot see, even if it may face the opposite charge that it blinded itself by design. And we will see to what extent we're going to have this kind of uh, accusations. This does not mean that meta is not doing moderation. They are doing a lot of moderation, but it is done inside, automatically inside the translate environment. So it is done automatically. And of course they will train the models in order not to give um, a response uh, or to block accounts. M. You know, they will train and then people. I think that the first thing that researchers will do when Meta AI launches this is that they will test. They will test the system to try to see if they can, um, um, um, you know, bypass the protections and they can refer this to Meta, and Meta can improve. So, uh, just. I wanted. I'm sorry to interrupt. I wanted to add that an architecture the provider cannot read is not an architecture that cannot act, uh, that cannot moderate. It's done by automatically. Like, also for social media, it's mostly automatically. M. Um, there is human escalation, only some cases. And, uh, somehow, uh, we will see if the safety of the systems will be strong enough in order to avoid this accusation of blinding themselves by design. I'm sorry for interrupting. I just wanted to add this important point, uh, not to create misunderstandings about this.

Speaker B: That's perfect. And that kind of automation, which makes sense, but it's going to need to be fed some. Some flags on the other end. Uh, what was a false alarm? What were false positives? Sort of like Instead of conversion APIs, we have sort of disaster APIs. They need to. That's what I'm thinking. Meta has to look at how many of these things actually happened, which is a terrible thing to think. But, yeah, if they want to feed and train that system, you need to know what's success and what's failure. Okay, so you are in Europe and you're a European law professor, but you're looking at, to the US A lot. So why. Why should Brussels look so much at so many examples and so many developments across the Atlantic?

Speaker A: Yes, because for a simpler reason, you know, it's interesting that I mentioned this in my preface, uh, that, uh, you know, we often say there is no air regulation in the United States, it's only in Europe. This is not true. Because in all the cases that we discussed today, in much more cases than discussed in my study, all the developments come right from American courtrooms in discovery fights in war and disputes, moderation issues. So, uh, if you work in privacy in Europe and you're not watching what is going on in the United States, then I think it is an error. And one of the big, um, uh, objectives of my study was also to raise awareness in Europe about all these things, uh, because I think it is very, very important, including, you were talking very practical things, how, uh, defendants can protect themselves when they use chatbots to prepare their defense, or how can you protect, um, against discovery in your own cases. Uh, and all these Things. There are important things happening in Europe also. We mentioned earlier the Spanish regulator asking the EDPP their GDPR has a of lot things to say about all this, but I think that we should definitely turn our attention towards the United States and what's going on. And I was very privileged, I want to say this, to have uh, the help of uh, I'm not an American lawyer, I'm a European lawyer, as you said. Uh, and uh, I tried to do my best to present all this in an accurate way. And I benefited from tremendous helps from some very, very good American lawyers such as uh, Peter Zweir, Richard Salgado, um, um, Debray Kennedy Mayo and others. And uh, I want to thank them very much. With all the disclaimers, of course that

Speaker B: apply here, you end up with uh, 19 recommendations in blocks, very well structured. Could you give us a few. The few that you think matter most?

Speaker A: I think that for each topic there are different recommendations, especially addressed providers. Let me, let me try to pick the four. Let's say that, summarize. First, I think that providers should publish how often they hand users over to the police and to governments. Um, they could include this in their transparency figures, for instance. This could be uh, useful to bring more transparency concerning, uh, disclosures. The second will be to tell people honestly what the chatbot is and what it is not and uh, to explain to them that it's not your lawyer, it's not your doctor, it's not your friend. Um, because somehow people start to have um, misconceptions about this. The third to be to warn users that their chats can surface in their own lawsuits and give them uh, a delete button that actually deletes somehow because uh, there is a lot of data retention going on without knowing it, despite you deleting. And the last would be the most important to build privacy by design privacy into the architecture, not into a policy document that can be rewritten without you knowing, uh, on a Tuesday. I think that uh, what I proposed about Silk mode, you don't need eventually in all cases, this is for providers. I would be very unhappy if I lose the history of my discussions with ChatGPT or uh, Claude or Gemini, for instance, because they transform all these to an encrypted thing that disappears immediately, but on the other hand offering other elements, important elements of the scene mode for some very sensitive, sensitive conversations, like the example that was said earlier, you cannot have health conversations, uh, in the same way that you ask the ChatGPT to write you a, uh, birthday card, you know, it's something really different and they need to create protected environments and privacy by design.

Speaker B: Should everybody just stop using the tools? Um, you know, from a very practical point of view, if you do not really understand the nuances and the settings,

Speaker A: no people, even if I was calling them to do so, and I will definitely not call them to do so because I'm uh, entirely uh pro innovation. Uh uh, people will still keep using because it's very useful these tools and can bring a lot of uh, positive elements. But they should do this responsibly. People should become literate users I think and uh, uh, decide what kind of conversations they're having. Um, uh, and uh, sometimes avoiding having. People should be aware of the fact, for instance, that their conversations could be used in a courtroom, you know, uh, in a specific litigation. Uh, you cannot work for a company, you know, and start discussing all the uh, secrets that can be used against you in such, in such a way. So, so I uh, think that uh, uh, people must uh, get aware. If you have something very, very, very private and you are afraid of this disclosure, at least use either an encrypted model or a temporary chat, uh and uh, uh uh, protect uh yourself. And I think that for the providers there are a lot of, of improvements going on. Providers have an interest. They want to create trust. Uh and trust will be uh. Of course efficiency of the model will be very important, but trust will also be an extremely important element for the one that you uh, will use, uh, the chatbot that you are going to use. And uh, somehow I think that users uh will uh, and should reward the companies that build confidentiality. You can verify over the ones that merely assure you or even say nothing. I have the example of some companies, uh, so demand turns a virtue into a future. And I think that um, the more their providers do for uh, protecting our privacy and our interactions with their models, uh the more uh, they will be uh, successful.

Speaker B: And then we can take that, that um, folder of the face in the painting.

Speaker A: Exactly.

Speaker B: Bring the apple.

Speaker A: Exactly, exactly. Exactly. This is the idea.

Speaker B: All right, thank you to. Thanks again.

Speaker A: Thanks a lot Sergio for uh, having me. Always a great pleasure.

Speaker B: Okay, that's it for today. Find more about this topic or access the transcript for this interview on Masters of Privacy dot com. Also feel free to join us for a breakfast, workshop or live recording session at a venue near you by visiting the Events section or subscribe. Subscribe to get all the things that we keep adding to the list. I would also like to encourage you uh, to join our LinkedIn group or BlueSky feed. If you're really passionate about this particular interplay between privacy or data protection law on the one hand, and marketing, e commerce or media on the other, thank you for listening.