AI's Double-Edged Sword: Risks, Rewards and Race Dynamics in Cybersecurity with Dr. Joanna Santos

Speaker A: Welcome to Paradigm Shock, a podcast where we explore the historic changes defining our times. I'm Anjan Roy and today I'm joined by Dr. Joana Santos, Assistant professor of Computer Science and Engineering at the University of Notre Dame. In what is the first in person video recording of an episode, we discuss the very timely nature of her research which traverses artificial intelligence, cybersecurity and software engineering. We discuss the significance of the latest AI models such as Anthropic's Mythos on cybersecurity. We also discuss how AI generated code presents both new and unforeseen risks and opportunities, not just for industry, but for society more broadly. Recorded on Thursday, June 4, we also end with a brief discussion on her life journey from small town Brazil and thoughts on the 2026 World Cup. If you like what you hear, then please subscribe and spread the word, especially regarding our new YouTube channel that is just getting off the ground. Every little bit helps. So let's get right to it. Professor Joana Santos, welcome to the first video and in person recording for an episode of Paradigm Shock.

Speaker B: Thank you so much. Thank you so much. I'm so happy to be here. And uh, thank you so much for the invitation. I'm excited to share some of my thoughts on AI and CyberSecur.

Speaker A: Well, I am. So it's an honor that you are the first, uh, as an in person video recording, uh, an episode of Paradigm Shock. So we'll get right into it. So you're a professor at Notre Dame in the Department of Computer Science and Engineering. You've had this very, uh, interesting trajectory. So you started off in Brazil. You graduated the only person in your class in 2013. You were the only one. Audience would love to hear about that. Then you moved to Rochester Institute of Technology. You got your Ph.D. from there. Now you're leading department over at or, sorry, leading a lab at Notre Dame. M. In the Department of Computer Science Engineering. You've been a professor there. You've published widely. You are a Google Research Scholar recipient, I believe, from a couple years ago. And given everything that's happened in the world, your research is really timely now with the convergence of AI and how it's merging with cybersecurity, both in good and not so good ways. So tell us about your career trajectory, how you landed where you are and what got you interested in the architecture, level of software and security.

Speaker B: Yeah, so as you said, my journey happened. I started in 2009, so I did my bachelor's in computer engineering. And then back then I realized, ah, uh, because computer engineering is a mix of, you know, software and hardware. And I realized well I really not good at any harder so I'm just going to switch to software. And that's when there was a scholarship from Brazil to actually study in us a full time scholarship. I applied to that. I ended up coming to RIT for to do my master's and back then I met my PhD advisor which happened to be my PhD advisor and he worked exactly at that interview intersection of software architecture and security. And that's how I kind of got into that field. And one thing that kind of got me excited on that domain, why actually I focus on software architecture is because oftentimes when research when it comes to security was focusing on after the fact, after the software has been built. And my goal is to shift that thinking about security much earlier during the software development life cycle and specifically even before any code is written, even when you are still thinking about the requirements, how are we going to implement that system? Let's bring security in. Because the goal here is that if you have a system that is engineering engineered with security in the first place, would be able to avoid zero day vulnerabilities. That's kind of what got me into there. And that's something that I have worked throughout my PhD studying these vulnerabilities that were caused by flawed design decisions and then also working on techniques that could detect those issues. So that's sort of my career trajectory. And then when I started Notre Dame I did continue on that but then I joined I think in 2021 and then at the end of 2021 that's when ChatGPT was really not ChatGPT. I apologize. It was actually GitHub Copilot was first released and uh, and then LLMs came into play, became widely popular and that's when I started to bring LLMs into the picture as well. So that sort of has been my trajectory so far and lately I have been out dipping my toes on quantum computing which I think is going to be the. That's what I'm betting on.

Speaker A: So well that we're going to get into everything that you said there and later on we're definitely going to ask. I'm going to ask you about quantum computing first for a general audience. There's been a lot of news, a lot of hype on Anthropic. Well, Anthropic, everything. Anthropic is in the news right now. I don't think there's any company more in the news than them arguably, but specifically the Mythos model which they chose not to release because it was so powerful that it could find vulnerabilities in almost every single operating system in the world, or the vast majority of them, and then be able to autonomously patch them together, string them together, adapt, and then that gets to, gets us to a really, really bad place when it's able to autonomously patch them together in terms of how our traditional cyber defenses can handle that. So what was your first reaction when you heard about the Mythos model that Anthropic announced and how they were not going to release it? And how have your thoughts evolved as you've learned more about it and studied it and news has come out on what its capabilities are?

Speaker B: I think my first thought when it came out is that this is great because in the end of the day to find vulnerabilities is a, um, labor intensive manual process. Of course there are still tools that you can use to help find these vulnerabilities, such as aesthetic analyzers, fuzzing techniques, but in the end of the day it's um, an intensely manual process that takes, requires people that have security training and not an average developer don't have that background. So it is great that we have models that can automate that process and find far more vulnerabilities in a shorter period of time. And I'm appreciative on the fact that Anthropic is not releasing that to everyone because if that type of technology goes into the bad hands, that means that attackers, uh, would be able to find those vulnerabilities that are not patched yet and actively exploit them. So I appreciate the fact that they are not releasing to everyone and they are doing that in a controlled fashion so that only the organizations that have critical software system, they have access to it first. Exactly for that reason. So I'm quite excited about it and I'm hoping that I also get access as well to it as a researcher, because I would love to see, uh, the capabilities that it has, where it fails and how it could improve. Um, but yes, I'm very excited about that technology. I think it's very helpful for us to detect these vulnerabilities and soon enough to be able to also help us patch it. Because detecting vulnerability is just one piece of the puzz, the other piece of the puzzle. And the most important one is patching it. Right. Finding the vulnerability is difficult, but it's really the easy part of the job. Now let's fix it. So, um, I think all these technologies are changing the way the cybersecurity field will go through. And especially because the way you think about security is that it's a tug of war where on one side you have the software vendor trying to protect the software systems and the other side we have the attacker and it's a race where they're trying to. Whoever finds the vulnerability first will have the first, um, dibs on it. So, uh, as a software vendor, if I can have a piece of technology that can tell me quite quickly all these true positives, these are vulnerabilities you have to patch and the severity of them, that's great. So I want to avoid these zero day vulnerabilities from happening. So I'm quite hopeful for it.

Speaker A: Well, that's an optimistic take in the sense that from a defensive standpoint, because they chose to release in a very controlled fashion, this helps cyber defenders patch them before attackers can get access and before attackers can exploit them. If I'm, if I'm flipping this around from a risk standpoint, the fact that this kind of a capability is out there, uh, are, should we be, how concerned should we be? Especially that, not that, not just that there is a, there is now a technology out there that can find these vulnerabilities, but that they can do things that traditional cyber offense perhaps could not do before, you know, that they can autonomously chain these, um, systems together. Sorry, change these vulnerabilities together in an adaptive way. So it's also in theory, in the future could elevate the offensive side. How do you think about that? Because it's great that if it bolsters defensive. But if it ever got out and bolstered the cyber attackers or the hackers or various, uh, malicious actors that are out there, that's a different, you know, that's a different problem set we've got then.

Speaker B: Right, agree, absolutely. And in fact, it's just a matter of time that we will have a similar model that is open source, that's available to everyone to use defenders and attackers, uh, and they will be used by attackers to generate exploits and hack systems. So in the end of the day, the same problem remains with or without LLMs, which is it's a race. Whoever finds the vulnerability and patch first is going to win. Right. So it's just making things faster. Sides will have access to this technology eventually and it's whoever finds it first. So it is, it is in the end of the day to steal the race that we always have had. The difference is down that the race is much faster. We went from you know, cards to formula One cars. That's essentially it. But the race to find vulnerabilities and patching them is still the same. Yeah, but it's still definitely concerning that attackers, uh, will have access to these technologies and hack system way faster than what it was 10 years ago. That's for sure.

Speaker A: For, uh, step back for a general audience. What is zero day? Excellent question. And what does that then mean? And the next question I'm going to ask you is the three day. The fact that now, you know, patch times are going to three days. Like that. What does that mean? Like that sounds very short, but it's shrinking, meaning you have to be able to patch these very quickly Right now if you're on the defensive side of it, or else you're going to be hurt badly from being vulnerable from the offensive side. Could you explain both of those? So start with the, uh, what did I say?

Speaker B: The zero day.

Speaker A: The zero day, sorry. And then what does it mean when, you know, patch times are dropping to three days?

Speaker B: Excellent question. So what is a zero day? A, uh, zero day is basically a vulnerability in which the software vendor only had zero days to patch it, meaning that that vulnerability was there in the software system and the software vendor did not know that software was then released to the market with that vulnerability there. That was hidden. Someone else found it, not the vendor. And that a, uh, vulnerability is being actively exploited. And now they have zero days to patch. It's being actively exploited. You have to patch right off the bat. And that kind of connects to the second part of the question, which is the windows of patching these vulnerabilities is shortening. Exactly. Because nowadays we rely on software for everything. Right. I used my phone to be able to get here, otherwise I will not be able to know how do I get from my house all the way to the recording studio. Right. So we have software systems all over the place. We rely on it more and more. And because we, we are so attached to technology, when there is a vulnerability, we want to minimize the impact of it. The longer the vulnerability stay in the system, the more exploitation is going to happen and the more hurt we are going to, uh, introduce in the society. Right. So that's why the guidelines are shrinking and shrinking this window of patching to be shorter and shorter, putting way more pressure to software vendors to patch them. Imagine how overwhelming it is to a software vendor has several vulnerabilities in it and has to now prioritize which ones do patch first. Because we have to patch all of them. Right. So it is a good guideline for sure, but definitely puts a lot of pressure on the software vendors. And again going back to the first question, which is what if now we can have autonomous agents that can help us patch, help us triage these vulnerabilities? Because another thing is important to highlight in here is that when you receive a vulnerability report you have to see, well, is this a true positive or not? Especially if it is coming from a tool that may be a false alarm. So that is the uh, work of triaging it. So if we can have LLMs that can help us triage it, that's great, minimizes the effort. And the second piece is now patching it. This is a true vulnerability. This is the problem, this is how we can exploit it. How can we now prevent that vulnerability and make sure that the system is no longer exploitable? So that's the thing.

Speaker A: Thank you. So for the average person now or an executive that's uh, out there, what systems are actually at risk right now? Um, in terms of like should we worried about whether it's a hospital or water treatment plant or your bank account, like we've heard that some of this capability that's now out there, that if anthropic did release this into the wild, which they did not, but if they did, like how worried should we be? Either that firm or a future firm that open sources something, uh, how worried should we be about everyday things in our physical lives that are now potentially vulnerable if we're not taking proactive defensive measures, how should we think about that?

Speaker B: How worried should it be? We all should be worried. I don't want to be negative or catastrophize the situation, but any software system, regardless if it is a safety critical system such as hospitals or if it's just, you know, worldly, the worldly game that I like to play, I used to play a lot every day that is way not that critical. But any piece of software, any piece of code that takes user input can be prone to a vulnerability. That's an inherent risk of a software system. So everyone should be worried about because any software that you use daily can have a vulnerability that might impact your day to day life. Could be something as simple as let's say someone is hacking the worldly game and you no longer have access to the game. Sure impacts my life, but it's not the end of the world. What would be more concerning is that let's say software systems that runs in hospitals because if you go to hospital, as we see now is digital and now it forces us to go to Paper. If there is an attack happening, more like the host is going to shut down the digital systems and goes back to paper and that's going to make everything to slow down. And in fact, I don't want to start a spoiler in here, but if anyone watched the show, the pit that is on the gbo, there was exactly an episode exactly on that there was a cyber attack going on on the hospital and they actually shut down all the software systems so they have to go back to paper, pencil and paper. And that led to delays and in fact one of the patients, the health ended up deteriorating quite a lot because of those delays. Right. So everyone should be worried because it's going to affect you in different ways, some of them more severe than others, but it's going to affect you. So any software vendor that you have out there, regardless of the domain of your application, whether that's gaming, whether that's water treatment, whether that's banking, you can be prone to vulnerability. That's just the nature of things. Right. Mistakes will be made and those mistakes will be exploited by attackers. Right. That's, that's sort of the nature of things. Right. And it's a follow up question that I also had in my master's degree when I was taking a course, is that how can you guarantee that a software is 100% secure? Well, make the software not take any input. But then the software is useless. Right. Because if it is, taking input from outside it is a risk.

Speaker A: Yeah, it's like the safest thing if you want to stay healthy, stay in bed.

Speaker B: But uh, but even then it's, even

Speaker A: then it's not actually, it is not.

Speaker B: Right. So it's. Right. That's sort of the accepting the risks.

Speaker A: Yeah, I think that's a good transition to a policy and regulatory discussion. Yeah. So in the US you had cisa, I believe that stands for the Cyber Infrastructure Security Agency that there's reportedly they've lost a lot of staff. Um, by some reports out, they've lost a third or 30% of their staff over um, in recent months and years. They're facing potentially really big budget cuts. How should we be looking at the regulatory landscape and the defense landscape if CIS is now on one hand, um, losing staff. Right. Vulnerability, uh, capabilities are going up on AI cyber capabilities, both the defensive side M as well as the offensive side. And then you have this other issue of AI generated code which is also a big part of your research. I want to get to and to what extent regulations haven't really caught up to that Our cyber regulations haven't really caught up. So how do you think through these combinations of risks and there's opportunities, but specifically how do you think through these combination of risks now?

Speaker B: Yeah, uh, it is a race again, going back to what I said, it is a race because the technology field is changing and the policies have to evolve alongside of it. Right. And one thing, I'm not too familiar with the policy side of things because I work really on the technical part, but one thing that happens lot is that as soon as the domain change, new guidelines are put in place. So for example, what is a secure password? A secure password nowadays have to follow certain guidelines, has to have special characters, has to have at least this many, um, characters as well. Why? Because given the hardware that we have nowadays, if you have a password that is less than, let's say eight characters, it's very easy for you to break that password. Right. But it's very likely that down the road, when you have quantum computers that are widespread, then cryptography algorithms will have to change. What is, uh, a secure cryptography algorithm right now will not be a secure one after the post, uh, quantum takes place. So in the end of the day, policy is catching up, but it's really lagging behind. Right. The technology evolves and then the policy tries to catch up, but it's really always kind of lagging behind what is happening. And that's sort of the nature of things because creating policy takes time. It requires a discussion from lots of stakeholders from industry, from academia, and that takes time. That's just. Can we have an LLM to create the policy for us? Maybe, but that's a, uh, process that is always trying to catch up and it's always lagging behind.

Speaker A: Uh, I'm going to dig in a little bit now on some more of your research and you have a paper, your security eval paper, and it's, I think very timely now given how LLMs have not just evolved, but evolve implies evolution. This has been much more of a revolution we've seen over the last couple years, but specifically on the security side, and we've now seen over the last few months. This last Christmas was called Claude Code Christmas. Everybody's talking about AI generated code. We had an episode a few months ago with Nicholas Arcolano from Jellyfish on this massive data set on how, uh, coding agents are impacting software engineering. Your research here going back several years is really interesting to me because you've researched the vulnerabilities that in software code that's generated by LLMs and that they, and that the, the errors that they create and the vulnerabilities that they, in the code they produce, they don't come at random, they cluster. So how does that impact how we should look at security from LLM generated code? Especially now that basically the entire software engineering industry is moving to at least LLM assisted coding agents. How should we look at that?

Speaker B: Absolutely, that's an excellent question. And one thing that kind of motivated that work is that back then, now its security is being discussed more frequently. But back then when these models were released, they were often benchmarked in terms of tests. Thus this code in here that was generated by NLM passing the test or not, and those tests were testing the functionality of the code. So basically the input and the output that we obtained is matching the test cases. That is great. But just because it's passing those tests, it does not mean that it will not have a vulnerability. So perhaps that code is indeed, let's say encrypting the data, but it's using a weak encryption algorithm. So that makes it easier for an attacker to break that encryption. Right. And why that happens to begin with? Well, our hypothesis was that, well, these large language models are in the end of the day trained on open source code, public code, that it has permissive license. And just because it's on the Internet does not mean it's high quality. Right. It's unlikely that it's to be able to collect all of this open source code and do a curation to make sure that all these codes in here, they don't have any vulnerabilities. So since there was not curation in process, those vulnerabilities that you have in the real life will also be learned by the models. So the type of vulnerabilities that the model ended up generating is exactly what they have observed from these codes. So what we have started doing is that let's create data sets and benchmarks that can test that the code generated by these models, that is passing the functional test case as they should be, but that they also do not introduce any vulnerability. And we not only created these sort of benchmarks as well, but we also studied the models that were released back then. Those studies were run in 2022, 2023, so it was way older models. Right. GPT4 was not even there yet, was GPT3 was the latest one and we were investigating exactly that well, using the benchmark that we have just created. Let's see what is the performance of these models in here. And one thing that we have noticed is exactly that they pass, just pass the functional tests, but they do have vulnerabilities in it, whether that's SQL injection, whether, whether that's a buffer overflow or maybe using encryption or not using encryption at all to store sensitive data. So and then another line of research kind of ended up coming into place as well, which is how we can make these AI assistants security aware, meaning that they do generate the code, but they do in a way that it passes both test cases, functional tests, but also the security. So you would see that nowadays you see more and more papers are exactly focused on it. How can we make these AI assistants generate more secure code? Because and why this is so important. It's as you mentioned before, LLMs are widely used in software engineering. And there is a study that was I think back in 2022 or 2023 where they interviewed I think 47 or 50, uh, developers. So they made, I study where half of that used a model, used an AI assistant, the other half did not. One thing they have noticed is that the group that used AI assistants to write code, they wrote more insecure code and they believed that their code was secure because they trusted the models. So definitely there was a little bit of overconfidence that the models would do a better job than them in introducing, in not introducing vulnerabilities. So we want definitely these models to be security aware because the developers might be trusting these models too much that they would not generate SQL injection, they will not introduce buffer overflow, but they actually might. Right. So that's why this line of research is so important. Because if that code that was generated gets into production and has a vulnerability that in the end of the day and does not get catched, that will turn into a zero day vulnerability eventually. It's just a matter of time.

Speaker A: I didn't know that. I want to drill into that a little more because there are implications here that go way beyond just cybersecurity and the impacts of AI and cybersecurity, but the impacts of AI everywhere. That software engineers that used LLM assisted coding agents, they produced more vulnerable code and they trusted that the code was less vulnerable is what they thought. That is. Wow. There's probably a psychological phenomena of some sort that uh, we need to get a psychology professor on to explain that. But they talk about um, an overconfidence problem, but that, wow. That I think about the gradual disempowerment thesis in AI, uh, where humans are kind of pushed out of the loop to the point where we're relying on AIs to do more and more and everything and how that can impact human capability, just generally speaking. But this is a very, very clear example of it, an empirical example of it in a very important field. Right, because this is one where you get wrong. It's one thing where you get something wrong in a low consequence area, but in a high consequence area you get something wrong where, where diligence is of like, is at premium. Uh, like how should we be like, how do you think about that? Because that seems really problematic. If the cause, those are the worst. Those are the cyber defense people are the people you want to be the least trusting. Like there's other fields that you want people to be more trusting. You want them to take risks. Right. But cyber defense, cybersecurity is the opposite of that. You don't want them to take risks. You want them to be difficult. And often they are difficult to work with. Not you, but your colleagues, I'm sure, you know, some of them are, you know, everybody knows the guy in security, right? Like he's difficult. That's his job, he or she, that's really a problem if it's not only developing more code, but to me, uh, not only developing more vulnerable code, but the fact they trust it more. Like as somebody outside, not a cyber professional. Like that sounds like really worrisome to me.

Speaker B: Oh yeah, absolutely. And I think part of the reason as well is because the average developer may not necessarily have these skills to catch that the code that is in it has an insecure pattern. Right. Because security is part of the computer science curriculum, but it's not always a required course. Or maybe it's just going to be one course out of all the other courses that you're going to take as a computer scientist or a software engineer. Um, in fact, I do offer a security software class and I always say, hey, I'm trying to do my best in here to bring security into the thinking of students. But look at this. Uh, it's difficult because I really wish that security was part of the curriculum throughout that security was sprinkled in Introduction to Computer Science. All of these data structures all have had, uh, a little bit of security embedded. Exactly for that reason. Because my hypothesis, of course, I would love to do a study on that. But my hypothesis is that the developers, they trust these assistants often because they just lack the knowledge to understand that there is a pattern here that is insecure. So it's out of lack of knowledge, not necessarily, uh, um, uh, trusting too much. But it's really just, ah, they are not really able to see what is going on here. Um, but it's definitely worth exploring. Is that why did you trust that model so much? Why did you think that your code is secure? Because you use an AI assistant. It's definitely an interesting follow up study. Um, but again my hypothesis I would love to test that on.

Speaker A: This is a good transition point to a paper you wrote, I believe last year or you published last year, the Seneca paper about a certain class of vulnerabilities that are invisible to traditional cybersecurity methods. Talk about that and then start at a very general level. What does that mean by being invisible, traditional methods? And how do you think through this? And what are the implications of this?

Speaker B: Excellent question. So, and that actually is going to connect very well with machine learning. I'm going to explain a second. So the type of vulnerability I was studying on that paper was called object deserialization. What is that? Any piece of data that you have in your software, it's very likely that at some point you would like to store it, let's say store in a file or send over the network. Let's say you have a piece of software here, you need to transmit that data to another piece of software elsewhere. So this is a problem called deserialization. You get the data from one place, send it to the other. So you have to deserialize it in a format, whether it's just a sequence of bytes, zeros and ones, or perhaps in a text file. But you have to convert that data that is in the memory to something. And then during this conversion process you send that over the network or you save when you recover that data, which is deserializing the data, that's when a vulnerability can appear. What is that? It means that depending on how you serialize the data as an attacker, I could inject a sequence of bytes that once you load that data back to the memory, that sequence of bytes will trigger the execution of uh, a command and could be a command such as deleting all the files on your machine. And in fact I have a funny story around that, which is when I was writing a follow up study on this, I ended up deleting all my files in my machine. I ended up hacking myself due to a mistake that I have made. But, but this is just to say that that's a possible attack, right? Because I ended up hacking myself as I was doing the history. And why is that? This vulnerability is harder to catch by traditional methods? It's because that type of behavior of seeing this sequence of malicious command when you're loading the data, you cannot really observe that by just reading the code. Lots of techniques that we have out there to detect vulnerabilities, the way they work is by doing static analysis, meaning that they look at the code and try to find insecure patterns in it without running it. The problem is that this type of sequence of commands, malicious commands, they actually arise at runtime. So if you read the code you don't see that pattern in there. You actually have to run the code to be able to catch it, to be able to find that vulnerability. And you might be wondering well if I have to run the code to be able to find that vulnerability, so why don't I just do a uh, runtime analysis that is totally possible. Fuzzing is a way to go to detect this type of vulnerabilities. The problem about fuzzing is as you would expect, it takes longer to run because essentially you're trying to every possible type of input to see if a vulnerability is there. You try this combination, no vulnerability, try another combination of vulnerability and you have to keep trying, trying, trying. So it's a trial and error process. So my goal of this uh, my work is that what if I can improve the static analyzer so that they can infer this trial an error much earlier without running the code. So that was exactly the goal of that paper. What I called the taint based uh, call graph construction. The idea was exactly that, is that for certain parts of the program, which is exactly where uh, attackers are more likely to try to exploit, try to introduce malicious code, I'll uh, infer what are the possible combinations in here such that down the road I can use all these different combinations to see if I can actually craft an exploit out of it. It. And that work when I have done back in 2022, 2024 actually got published in 2024 was around Java. So basically software systems written using the Java language. But I want to highlight that this type of vulnerability, it's not only specific to Java. Any software system that serializes data and deserializes data and allow commits to be executed during the deserialization process is prone to this type of attack. And in fact in a follow up study, exactly the one that I ended up hacking myself, we studied the serialization of models because in the end of the day all these large language models that we have nowadays, the way they work is that they got trained with a vast amount of data and they have to be saved. So that down the road when you want to use, let's say ChatGPT or whatever assistant that you want to use, you have to load those models to the memory to be able to use them. You don't want to retrain every time you want to use these models. So that's exactly deserialization. You're reading the sequence of bytes back to the memory. And in fact, we studied the models that are available in Hugging Face, the open source ones. And we found that majority of those models that are out there, at least, uh, as of the time of the study, they were using unsafe, uh, serialization formats. What does unsafe mean here? It means that these models, they are using serialization formats that allows an attacker to inject a sequence of bytes in that model, that serialized model, so that when you load that model in your machine, that sequence of bytes will trigger code to be executed. And that's exactly when I was testing my exploit. I ended up deleting all my files. When I was demonstrating that that attack was viable. I end up hacking myself. That was not very smart of my time. But that's fine. Um, but yes, so we did find that this is a problem. And in fact we did also found cases of models that were available on Hugging Face that exactly had a malicious payload that had payloads that. For example, one payload is to create something called the reverse shell. What is a reverse shell? It's basically attacker. I would inject a stealth piece of code that runs on your machine in the background. And through that piece, uh, of code, I can now remotely access your machine. So I can start executing commands, for example, to see your files, to create files, to read your files, to read the data you have. So it's running on the background, so you don't even know, meaning that you load the model, you're using the model just fine. But there is something running on the background that is allowing an attacker to remotely access everything that is on your machine. And that's really bad, right? So we did find cases like that. And that was exactly facilitated by the fact that you're using a serialization format that allows kodun to be executed during the loading process of the data. So that's something that is severe. And we have been actively working on it, which is how we can actually make these formats to be more secure. So it's sort of another piece of work I'm exploring right now.

Speaker A: So one thing I heard there m open source models and hugging for generality, hugging faces is one of the biggest big, uh, sources of this, of open source models anyone can use. Some of these are m inherently More vulnerable, they have vulnerability. So on one hand that's a problem. It's obviously a problem because you're bringing in open source, your developer, you're bringing in open source models and they're not as good as they have certain problems. Right. And the fact that open source means anybody can use them and if they're widely disseminated, then you have a lot of software that has vulnerabilities. Are they vulnerable to traditional attackers? And in addition to that you have autonomous attackers like AI enabled offensive, uh, hackers and attackers. Are they more vulnerable to one than the other? And how should we think through that?

Speaker B: In the end of the day, all of these sort of um, dynamic attacks, the ones that they are using the AI assistance, they are still doing what has been done, which is they're still trying to find a vulnerability and inject a payload in it. The difference is now, now is that it's going to be at a much uh, more large scale, right? It's going to be faster. They're pinpointing these vulnerabilities faster than before. But in the end of the day, all these vulnerabilities that we have seen, whether it's, you know, deserializing the model or whether it's a SQL injection, these would all be caught by these tools. So in the end of the day, uh, everything's the same, it's just now everything's faster, right? Because we have now more and more assistance both at the defense side and also the offensive side. But it's in the end of the day, when you look at the neat, the details, low level details, it's still the same. It's vulnerabilities, it's flaws in a software system that whether it's a human being or an AI assistant, they are catching it and exploiting it. But in the end of the day it's nothing fundamentally changed from sort of the way things are attacked. It's just that the tools that we are using have more capabilities than before than 10 years ago. That's, that's it there, there's a school

Speaker A: of thought that says the only way to really defend yourself against an AI enabled attacker is an AI enabled defender.

Speaker B: Yes. How do you think through that, um, that is possible? I think it's because it's one of the things about um, um, generative, um, I forgot the name. Adversarial. Ah, networks. Right. Where you have these machines competing with one another. It is, I think it's a valuable way. And then at the end of the day, what might end up happening is that perhaps the most powerful model will win. Right. Let's say you have GPT5 versus GPT4. More likely GPT5 will win. Of course, assuming that GPT5 indeed has more security reasoning capabilities than GPT4, maybe it's the other way around. Right. But yes, in the end of the day, why not using um, if you cannot win them, use that technology against yourself. Right. So I think it's uh, just another layer of security. Because I think one thing that's important to talk about as well is that security is something that should be layered. Right. We often times talk about um, security in layers, like layer security, where you want to approach security from different perspectives. Right. You want to use AI assistance, you want to use traditional tools. And end of the day, the goal in here is that each of these techniques, each of these assistants, they are very good, maybe in catching this type of vulnerabilities, but not those ones. But when you combine all of them together, they will be able to give more security compared to just one of them. So that's why we have this term defense defi. Defense in depth. Exactly. So it's like uh, the uh, Swiss cheese, right? If you slice a Swiss cheese, they have holes in it, but if you stack several slices together, it's unlikely you're going to have a hole that's going to be able to go from the top to the bottom. Right. So that's sort of the idea here.

Speaker A: One school of thought I've heard, or another school of thought. There's a lot of schools of thought that are out there.

Speaker B: Correct.

Speaker A: One of them is that this idea of specialization, that weaker models can defend against more stronger models, attack oriented, strong uh, models, if those weaker models are specialized in say cyber defense or even a specific type of cyber defense. So the theory here would be, yes, something like Mythos is released, perhaps uh, some other entity will release an open source model that's almost as powerful as Mythos and has all these attack capabilities. And that's obviously highly prob. Would be highly problematic, but that the defensive side could potentially address that through weaker models, but maybe a series of weaker models, each having their own type of specialization as a type of defense in depth. Like how do. And even if it's just speculation, like how would you think through that, and how does your research, uh, and your analysis, uh, assess, uh, that framework?

Speaker B: Yeah, that's definitely another strategy to employ. Right. Because one strategy can employ is that, let's just use a general purpose, very large model to try to help us or let's have smaller models that are specialized, perhaps they were fine tuned to work very well on this case in here and kind of going a little bit off topic, but it does connect very well. And that's exactly what we have done on a paper that was in the context of security computer science education. We were trying exactly to guard the real models against malicious prompts. And we did exactly that. We have a small model that is just classifying the input to see if the input is quote, unquote malicious or not. And then we have another layer which is as part of the model itself. So you have a large language model that is going to be used to generate code, but you have a smaller model in here that is helping get the output and see if the output is derailing a little bit. So as you can see here, it was two smaller models combined together to help protect a much bigger model. So it is, in the end of the day we are pursuing the same goal, achieving security, but from a different angle, which is, let's use smaller specialized ones. Because our insight was that let's say, um, in a, uh, real life, if you're going to hire a lawyer, you hire a lawyer depending on what type of case you have. If you're dealing with criminal case, you're going to hire a criminal lawyer. If you're dealing with immigration, it's an immigration lawyer. You don't hire a criminal lawyer for your immigration case. Right. So that was sort of our insight as well, where. Well, isn't that what we already do in the real life? Let's do exactly the same in the digital world, which is, let's have specialized models that work very well for these cases in here. But of course they won't work very well in other domains. So for example, we want a M model that is very good at teaching computer science and security. But if you ask the model, can you tell me the recipe for an apple pie? The model will refuse to answer that because, hey, this is outside of my expertise. Which is exactly what happened in real life. If you go to a lawyer that does immigration and say, hey, can you defend me against this crime situation? The lawyer will say no, I don't have the expertise for it. Go talk with my colleague that is in that domain. So that's another strategy as well. Uh, which one is better? Well, we would have to benchmark and see what is the performance of each of them. But it's just different strategies again, uh, with the same goal.

Speaker A: If you're bullish on general purpose technologies or the general purpose capabilities of the, the most frontier of the frontier models, probably leading with Claude and the various anthropic models that are out there, they argue, and I'm not saying anthropic argues this, but those who are bullish on them. Well, some of the anthropic people do argue this, that the returns to scale are just so much, that is general purpose, that they can adapt in ways that specialized narrow models can't. Then there's others that argue that you get diseconomies of scale, that they get so big and hard to control that they are not going to be. They look at it more conventionally that specialization will win. Provided you have a portfolio of specialists. What would you speculate to the general purpose guys that are saying that um, there's not diseconomies of scale, but the opposite, that uh, the returns to scale, at least at this point are tremendous. And it's very hard to defend against that. That's what a pessimist would say if, ah, if you're like a safety or securities pessimist would say, or a uh, general capabilities bull, uh, optimist would say. But how do you think through that?

Speaker B: Yeah, I see the point of, well, general purpose models that are very large, they likely will beat the smaller ones because they can adapt. But the flip side of it is that when you have a very large model, there are the costs of training them, fine tuning them. Right. So perhaps, perhaps would be more cost effective to have a smaller model that has been specialized to that particular domain. Of course, granted that indeed that M model will only be used for whatever use case you're looking for. Let's say in my case, my use case was computer science education. I want this model to only be used in the context of teaching instance how to write code more securely. Um, so my speculation is that, that I think. Again, I don't want to. Again I'm speculating because to answer accurately which one is better, it's better. We have to do an experiment, a systematic study to compare and see the each of them. But my speculation is that um, it's likely maybe that the general purpose model could indeed adapt to certain use cases. But I still see that there is a benefit of smaller models that are specialized. I don't think should be perhaps one or the other. Why not both? Both. But if we have a large language modeling, a specialized one, why not combining all of them? No need. The two make one or the other. Let's do uh, both of them. So that's, that's my view.

Speaker A: You had Mentioned earlier, we started this interview with you that your forays into quantum computing, so big field, potentially the next next big thing.

Speaker B: That's what I'm betting on.

Speaker A: So when people think of quantum. Well, when some people think of quantum computing, I uh, think of two things. One, I, the first thing that comes to my mind is it can it, it's encryption like it can break any, it can ha into anything now or it will be able to in the future. The other side I think about is more quantum physics like superimposition, entanglement and those very fascinating almost metaphysical concepts that seem crazy. How do you think through quantum computing in terms of how it's going to impact your field? And particularly in cyber defense and when it combines with AI, quantum computing seems to change the entire problem set. How do you think through this in the field, how it's going to change the field?

Speaker B: Field? Excellent question. So yes, when we talk about quantum computing and security, I think the first thing that indeed comes to people's mind is quantum computing breaking all the cryptography algorithms that we have out there. Which is true. But the thing is that what I see as an opportunity is that uh, quantum computers, they work very well when it comes to optimization problems. Finding the most optimal solution in a uh, very short period of time. Exactly. Because it relies on the fact that the bits, the quantum bits, they are in superposition, they are entangled. So you can explore us source space much faster compared to classical computing. Even if you have multiple processors and you can have multiple threads in the day, there is so many threads you can put on compared to a quantum machine. So I do see an opportunity here from the perspective which is there are certain problems in the cybersecurity domain, software engineering domain, that are computationally expensive because you have a very large storage space. And solving that problem using a classical computer is just. You have to put a of bunch bound right. You run that technique for let's say 24 hours and then you have to stop because there is so much things that you can compute in a given period of time. So what I have been excited about is exactly, uh, formulating problems that we have in security as an optimization problem such that a quantum computer can solve too for me in a much faster, um, in a much shorter period of time. And for example, one thing we have explored is that how we can formulate the problem of synthesizing exploits, which is given a vulnerability as input. I want to verify whether that vulnerability is a true positive or not. The way I can do that is by generating an input that can exactly trigger the malicious behavior. That's a uh, combinatorial problem in nature because you're trying every possible different combination to be able to find which one is going to trigger that vulnerability. And my job lately has been, well, how can actually reformulate this problem as an optimization problem that I can pass to uh, a quantum computer to solve. And we did publish a paper very recently, I think last year on exactly the preliminary results on it, which is, hey, we demonstrate, hey, look, theoretically if you formulate the problem this way, we would be able to solve this problem. The thing right now is that it's uh, theoretically it works. But the thing is that the quantum computers are still in early infancy in the sense that there are so many bits you can fit in a quantum computer right now compared to a classical computer that you have large amount of memory. For example, my laptop I think has three to gigabytes of memory. That's not something that a quantum computer has. You don't have that many bits, I think maybe a few thousand, uh, qubits you're going to have available. So there is. The scalability is not there yet. Not because the quantum computers cannot do it. It's because the hardware is not there yet. But the moment we have more powerful hardware, I'm betting that all of these, these problems that are really hard to solve in the classical domain because they take long to solve, would be now be solvable in a much shorter period of time using quantum computers. So that's kind of what my research is betting on, is that how can I leverage the power of quantum computers to solve the challenge that I have in the security domain. That's um, my next line of research.

Speaker A: How do you think of the offense versus defense dynamics? Because you hear that yes, quantum computing means quantum hacking, but it also means quantum defense. And I think that's what I'm hearing from you here.

Speaker B: Yes, correct.

Speaker A: But where is that balance of power? How do you see that folding out in the different scenarios and think through that? Because it's both. There's reason for cause for optimism and which is good to have that be our default, but also cause for pessimism and concern. So how do you think through that? Quantum defense versus quantum hacking?

Speaker B: Yep. I think it again, it's going back to the same issue that we have with LLMs. LLMs for defense, LLMs for offense. Right. So it's the same problem is that if a hacker has access to a quantum computer to break cryptography algorithms, that's bad. But as a defender, if I also have A uh, quantum computer can help me find vulnerabilities. I can also patch them faster. Again going back to what I said before, and I know I sound like a broken record at this point, but again it's uh, a race. Both of them are having access to the same tool, so to speak, and it's who wins that race first. Right. So, so in the end of the day it's optimistic and pessimistic in the sense that I have the optimism which is that technology is also available to the defense side, but I also understand that, well it's also available to the offense side. So what my goal as a researcher has been is that studying vulnerabilities extensively so that we can create defenses that prevent those attacks from happening to begin with. So my job has been is that try to stay ahead of the competition. And the competition here means the attack. Soccer. Mhm.

Speaker A: How do you think of how the democratization of this technology, and this isn't necessarily specific to quantum, but just more generally with cyber and uh, cyber defense. Cyber and as well as cyber attack, how does this impact the grassroots, the smaller players? Because on one hand the attack technology is being democratized to some degree. Um, ideally defense technology is being democratized quicker. The ideal scenario. But does it mean that say a small business or a small institution or player. Because you're at a very prestigious university but in a small town. Right. So these smaller institutions that are out there, they now have access to defensive technology that a, uh, few years ago only nation states had access to. That's the optimistic case. On the other hand, they're more vulnerable if they don't access that defense. They're more vulnerable to nation state level attackers that might be just small bands now, you know that even the hackers now have access to uh, uh, you know, nation state level offensive capabilities. How do you think through this problem set if you're like a smaller institution or a smaller player, or even a single player.

Speaker B: That's an excellent question. Because indeed if you want to use a large language model, for example, you're going to need a gpu and not only a gpu, it has to be a GPU that has enough memory to be able to support that model. Right. The larger the model, the more uh, GPU capabilities you need. So indeed it creates a strain on the smaller players because if they don't have access to those resources, that becomes a problem. Let's say you are a software vendor but you don't have a powerful GPU in house. How are you going to handle that? But There is a way that you can handle it, which is using the cloud. So you don't run your analysis locally, you're going to use the cloud, but in the end of the day you have to pick for access. Right? So you, for example, if you use OpenAI models, you have to pay per token. So for every token I use that the model generates or that I give as input, I have to pay a given amount, I don't know, 10 cents per 1 million tokens or whatever. And that introduces costs. Perhaps these costs are not a big deal for a big company, but for the smaller companies that's definitely a cost, uh, for them to use. And security is cost costly, period. Right. For you to uh, tackle security as part of their software development, either you're going to need people in house, security engineers in house to catch those vulnerabilities and patch them, or you're going to have a consulting company that you have hired to be able to identify and fix those vulnerabilities. And both approaches will come with costs. And it's indeed a challenging problem for the smaller players, which is how we can, can bring security in and also help the smaller ones that don't have access to resources. It is a challenge. Um, unfortunately I don't have a good answer on how we can solve that challenge, how we can make this technology more accessible. Because in the end of the day there are costs to use it. Um, but it's definitely a challenge. Security is costly. That's exactly why one of the reasons that security ended up becoming, ah, an afterthought is actually because of that, because there is uh, that pressure to, to the market release the product. Release the product and security becomes a feature on the side. Right. So let's make sure that release to the market and then security is just, we handle later when things happen. So.

Speaker A: So, professor, you've come a long way since your days in a, uh, small, small province in Brazil, correct? When you went into cybersecurity and software engineering and then eventually, you know, a PhD program and then leading a lab at a university. How does your family, what does your family react to that? I want to go into cybersecurity. What is your dad, mom think like uh, how they think through this.

Speaker B: That's an excellent point. I remember very vividly that uh, when I finished my undergrad I saw this scholarship, um, that I wanted to apply and I was excited to. It was uh, happy. She was happy. Oh my God. My daughter is going to study abroad. She's going to get a master's degree. That's great. My mom always, uh, incentivized me to pursue higher education, but she was definitely also sad with me leaving the country and pursuing all of that. Um, it's definitely something that is way over their heads, right? In the sense that it's like, ah, you know, my family did not grow up with, uh, computers, right. Because they are from a different generation and we were also lower middle class. So it's not like we have access to lots of resources. So it's working with, uh, models for them is just way over their heads. And in fact, my dad really don't even use. If we ask my dad's chatgpt, he's gonna be like, right.

Speaker A: So it's, uh, maybe that's the safest thing, actually.

Speaker B: Yeah. And I'm, I'm glad. But it is concerning because, um, oftentimes now the. I don't want to generalize, but they will be sharing all of this content that it's AI generated and they are not able to tell this is AI generated. I can't tell, hey, this is AI generated. This not. This is not real. This image you see here, it's not real. Of course, some of them are very obviously not real. Like a cat singing. It's clearly a cat cannot sing. It's. It's not real. But it's definitely concerning that some, they are using the technology, they're spreading these images, but they don't. Are not aware that they are from AI in not real. Right. So, but yeah, but long story short, yes, it was definitely, um, bittersweet for them to. To have their daughter abroad pursuing higher education in here and, but definitely away from, from them. So, yeah, yeah.

Speaker A: Do, do they understand like, like you're going to these conferences and speaking, publishing papers, coming on the Paradigm Shock podcast?

Speaker B: I think they do understand the gist of it, but not really.

Speaker A: Right.

Speaker B: Uh, because I think at the end of the day, uh, when I say, hey, I'm a professor, they probably think, oh, she just teaches, she goes to classrooms, she cheats, she teaches, she grades, and that's it. I don't think they truly understand the

Speaker A: nitty gritty details, but they don't get that you're a big deal now. But I'll say you're a big deal now. Your papers, you probably.

Speaker B: Yeah, uh, yeah, I probably know, but, but they're definitely proud. Proud of what I. How far I have came considering that, you know, I came from the smallest estate in Brazil. I came from lower middle, uh, class families, so it's not like I grew up with a chip on my shoulder, you know, and uh, how far I ended up going with, uh, the resources that I had. So they're definitely proud of that. And I'm also very grateful for them as well. For, for them, for the support they have given to me. They, they, they've done the best they could to, for me to be where I am nowadays. If it was not for. I'll not be here right now having this interview with you. So.

Speaker A: Well, you haven't just arrived. You got a long way to go. You're, you're, you're, you're, you're, you're a

Speaker B: young scholar, you know, and I'm definitely ambitious. So yes, I'm hoping to be, climb up the ladder, but yes.

Speaker A: So the World Cup's coming up. I've got to ask you this. So is, of course, is Brazil going to get past the quarterfinals? They can't seem to do that the last, uh, 20 years.

Speaker B: That's the million dollar question. Perhaps the most difficult question that I had on this podcast so far. Uh, I'm, um, hoping that Brazil makes to the finals. But, but the last time Brazil was, I was confident of Brazil was in 2002 when I was still a little child when they were on the World Cup. But yeah, it's been many, many years since Brazil won the World Cup.

Speaker A: I'm hopeful, but I.

Speaker B: Not confident. I'm hopeful, but not confident. I'll phrase it that way.

Speaker A: I remember that 2002 team that by the, I think there was one analysis done by, it may have been in some analytics company did an analysis and they said the Brazil 2002 team was the best, best world, uh, cup team, not just for Brazil, but for any country and better than all the other winners. Post Brazil, 1970. Post Pele. That was the best team that was a dominant team. And since then you've, uh, been knocked out in the quarterfinals every year. And I think some analysis says that if you win your group and England wins their group, you meet in the quarterfinals and that'll probably be tough, but both countries have a lot of baggage. So I've had English guests on this podcast that are, are as nervous as you, so.

Speaker B: Oh, yeah, but let's see. I'm hopeful, but let's see. I don't, Yeah, I don't know. I hope that Brazil makes to the final because I love watching the World cup when Brazil is playing, but, you know, one day at a time. But I'm really hopeful that Brazil makes it. But confident. No, sadly.

Speaker A: Well, Professor Joanna Santos, thank you so much for coming on the first video in person studio episode of Paradigm Shock. I think we're going to be hearing more from you in the future and your research. So um, we'll post in the show notes, ways people to see Your research, your GitHub page, your Google Scholar page, any other relevant resources. Um, and uh, thank you for coming on the show.

Speaker B: Thank you so much for inviting me and I'm also honored to be in the first uh, in person video recording. I'm so happy and thank you so much for the invitation. I had fun recording this episode.

Speaker A: I think you were the perfect guest for the first episode in, in video in person.

Speaker B: Thank you so much. And I'm hoping that in the near future I can come back as well with more results and hopefully more positive.

Speaker A: Probably the next major model release that has security issues which, you know, could be like tomorrow. But, but I, I, I suspect in the next year there'll be some other big news of some post mythos model that's being released or some other company releasing something, maybe a Chinese model that gets released into the wild. But ah, I shouldn't presume but you know, something like that's going to happen and uh, when that happens, we'll have you back on to talk about it.

Speaker B: Absolutely. And there is so many things going on in the AI domain that is really even hard to catch up every day. I'm just like, can we please just take a, let's just take a pause in here so I can catch up. Because it's hard. It's hard. It's definitely fast paced.