How Hackers Are Using AI Right Now: Faster Attacks, Smarter Malware, and a New Arms Race

Speaker A: What if the malware targeting your organization today was built with AI and specifically engineered to confuse the tools that your analysts use to fight it? Because this week, that's exactly what researchers found, and it's what we're unpacking on today's episode.

Speaker B: Welcome to Shadow Talk, a cybersecurity podcast powered by ReliaQuest, the leader in agentic AI security operations. I'm Brandon Serato, Director of Gray Matter Operations.

Speaker A: And I'm John Diligent, Threat Intelligence Analyst. And this week, our threat research team published a report on how threat actors are using AI and real world attacks. We previewed a little bit of it in last week's episode, but today we're really deep diving this report, and it includes insights that we've taken from the incidents investigated firsthand across our customer base, as well as from research on the Dark Web. And what we found might surprise you, that threat actors are using AI not to replace themselves, but to instead work cheaper, build faster, scale more, and make attacks harder to stop. That's the threat that we're up against.

Speaker B: And there's some key areas that, you know, where we're seeing AI show up and how it's being used. Everything from initial access and phishing to social engineering and malicious code generation.

Speaker A: Yeah, and on top of that, the report is very timely as this week a breaking story landed that shows exactly where AI is heading, specifically a newly discovered piece of Mac OS malware called Gaslight, which doesn't just steal data, it embeds fabricated system error messages to confuse analysis into thinking it's benign.

Speaker B: And so today we're, we're going to uncover, you know, what our research has found across those attack areas and what that Gaslight story tells us about the threat landscape and the concrete steps that, you know, organizations that are listening need to take to stay ahead of this threat. So, John, let's kick things right off. Um, what are the details from our report that are worth sharing?

Speaker A: Yeah, let's jump in here. So, in our report, there are a few key areas where we saw threat actors using AI, and that is include phishing and social engineering, initial access acceleration, malicious tooling and code generation, identity fabrication, and AI being a lure in and of itself. Now, uh, let's break these down and start with phishing and social engineering. Attackers are using AI to easily impersonate brands, and then they layer in automation to quickly create massive campaigns for both email and brand voice. Phishing, as we mentioned at the start, AI isn't rewriting these tactics, but it's enhancing the speed of them, as well as the capabilities. Additionally, AI polishes the emails and the phishing web pages so that there are fewer warning flags that we would traditionally see and that would alert users and defenders of malicious activity like the typos and the grammar errors. Now we spoke last week about how AI phishing toolkits are enabling large scale campaigns. And our report highlights, uh, a very specific campaign. We saw thousands of AI assisted phishing web pages targeting our customers, with as many as 40 being created simultaneously through phishing toolkits.

Speaker B: And so, John, like, so what we're saying is the AI, you know, systems here are helping these attackers just easily impersonate the brands once they just essentially supply the URL of, of the victim that they're looking to impersonate. Is that right?

Speaker A: Yeah, that's exactly it. You know, they put that URL in and the AI quickly recreates the imagery and the branding of whatever they're trying to impersonate. And in the campaign that we saw against our customers, users received phishing emails with malicious links. The links directed them to payment card harvesters that were designed to mimic legitimate travel and hospitality businesses, helping to ensure that trust of the user.

Speaker B: Interesting. And the next area of how we're seeing AI show up you mentioned was social engineering. Um, memory. Serves me right. Wasn't Shiny Hunters utilizing this some way to enhance their tactics?

Speaker A: Yeah, they absolutely were. There were reports that stated Shiny Hunters used voice agent tools to adapt their conversations in real time. One of the major campaigns that the group orchestrated was calling target users over the phone, often directly on their personal devices, and then getting them to manually navigate to device code phishing sites. Uh, now the group has likely used AI voice agents to help them facilitate these phone calls and even be able to respond to targets in different languages.

Speaker B: Yeah, and I think I remember similar reports of uh, you know, another threat acting group. I'm forgetting exactly who, but using AI to essentially impersonate those compromised users, like when they were able to gain access to an email inbox. M. You know, those actors were essentially utilizing uh, AI to help analyze what they, you know, have landed on. Um, the, the information that they, they're trying to steal, summarize the email history and you know, craft really targeted and highly personalized follow on phishing lures to essentially target the next victims of those contacts that the original victim had.

Speaker A: Yeah, that's, that's another great example of how AI is being used. And you know, a key point I want to hone in on that is, you know, that AI analysis allows Them to quickly stem more and more phishing from those attacks after one compromised user becomes more so. Next topic, and this is one many of our listeners are likely familiar with. But North Korean threat actors have been using AI to build full professional Personas that they use to pose as remote IT workers and get hired by Western companies, ultimately collecting earnings for their regime. Now they use AI, manipulated LinkedIn photos, LLM generated resumes, and video deepfakes during these attacks. The next area of AI use is around malicious tooling and code generation. And we're seeing AI leave fingerprints on malicious code across incidents. AI generated malware is showing up with patterns that indicate LLM authorship, things like over explained comments, clean formatting, and a parallel structure that narrates what the code is doing rather than why. And we've seen this across multiple campaigns linking things back to shiny hunters. There was a credential harvester that was linked to the group that match the same pattern of AI generation. We also observe that pattern in the web shells that were deployed following the Zero Day and Sapnet Weaver that our team discovered. CVE 2025 31324.

Speaker B: Yeah, John, and that's a, that's a significant one for many reasons, but, and I know we've discussed this a bit on the podcast in the past, but just help walk us through again, like what made the deployments of those particular, um, that, that SAP netweaver vulnerability that we discovered so significant.

Speaker A: Yeah, absolutely, and we'll, we'll keep it very targeted here. So across multiple SAP netweaver incidents, we saw patterns that would be difficult for manual operators alone. This included identical payloads with randomized file names being deployed quickly across hosts and dropped into the same directory on one host. There was just 60 seconds between the web shell being dropped and the first reconnaissance commands. Manual pivoting and execution at that speed just isn't plausible. Automation and AI are enhancing these attacks. AI is used to generate the functional component, in this case the web shell, and then the automation handles deployment and the early post exploitation. And that's another example of how AI is being applied in real intrusions today. Not replacing the attack chain, but enhancing it. Now, AI is also being used to vary or pad code between deployments. So even when the behavior stays the same, the payload is unique each time. And that directly undermines signature and has based detection and puts more weight on behavioral and runtime controls.

Speaker B: Yeah, and that really ties into, you know, initial access acceleration too here. Right, because, you know, weren't we also observing something similar happening with a lot of the clicks fix campaigns that we were tracking.

Speaker A: Yeah, definitely we were. And just as a reminder, Qlik fix is that copy paste abuse malware delivery technique. And specifically we saw signs of AI in the deep load malware which used thousands of AI generated fake variables to bury its actual payload and overwhelm static scanners. That same principle, AI isn't the weapon, it's the tool that's making the weapon better. Now, just last week, as we mentioned, there was a striking example of of AI coded defense evasion a newly discovered Mac OS malware called Gaslight, which has been attributed to North Korean threat actors. Instead of this tool just evading security controls, Gaslight actively targets the AI assisted analysis pipelines that analysts often rely on as embedded within the implant is A block of 38 fabricated system error messages, including things like fake token expiry errors out of memory warnings, disk exhaustion notices. All these things are designed to make an LLM assisted triage agent doubt its own session and abort the analysis entirely. Now, on top of that, notably, the Python based information stealer component within the malware also showed signs of being LLM, um, generated.

Speaker B: Yeah, and depending on the way we're looking at it, that's a bit incredible. Right here we have a piece of malware that was very much likely built with AI and it's actively using AI techniques to defeat AI based detection. I hear that and I hope our listeners hear that, but that's a bit of AI inception.

Speaker A: Yeah, it definitely is. And like we said earlier, it's a sign of where AI in attacks is moving towards. So the last area we want to cover from the report is AI being the lure itself. Now, in a single month, we saw two malware campaigns use impersonation of AI to deliver malware. They use Claude themed VS code extensions and other fake installation guides to exploit developers. Now, on Windows, one variant quietly created defender exclusions so that it would re execute every time VS code launched. Whereas on Mac os, attackers reproduced Anthropic's legitimate installation commands for homebrew style installations to trick users into thinking it was the legitimate installation. All right, so let's take a step back. We've covered a lot of different areas that threat actors are using AI, including the phishing, the social engineering and the code generation. But Brandon, can you, uh, help us step back and take a look at all of it? What's the structural challenges for organizations on the receiving end of this?

Speaker B: Yeah, what kind of sticks out and may strike some others here too, is that AI really hasn't created an entirely New set of attack categories. Right. It's taking three problems, us as defenders that we've actually like for a long time been continuously losing ground on and made them materially worse. Right. And these three areas are really come down to speed, scale and our ability to detect, you know, on the speed, uh, component. And I just want to be clear, this is much about automation as it is about AI. You know, the attacker isn't going through and manually executing each of these steps. Now, you know, what's happening is speed, say with AI is generating a payload, automation is now handling the deployment and these pre built post exploitation playbooks are firing across multiple hosts in seconds. In that SAP netweaver incident that we were referring to, we observed firsthand within 60 seconds between that web shell first being uh, deployed, uh, the first reconnaissance command running, uh, shortly after in that 60 second window. And likewise, we've seen threat actors moving laterally in environments in as little as four minutes and exfiltrating in as little as six. There's no place or way that a manual review can keep pace with this threat. The scale, uh, aspect, it really compounds that speed problem, um, significantly. In the phishing campaigns we saw, there were over thousands different pages that were created. You know, likewise, we've been talking a lot and the industry has been really, really um, uh, interested in the evolution of Mythos and discovering all these zero days faster than organizations can fix them. So it's only a matter of time before those are scaled through automation to target organizations in mass. This won't just be more patches for your team to fix, but also more alerts to investigate as a result. And that's the ability, and the last part here, uh, on detection and the ability rather to detect malicious activity that mirrors legitimate operations, which has always been in a lot of ways one of the most structurally difficult of these three that we've discussed at the phishing layer, ah, the tells that both users and security tools have been trained to catch those typos, poor grammar, awkward phrasing, um, et cetera, are largely gone now as a result of AI generated content. You know, at that malware layer, you know, static detection is being deliberately, you know, degraded. Kind of going back to that Gaslight example that you were mentioning earlier, John, you know, malware is now padding itself with thousands of different AI generated fake variables to overwhelm these static scanners. And the payloads that are being regenerated are being regenerated to be unique to every single deployment. And at that analysis layer, again, um, going back to the gaslight example, uh, that we were talking through, it's not just evading the detection, it's actively trying to deceive the AI assisted triage tool that is being used. So really, really interesting stuff here.

Speaker A: Yeah, it definitely is. And just to kind of summarize everything in a quick sentence, ultimately there are more threats that are moving towards critical damage faster and are more likely to get past organizations defenses, more threats moving

Speaker B: faster, built to beat your current defenses at every layer. After the break, we get into what it actually takes to fight back and why the answer has to match the speed of the problem.

Speaker A: Shadow Talk is brought to you by ReliaQuest, the global leader in AI cybersecurity. ReliaQuest helps enterprise cybersecurity teams contain threats in many minutes. With its agentic AI security operations platform, Gray Matter, ReliaQuest makes security possible for the most trusted enterprise brands in the world. Learn more@ReliaQuest.com

Speaker B: all right, so welcome back. Um, now let's get into how organizations actually need to defend. John, you had mentioned the volume, right. Of threats. You know, it's at an all time high and how does, how does someone go about, you know, fixing what is primary here?

Speaker A: Yeah. So number one, and we've said this before, you need to fight AI with AI. The volume of alerts that analysts need to triage is at an all time high. It's driven by these AI generated tools like phishing, emails, web pages and malware. Then it's being scaled by automation. Organizations need AI and automation built directly into their workflow, embedded in the process, just like the threat actors are doing. And really the organizations that are able to free up this triage work from their analysts, which is really never ending and growing in volume, are the same ones who can focus on the proactive work before the next wave hits.

Speaker B: Yeah, and the numbers really help back that point up. Right. You know, back in 2025, you know, we called out, um, on a, on a podcast earlier in the year, you know, where external research was again showing that, that SOC teams were receiving on average, you know, 960 alerts, you know, a day. Uh, and 40% of them were going completely uninvestigated. So just imagine what that volume is in in today's world.

Speaker A: Yeah, again, only, only getting worse. So the next major challenge, as we mentioned here, is speed of the attackers. Attackers are moving from initial to post access exploitation in seconds. So how do defenders close that gap, Brandon?

Speaker B: So I think the answer here is twofold. Uh, you know, first you need to be able to detect as early as possible, you know, ideally, uh, at the source or in transit, um, as we refer to it here at Relyquest, you know, before the threat reaches its actual target or objective. Catching a packet, in this case at the gateway, or flagging a malicious command when executed are very different problems than trying to contain, say, a web shell that has been deployed across six different hosts.

Speaker A: Yeah, they definitely are. And to layer in, detecting its source has secondary benefits to your organization and the fact that it reduces cost, not only are you able to detect faster, but you're also reducing the amount of logs that need to go into costly storage solutions like sims.

Speaker B: Yep, and that's exactly right, John. Um, the second part though is the autonomous response component. Right. So once a threat is confirmed and the containment steps, such as, you know, maybe isolating a host or revoking a token, you know, deregistering a device, etcetera, Those actions need to happen at machine speed within minutes. Organizations that are using and defaulting to a manual analysis and containment, um, having an average measured, uh, in hours, uh, are losing ground here. AI driven action isn't just a nice to have here. It's truthfully the only realistic way to match the pace of the adversary.

Speaker A: Yeah, it certainly is. And it's important to understand that the autonomous response doesn't mean removing human judgment from the loop, just like we're seeing the threat actors do. It means making sure that by the time your analysts begin to review the incidents, the host is already isolated so the threat cannot spread further or the token has been revoked so that the account cannot be abused more. Um, now the final challenge here is detection. So Brandon, what can organizations do there?

Speaker B: Yeah, and this is where defense in depth becomes critical. Right. So if attackers are regenerating unique payloads to uh, defeat the static scanners and, or they're disabling security tools mid deployment and feeding fabricated messages to those analysis pipelines. You know, no single detection is going to hold, ah, hold on its own.

Speaker A: Right.

Speaker B: There's no silver bullet there. So the answer is, you know, layering in diverse, you know, sets of telemetry, you know, uh, endpoint network, email identity cloud, right, all of these different disparate, um, sources so that an attacker evading, um, or disabling one of those doesn't blind you entirely. You know, then obviously the ability to utilize, you know, some, some AI, um, assisted correlation across these sources so that you can catch the patterns that can't be seen by just an individual tool.

Speaker A: Yeah, exactly. And to that point, research shows that organizations who have highly fragmented and siloed security tools spend 40% more on operational labor than those with correlated coverage. Not to mention the speed that's cost in the investigation and remediation cycle as you tool hop, huh, across the different tools.

Speaker B: Yeah, exactly. You know, so ideally, you know, that correlation of tools should extend beyond your own environment. Right. And what I really mean here is like benefiting from what we've spoken about in episodes past of what a true network effect looks like. So that paired with the importance of including multiple different intelligence outlets, um, from as many different sources as possible and being able to curate, uh, as well from the deep and dark web. So with that, uh, let's close out and what exactly you should take from today's episode and what you should do with it. Uh, starting as early as tomorrow, I'll start on the leadership side and then John, um, taking the practitioner piece. So for security leaders, today's episode is really highlighting a challenge that goes beyond having the right tools in place. It's about whether your team can actually act on them when it matters. The volumes of alerts is at an all time high and that volume is creating a genuine alert fatigue problem. A problem that has existed as long as time just now getting worse. So the honest question to put into your team isn't just are we detecting threats, it's how long does it take to us to identify and contain a critical alert. Right. And then I'll ask that same question about a high severity one as well. Same thing for a medium, same thing for low. If they are measured in hours or days, that gap is exactly where attackers will thrive. The tomorrow morning action for leaders. Pull your team's mean time to contain for critical and high severity alerts for just the last 30 days and put those numbers side by side. If there's a significant gap or if getting that answer requires more than one conversation, that's your signal. The question to bring back to your team is is straightforward. When in our triage workflow, could AI absorb the volume work so analysts have capacity for the decisions that actually require human judgment?

Speaker A: Yeah, those are some good recommendations. And now for the practitioners here, I think the most important structural defense against AI assisted attacks is the source diversity. You know, we mentioned that deep load is built to beat the static scanners. Gaslight is built to mislead analysis tools. So neither of these techniques defeats a team that's correlating across sources simultaneously. As we mentioned, defense in depth means an attacker evading or disabling any single log source doesn't blind you entirely. But that Source coverage is only the first step and it only matters if the alerts are firing fast enough to be actionable. So having logs from five different sources doesn't help if there's a four hour lag between an event and an alert in your most critical source. So the real question here is what is the actual latency between an event occurring in your most sensitive log sources and it being analyzed? If you can't detect in minutes, then you certainly can't contain in minutes. So the tomorrow morning action that I have for practitioners is to run a tabletop based exercise that pairs attacker speed with your actual detection speed. You can take a campaign from any of our recent Shadow Talk episodes and match each step against your real world detections with analysis and your remediation structure. Now clock how long each step would realistically take to surface an alert and which log sources are ingesting slower than the attackers are moving. That's your starting point. So attackers have always adapted and now AI is helping them move faster, scale operations and become harder to detect. That's all for this week. Thanks for listening to another episode of shoutout Talk. If you want to read more about the AI report, a link is available in our, uh, link tree below. And if you'd like to get in touch, we're just an email away. You can send your questions to shoutouttalkeliaquest.com please don't forget to subscribe so you get next week's episode delivered direct to your podcast platform of choice. We really appreciate it if you can rate and review shoutoutalk wherever you listen. It will make a huge difference and help us reach new listeners. We'll be back next week with another episode of Shadow Talk.