Fighting Fire with Fire: How CyberProof Is Automating Cyber Defense with Edy Almer

This transcript is produced using transcription software and reviewed for quality. Despite our best efforts, some passages may be incomplete or contain errors due to audio quality or software limitations. John Richards Welcome to Cyber Sentries from CyberProof on TruStory FM. I'm your host, John Richards.

Here we explore the transformative potential of AI, cloud, and cybersecurity, where rapid innovation meets the need for continuous vigilance. This episode is brought to you by CyberProof, a leading managed security services provider. Learn more at CyberProof.com.

On this episode, I'm joined by Edy Almer, who leads the Agentic product offering at CyberProof. We talk about how AI is moving cybersecurity beyond just detection to automating parts of the response itself, and the organizational hurdles that come from making such a shift. That speed is increasingly important as recent cybersecurity research from Anthropic is showing attacks are becoming less linear and more distributed. What does that mean for the way security teams detect, respond, and adapt?

Let's get into it. Hello everyone, and welcome to today's podcast. I'm super excited to have Edy Almer on the podcast, a former CPO and VP, currently really leading the Agentic offering at CyberProof. Edy, thank you so much for coming on the podcast.

Edy Almer Hey John, it's great to be here and great to talk about some AI. John Richards Yeah, and that intersection with security. So I talked to you slightly before we jumped on here, and you kind of hinted the reason you jumped at this is you were really excited about the opportunity to work in this agentic space. So can you tell me a little bit about your cybersecurity journey?

What led you here? And then what made you decide to make that jump really deep into the agentic pool, if you will? Edy Almer So I've done security from all sides. I've done endpoint at Symantec.

I've done network with AlgoSec. I've done SIEM specifically with LogPoint, who has a new name nowadays, but is still around. And the one frustrating thing with security is that you keep spending money and chasing the newest tools, and somehow you're always slightly more behind than you were last year. So as a product, I keep solving your problems better than last year, and we keep racing with those attackers and hoping for the best.

But the people who can really make a change are the service providers, because few organizations can build a team with the skill set that is needed to really defend a large organization. And most organizations wouldn't be willing to spend that kind of money. And for the people doing it, it will be kind of frustrating to sit even in a very large bank that gets a lot fewer interesting attacks than we get for several dozens of very large customers. So being at a service provider, and one that is willing to invest heavily into Agentic and that is closely partnering with Anthropic and with Microsoft and with Google, that's a dream come true for a product guy.

So being able to product manage that agentic transformation was an amazing opportunity, one that I couldn't pass up. John Richards You sound like a guy who likes challenges, because a lot of people I talk to say, "Oh, you go into a role like that early on to get that," because, as you said, you see things you don't see elsewhere, you get just a breadth of experience, and then they want to leave and go to one of those other places for a cushy job where you can relax a little, where it isn't all the crazy stuff that you see.

But you're on the other side. You're like, "No, I keep wanting to see new challenges. I want to be on the forefront." So I'm excited to hear about what these new challenges are that you're working to solve.

So what does the landscape look like right now that you're addressing? Edy Almer Even before the landscape, the challenges I started with were, "Hey, is this AI thing really good enough?" If you've been in cyber long enough, you'll remember there was a wave of AI in cyber roughly 10 years ago that kind of fizzled because it was not good enough, was not strong enough. All those UEBA vendors and all those statistical detections that maybe found something but created a lot of noise.

When we got started with the LLMs, everybody was still talking about hallucinations. I found out pretty quickly that I have a bunch of very, very good professionals, some of them a bit wary of AI because they've had similar experiences to me. They've seen some of the things not match up with the hype. Nothing ever matches up with the hype.

Hats off to cybersecurity marketers, they're amazing. I think Anthropic has done an amazing job in the past few months, and if we can meet together with them at even 80% of the expectations that have been created, we'll be in an amazing place. So that's one challenge. But the real challenge is adoption, getting people on board and using the agents that we're building for them, because those agents, they're not living in a vacuum, and they're not replacing the experts that are there - none of them yet.

So we need to make sure that we're building them in a way that, A, they're trusted, and B, they are used. So that was one challenge. The second challenge was, of course, getting them to be good enough in an absolute way - basically never hallucinating. And that has actually been an easier than expected success.

It's great that I have an amazing team, but I think that our approach and the harness helped move away from any hallucinations very early on. So from that point of view, we've done well. And now it's down to - as we are an organization that meets our customers where they are - maybe we got it great for Sentinel and we got it great for SecOps, but we still need to do it for Splunk, and we need to do it for QRadar, and we need to do it for every single combination of tools out there that our customers have.

Some of the things we're doing are tool-specific and creating great results for some tools, but other tools need to be fine-tuned and improved. So there's still significant work ahead of us, but we're past the initial adoption barrier. We already have most of our customers using at least one agent, and we have some customers using a big proportion of the agents that we made available. And the great thing is that every single one of them is seeing value.

John Richards You mentioned marketing announcements and stuff that come out of Anthropic, and the new feature sets as those advance. How has this agentic work and offering adjusted as models have improved? Is that a big impact on what's possible for your team, or are you building it in such a way that you're more dealing with the new challenges those bring? When you hear about new stuff coming out, is that like, "Oh yes, maybe we can do more than we could before," or is it, "Oh, here's a whole new wave of things I need to be prepared to handle"?

How connected are you to using those tools versus dealing with the challenges they bring? Edy Almer Great question, because what we're doing is we haven't placed a bet on a specific approach. What we've done is we've built a harness, especially for the investigation part, that allows us to continuously compare the results of various agents, various models, and our own human team - at least in the initial stages, slowly removing that, because we've gotten to a stage where the agentic ones are doing great and now we're more comparing agentic to agentic.

But we've built the harness so it can decide whether we're running something fully automatically in SOAR, or we're running it agentically end to end, or whether we're running it agentically and then handing it over to an expert who does the final leg. So, A, we don't introduce risk for any of our customers that are risk-averse. So actually our agentic response is sometimes slightly slower than our automatic response, because a human is always reviewing. In many cases - not in all cases anymore, but in many cases - a human still does the review.

And in other cases, we might actually need additional input that the agent doesn't have. For example, being able to ask a user whether they intended to do something when you're investigating really suspicious action on an endpoint. If you're able to make a phone call to the user and he tells you, "Oh yeah" - and I have many of those users in my group, my developers, getting those phone calls from my analysts quite often: "Hey, did you really mean to run that malware?" "Oh yeah."

Then the context is built, and after that, every new developer being added to the group of developers doing "bad things" - but with prior approval - it gets easier. John Richards So looking at the AI landscape where we're at right now, what are you seeing as where things are going, or the next step? What do teams need to be aware of? How important is it to be on top of these announcements and things that are coming out?

Edy Almer You have to be on top of the announcements, and you have to keep trying a huge number of things all the time. We're probably spending a significant percentage of our time simply testing new things - not necessarily writing new agents, but rather verifying: hey, if I'm using this model instead of that model, what are the guardrails like? What does the cost end up like? Are the outputs really complying with my requirements?

And is the amount of tokens I'm spending indeed what I'm expecting, or are there variations? There's a lot of that work going on, and I'm guessing the time is not far when we'll start doing that agentically. So whenever we have a new toy from Anthropic, we'll get it to work immediately across the board in all the use cases, and immediately verify how much better it is - not on a general benchmark, which is, you know, the best we can do. But the problem with benchmarks is it's very hard not to taint the training data with the benchmarks.

A benchmark really measures how well the model does on the benchmark, and this is one of the reasons we built our harness in-house. We're not testing against a very nice, very varied set of test cases - that would be a benchmark. We're taking the benchmark into account, but what we're really doing is running this on our own use cases, and always on new incoming use cases. We could afford to do it over a significant enough time and a significant enough number of cases that we're getting good results.

So this is what we're usually doing. And we've had a recent announcement of partnering with Anthropic, and we're already Microsoft and Google partners. So my developers are having a field day. John Richards Yeah, talk about wanting new challenges.

Edy Almer Yeah. "Hey, I need you back here, eyes on me." But yeah, they're having fun. John Richards So for folks out there, what do you see as the big gaps right now that people need to be paying attention to?

Where are new attacks starting to come from, or, as AI starts to open up new avenues or new scales of attacks that are available, what can people expect, and what should they be doing to deal with this? Edy Almer One of the challenges around AI security is that there's still a limited body of attacks. It's not like ransomware, where almost everybody has experienced it at some stage. You might not experience it every day, or every month, or even every year at a well-defended organization, but many of the practitioners have definitely experienced ransomware.

But when you're looking at the AI attacks, we're still looking like OT in 2018, when you could talk about pretty much one attack, or maybe two, and you had to make it work. So there's a bunch of attacks now. It's not really one or two - there's more of them already, but they're still few and far between. And every time somebody is publishing public data about those, it's interesting.

And again, Anthropic - I think they did a great job. It was somewhat shadowed by the Mythos announcement, but what they actually did is they published data and statistical data about 800 users that they've suspended for using their previous versions of the model to actually generate attacks. So first of all, even the previous versions before Mythos were quite capable, and quite a few users tried to build actual attacks. The interesting part there, maybe two things: the most dangerous attackers, the attackers that got furthest, were the ones that were letting the agent be most autonomous.

So you can see some attackers building the first stage and then stopping, having a human review it, moving to the next stage - also trying to be slow and possibly stay under the radar in terms of detection - whereas others were doing things completely autonomously. And it's the autonomous attacks that are really the most interesting. Because what happened there - and there were a bunch of those that Anthropic is talking about - those attackers were in no way linear, like human attackers are.

When we're talking about a threat actor, we're talking about: he does this reconnaissance work, and then he does this to actually get in, and he's using those three techniques to get persistence, and then he's using those four techniques to move laterally - but you can draw a very nice track of what that attacker went through. John Richards Mm. Yeah, makes sense. Edy Almer This is very helpful for detection, very helpful for prediction, very helpful for doing threat-based work.

And all of a sudden you have an agent, and the agent has a full library of TTPs at their disposal, and they're not running one path - they're trying to run five, or ten, or as many as they can afford, in parallel. They get very noisy and very messy, and there's definitely not a clear line connecting the successful stages. You might get that far in one path and then that path gets whacked, but you have three other paths that, by that time, were able to progress further, because there's that much that humans can do at that speed.

So you have to fight fire with fire. Some of the techniques that we've built for detecting humans - even the best humans, the nation-state attackers - they're not a good fit for agentic attacks, at least not the ones that Anthropic was kind enough to see and share with the rest of us. When I'm looking at my customers, we haven't yet seen attacks that are end-to-end agentic. We've seen stages - maybe it's hard to tell sometimes - but as long as there's a human with hands on the keyboard, you can tell it's a human with hands on the keyboard.

If he has a co-pilot sitting next to him providing advice on the best vulnerabilities and the best ways of taking the next step, great. But we're still seeing those attacks are pretty much linear compared to this stuff that Anthropic is sharing, which is very interesting. And the fact that it's noisy doesn't mean you can react fast enough to stop it. John Richards See, I was hoping that would be the clue to give it away.

Edy Almer Yeah. You know, take another example that's been in the news recently - the jailbreak that happened and attacked Hugging Face. That agent was also running its attack on its own. It was trying so many things until it found the thing.

We're hearing the story about what succeeded, and it sounds like it's linear, but what the agent really did was try a lot of things until it found what was there. And the problem is that the defenders had to sift through all of that, and they needed an AI model to help them, because they couldn't keep up with the rate. They weren't automated enough in terms of triage and finding that one path that was happening, and of understanding and reverse-engineering the steps taken. So where things are going is we'll have to automate the defense, because the attack is becoming automated.

Even if the examples for now are few and far between, there are more and more of them. And even with the guardrails that we have for Anthropic, for OpenAI, we still have open-source models from China and elsewhere that are almost as capable - or slightly more, depending on who you're listening to - as the frontier models. So it's just a question of time until we're seeing more and more of those attacks. Cybersecurity - those are interesting times ahead for our team.

John Richards Yeah, I'm reminded of some of the missile defense systems, where the result wasn't to build better missiles to get around them - it was to fire so many missiles that it just couldn't keep up. And it feels a little bit like we're shifting into that, where these autonomous things are hitting so many areas that trying to keep up gets overwhelming, and you're like, "I need my own kind of similar response to be able to handle that." Edy Almer And when you're talking about that - a reminder, I'm Israeli, so I'm looking up, trying to hear if there's anything - we've definitely experienced the method of basically sending that much ammunition, that many different points, that the defense systems get saturated.

And what we need is basically not just automation, but automation at a price point and at a volume that will be able to deal with that kind of attack. John Richards Have you started to look into what it looks like to do that? I know it's very early stages, but of course, once you hear about it, it starts to get buzzed, and you'll have copycats and things like that starting to occur. So as you think about that, is there anything on the team, or maybe you're seeing in other groups, where it's like, "Okay, do we need to build something very specifically to look at this and try to handle it?"

Or is it a reaction once it starts happening - you task something to start analyzing that noise and try to get ahead of it as a reactionary step? Or is it so early that we're just kind of waiting to see where this will develop, so we know what kind of defenses might make sense? Edy Almer Interesting question. So I have a specific developer who's building attack simulations specifically for making sure that our detections are all healthy.

When you're looking at that Anthropic attack I was discussing earlier, we can fairly easily ask an agent to chain all of those together, run a bunch of them in parallel, throw them at our system, and see how we're doing. No, we haven't done it yet, but it's a great idea, and we will - you've just made another developer happy. John Richards Yeah, well, I like making developers happy, so - Edy Almer That's an interesting one. John Richards So that's good to hear.

So what's next for your team? What are you guys focused on right now, and what should people be checking out at CyberProof? Edy Almer Getting to a hundred percent automated, or very close to it, should be the holy grail, and we're getting there for our customers. CyberProof right now is probably automated to 96 or 97%, and we're starting to automate some of the responses that are keeping us from going to a hundred percent.

When, for example, you're going to suspend a user for 15 minutes to finish an investigation, it's no longer a discussion with the CISO - it's a discussion with the CIO and the CFO and the CEO of basically, "Hey, are you going to get in the way of my employees, or are you going to protect them?" Those are the steps that we know are hard for our customers to get through - basically approving those automated responses, some of them easier than others, some of them quite hard. That's one step we're taking, and we're continuing to improve detection and put everything into an automated flow.

And when I say everything - detection - there's a bunch of vendors out there doing AI SOC or agentic SOC, and they've automated specifically the investigation part, but many of them have shied away, for good reason, from fully automating the response part, because not every customer is happy to do that - not with an AI vendor they don't trust yet. So it's never step one. And then there's a bunch of other agents around it - the ones building detections, because if your detections aren't good enough, the investigations can only do so much.

Basically, using threat intelligence in a good way, so that every time there's a new campaign, you immediately threat-hunt, immediately create new detections, immediately investigate the findings of those threat hunts, and immediately remediate and block whatever needs blocking - or be able to go to your board and say, "Yeah, sure, there's a SolarWinds-like attack happening right now, but we've validated they're not on our network, and we know we're protected from the point of view of our tool configurations and our SIEM detection," and being able to say that mere hours after the first news of that attack.

So that's what our customers can expect today. And those feedback loops that we need to close to make all of that not just possible, but stable and continuously improving - that's where the effort goes now. So the main path is already working, and now it's the feedback path that's being improved. John Richards Back to your point about how some of the challenge here is very human - are you comfortable moving to automating those responses?

Are there use cases or numbers you go to when somebody's asking, "Am I ready to move to this second stage and automate more?" - something that says, "No, here's the value you get. Even if you're willing to trust the vendor, here's why it's worth doing that." Do you have any stories or things there that you use?

Edy Almer This is sometimes more organizational than it is technical. You might even have a completely believable story with good statistics and a very reasonable, reversible blocking mechanism - if you make any error, you can very quickly roll back that error and let people work. But we're seeing many organizations - maybe not as many as we used to - happy to start doing automatic response, because as an MDR, our responsibility usually is to detect and manage everything that leads to the detection.

Not long ago, it would then be handed over to the customer to deal with the incident response and remediation. When it took us a few hours to do it, and it took them a few hours to remediate, okay. But now, when it takes us five to seven minutes to do all our stuff, and it still takes them a few hours, then it's putting pressure on the process. We still have customers where the team doing the response isn't even the security team.

The security team hands over to an IT team, and they're the ones suspending a user, if they're happy with what they're seeing. And then you might have not just one person who needs to agree with you, but possibly more than one. And that's one of the areas where there's a pretty significant disconnect between what can be done. At the best, we're down to under 10 minutes - we can detect a completely unknown threat, remediate it, put controls in place to make sure we're catching it next time, and report.

That's at one edge. And at the other edge, we can have a completely standard case of impossible travel, where somebody's logging in from a country where they shouldn't be, but it happens to be your CFO's login. And nobody in the organization is comfortable bugging him, and it might take a few hours until that's remediated. John Richards Yeah, the tech - and AI itself - can't solve these human challenges.

But it sounds like part of your strategy is to improve what can be automated so quickly and effectively that it becomes more glaring the more steps you put in front of it. So now you're like, "Are you really going to wait hours and be at risk, versus letting this automation happen?" Edy Almer We're helping our customers convince their organizations that it's the right thing to do. I don't have many direct customers - the security guys - who are saying, "Oh no, don't do this."

What they're saying is, "Yes, this makes a lot of sense, but I need to get approvals from people who don't report to me to do the things you're suggesting - which make perfect sense." John Richards Well, Edy, thank you so much for coming on here. This has been fascinating. I've loved hearing about the world from your viewpoint, and the dangers that are coming, but also hearing that folks are paying really smart attention to those and figuring out ways to get around that.

Before we wrap up here, where can folks connect with you? Anything you recommend they check out over at CyberProof? Where can they check out this product or learn more? Edy Almer I'll be very happy if you visit our Agentic webpage, or if you have any questions for me.

My email is edy.almer@cyberproof.com. edy.

almer@cyberproof.com - and I'll be happy to chat with any of you who would like to chat. John Richards Well, awesome. I'll make sure we put the link in the show notes, not your email - they've got to listen if they want to get connected with you, so you don't get a bunch of bots coming to you.

But thank you for that information, Edy. It was a pleasure having you on here. Edy Almer Thank you, John. Loved it.

Pleasure talking to you. John Richards This podcast is made possible by CyberProof, a leading co-managed security services provider, helping organizations manage cyber risk through advanced threat intelligence, exposure management, and cloud security. From proactive threat hunting to managed detection and response, CyberProof helps enterprises reduce risk, improve resilience, and stay ahead of emerging threats. Learn more at CyberProof.

com. Thank you for tuning in to Cyber Sentries. I'm your host, John Richards. This has been a production of TruStory FM.

Audio engineering by Andy Nelson. Music by Amit Sagie. You can find all the links in the show notes. We appreciate you downloading and listening to this show.

Take a moment and leave a like and review. It helps us get the word out. We'll be back next month right here on Cyber Sentries.