The REDCap Attack that Phishing-Resistant MFA Could Have Stopped

Can you imagine finding out that a Chinese state-sponsored hacker group spent over a year inside your network and your MFA didn't do a single thing? That's exactly what happened to some academic and medical research institutions, and why phishing-resistant MFA isn't optional anymore. Today, Prasanna and I break down exactly how attackers harvested credentials, hijacked REDCap, uh, which is a thing for research and education, Google Workspace with fake compliant rules, and exfiltrated sensitive research data for months without anybody noticing.

We'll also talk about what you need to do right now so that this doesn't happen to you. Here we turn admins into cyber recovery heroes. This is the Backup Wrap Up. Welcome to the Backup Wrap Up.

I'm your host W. Curtis Preston. I have with me a guy who is super jealous of the fact that in the last couple of weeks I have used, I think, every tool that I own. Isn't that true, Prasanna?

the one that I had you buy It, that is true. a significant amount of use, of the one that you basically really talked me into it, and I was like, "What do I need one of those for?" And now I have it. We're… What?

I found you the deal You did find me a deal. of course that, that, the tool we're talking about at this moment is the planer. and if you've never had a planer, it's a, it's an amazing device, but it's if you've never used one before, you don't know why you would want one. And then once use get one, "Oh my gosh."

you're like, "How did I ever live my life without this?" like the air fryer. It's like the air fryer of tools. I don't know about an air fryer meaning you don't know if an air fryer is awesome?

Or if I would justify having an air fryer that's because you've never had an air fryer. See? This is what I'm talking about. That and you know what else?

The West Wing. if you would just watch The West Wing. so wait, so here's another interesting thing about Prasanna. Prasanna owns, because you got a deal on Apple, like Apple, what is it?

Apple Plus? Yeah TV, and you bought the entire series for some ridiculous price, right? Like, how much was it? It was like $30 or something yeah.

and that was, like, a year ago at least? it might be going on a year and a half Yeah, and you've yet to see a single episode because you know that the moment you do, you're gonna get hooked and you're gonna, you're gonna here's my problem though, Curtis, is this is not atypical for me where I will buy something and it will sit Yeah. for years this is something you and I share Yes. I w- so recently I've gotten back into watching physical media, right?

Yeah. and all the rest. Yeah I had some which were still sealed for the last, 15 years that had never been Wow. Yeah.

that was kinda like me in the wood shop with, Yeah some of the, my dust collection, and now that's my new obsession, is my dust collection. Yeah. Anyway, people don't want… They won't, they don't wanna hear about this. They wanna hear about, Google.

Are you sure? Are you sure? yeah. so this is a, this was, this is another one of those stories where you're like, "Man," like 'cause it, it's another, living off the land attack where Wh- what's living off land for our Yeah.

So basically, living off the land is leveraging, i- is an attack that leverages y- the tools that you have in your arsenal, against you basically, right? and, the, what happened here and, so first off, who are we, talking about? Google's, their threat intelligence group, they have published a really big report which we'll link, in the show notes, that really detailed, I have to give them props, that they really detailed what happened here. It's, it's actually a Chinese, PRC-sponsored attacker, threat actor called UNC6508.

That is Is worst name ever sometimes they have these Titles, nerd titles, I don't know, labels yeah. of these attacks. But then the same organization, like what we might know externally as LockBit or we'll take whatever Yeah. what's the name of… ones Yeah, what's the name of the one, the one that did the last attack that we covered?

Shiny hunters It was the Canvas. Go the, so we didn't cover it, I'm sorry. It was the last attack you and I talked about. Yes and it was the attack of Canvas, which is a learning platform, and, there, because I saw an article that said it was the same, it was the same group.

I didn't check that, but, Google is calling them UNC6508. and they targeted mainly, North American institutions, academic, medical, things like that. And via this tool, called REDCap, which is a Research Electronic Data Capture, it's a web-based platform, and, they, th- which has a unique attribute that allows you to run multiple versions of the software simultaneously. And so even though you may be upgrading and patching more recent versions of the software, if there's a, an exploit against an older version of the software, they, they would be able to do it, and that's what happened here.

And once again, it started with harvested credentials. we don't know where… do they get? they were, it was an admin. it was an admin's credentials, but the, but again, this is they broke some of the rules, and we're gonna talk, the meaning the victims.

they, broke some of the rules that, that we talk about, and we're gonna cover them in the, that, that had they not done that, it would've been a different, the attack would've been different. but the scary thing about this is that they put in this, this malware that's called Infinite, Infinite.red. it sounds to me like that sounds like it's a s- a specific malware that they wrote just for REDCap, and it sat there for how long, Prasanna?

It looks like from what I could tell this happened like November 2023 Good. Yeah, it looks like literally they got in, and then they… this is we talk about dwell time. We talk about, just to go back to backups here for a minute, one of the real problems with these types of attacks is that sometimes having a really good backup isn't going to be enough because, what… If they've been in there for over a year, like a year and a half, maybe even more, probably isn't that far back.

no, right? You're not gonna, you're not gonna have backups that don't have this. And they… And what they did was they leveraged these, what do you call it? v- vulnerable versions of RedCap to harvest their credentials, and then, and then they somehow… But wait, go ahead.

Go ahead But even when they harvest the cred- credentials, it's not like they wrote it to a file. They wrote it back to a red capped database, Yeah. Yeah. end product database.

Yeah, they wrote it to talk about living off the land, right? and then they intercepted RedCap's upgrade process, so it then injected this malware into every new version, right? even if you were trying to patch things, you would get the new version, right? It's an interesting question.

I know we talk about patching and, making sure everything's up to date and having central IT. Do you think a lot of these issues stem to the, from the fact that these were, like, IT or software packages targeted at academia and researchers who sometimes pull things together on their own? It reminds me of, the shadow IT phenomenon, right? n- they're no, actually, yeah, I don't think so.

From what I'm seeing i- is that, again, we're g- and we're gonna get to the what could they have done better and what you can do better. I think just, i- it's not the upgrade process. it goes back to the harvested credentials. It goes back to limiting the ability for products to do what they did, and then we're gonna get to the next thing, which is once they had… They had been harvesting credentials for a year, right?

for over a year, right? And then they took some of those credentials, and they tried to log into other systems, and this is where Google Workspace comes in. And so what they had was, once they got… They were able to take those harvested creden- they started with stolen credentials, then they used those credentials to harvest other credentials. Then they took those credentials, and they logged into Google Workspace, Yep.

into, again, valid admin, as a domain admin, and they created something called compliance rules. You wanna talk about what those are? Yeah. So in an organization, you might have certain compliance requirements which are, hey, if an email is from this person or to this person, or if it contains these types of words, then forward it off somewhere else so then it can be secured and can't be deleted and is protected for compliance auditing other purposes.

And so they created a compliance rule that said if it matched certain keywords to send it out to a certain email address. And this was ingenious because it just looked like normal email traffic. So even if you were trying to detect something, it was like, oh yeah, this is just part of your normal systems operations and all the rest of Yeah address was controlled by the attackers And they would, and then it would BCC forward the, to a Gmail address that they owned, right? The keywords, very interesting.

The keywords, they're talking about geostrategic policy, military strategy, advanced tech, and specific pathogens. And by the way, that one was interesting because one pathogen that they had on the list was chikungunya. I don't know how to … Chika- You can say that? Okay.

checking I've never heard of that before. So they're saying it's a mosquito-borne viral disease that's responsible for an outbreak in China. Yeah. so interesting, right?

Oh, okay. Gotcha. hence your ability to pronounce that word. I've never even heard of that word before.

anyway. so they were… So on one hand, you're saying it just looks like normal email traffic, but, I do think that perhaps somebody could have noticed. By the way, the rule that they, the compliance rule they created was called Patriot, which is interesting, right? Didn't they misspell it?

You're right. So they did sp- so it's spelled Patriot. which again, might have been a, might have been a red flag, right? but, and so the, basically anything that, that met these, the filters that they were looking for gets forwarded to the bad guys.

And, d- Google, the Google Threat Int- Threat Intelligence Group, is that what it's called? Yeah, GTIG Yeah, GTIG. they, it says they disrupted the infrastructure, they disabled the Gmail account, that was being used for exfiltration, and they notified everybody and published, rules on how to stop this. But, I always go back to, what could they have done differently, right?

and I don't wanna focus on just this particular account, but I don't think Red Cap is alone in this idea of having legacy versions being, run side by side. Can you think of any other products that work like that? VMware Oh, yeah. Yeah.

Right? 'Cause when you were describing RedCap, Yeah I was like, "That sounds exactly like a hypervisor." yeah. you're right.

Is and, thinking or were you gonna give something else? no, I wasn't thinking. My brain was blank. but yeah, no, that's… I think that's a per- that's a perfect example, right?

where you leave, you leave many versions running. and so really what they could have done in this case is, again, obviously general password… we can start with general password, hygiene, right? The, that if they had been doing normal password hygiene, the passwords wouldn't have been harvested out there somewhere in the wild. And then because the initial attack started with, passwords that have been harvested out somewhere.

they don't know where that happened. and if that means that they were… somebody was using a username and password externally, potentially that, that then was used internally. so if they had good password hygiene, that wouldn't have happened. The next thing is to not… Go ahead I'm surprised you didn't start with the number zero.

What's the first thing that you do before password? list of three things, if you did these three things, Oh, patching? You talking about And so for patching, I was gonna say inventory management, Yeah, absolutely. and you knew what was out there and then you were able to patch it, you probably may, or you may have avoided some of these issues Yeah.

and then the big thing with the legacy version, support is to not do that, right? you don't run things that you don't, at a minimum you disable them. the best practice is actually to remove the old version completely. don't leave it sitting around.

so another thing is, to not trust application-level authentication by itself. to use something like SSO, this is an important enough thing that you don't trust the IAM system of just any old, piece of software that you use SSO to log into important things like REDCap. and had they done that… A- and by the way, this had, when we go back to the, we talked about the Canvas hack, same thing there that, I'm currently talking with a client that they were hit by the Canvas hack, but it was very minor because they use SSO for the vast majority of accounts when logging into it One other thing, and I don't know if we've necessarily touched on it in past episodes, using SSO or a central identity provider, you can enforce requirements.

So if a password needs to be rotated, do you use MFA, minimum number or, minimum number of characters? All the rest of these things you can enforce versus if Redcap had their own password system and, say, didn't have this functionality, kinda limited. Maybe someone hasn't changed their password in seven years. Who Yeah, or even if it had that functionality, that means you're managing that functionality in every, SaaS app that you use, right?

Yep. yeah. Good point. you're an organization, you really should have a central password system, Yes.

system There's an, entire, there's an entire industry built around that and, thumbs up to that. the next one… Go ahead. What? I touched on something which I think might be the next one Okay.

Touch away. one MFA? Is it MFA? It's always the next… like it's always DNS specifically phishing-resistant MFA.

yes. You wanna talk about what that is? Yeah. So like we've talked about, someone can guess your password or steal your password, like what happened here.

But with MFA or multi-factor authentication, just having the password isn't enough. It should also send you a way to authenticate in addition to just knowing the password, the username and password. Typically, you see this as, a code that shows up on your phone. email is not great, SMS is not great, but it's better than nothing.

but you really should be using like a one-time password Yes there. Many versions are out there. So that helps. And then the other thing that you mentioned, Curtis, is phishing-resistant, So you wanna make sure that someone doesn't go and steal your, MFA token and then start to use it, that you do have … I know we've talked about this in past episodes.

There's this notion of MFA fatigue, right? Where people constantly keep pinging you, being like, "Is this you? Is this you?" And then you click yes because you're tired of saying no and responding all the time.

So you want something that is more resilient to those. I think the big thing with truly phishing-resistant MFA is that MFA that is tied to something, right? The ph- it's tied to a security key, it's tied to a particular location, it's tied to a particular device, so that even if, a- again, just the… It's MFA for the MFA, right? That, that, that if they steal a session cookie, which is what was happening here, they were stealing session cookies, then they wouldn't be able to just replay that because the system would notice that it was coming from other place.

Yeah. Yeah. Yeah. and then of course you can and should investigate, passkeys, right?

and again, the beautiful thing about passkeys is that it solves all of this and it's, they're tied to a location, right? What's a passkey? thank you. So a passkey is basically a complete replacement for passwords and, MFA, and it's basically a… Think of it as a locally stored, I was gonna say password, but it is a key that is stored locally with the device that you have, that is tied to that device and tied to that account, and it's basically played on your behalf, whenever you need to log into that, that account.

So passkeys fall under what's called FIDO, which is Fast Identity Online, and the common thing that is stated… There are multiple, there are many known attacks for passwords in MFA, and you taught, you touched on one of them, which is the concept of, becoming, an MFA fatigue attack, where they send you so many things that you accept one of them, right? And then they take the accepted one, and then they go do bad things, right? there are known, there are no known attacks against FIDO-compliant passkeys.

So if you're not doing them everywhere you believe you can be doing them, you should be investigating that right now. if y- if you're familiar with that and you're working your way, I would just say prioritize that wherever you can. if you don't know anything about it, then, you should be looking into passkeys, One of the things that GTIG pushed, in terms of what could have helped here is the concept of device-bound session credentials and context-aware access. And w- we talked about this a little bit.

Context-aware access, if you enable that on an account, it's going to do things like, why is this coming from a different place than it was 30 seconds ago, right? and why is it coming from this IP address? Why is it coming from this country? I thought this person was in California.

Whatever. It's a context, right? A con- and every time you go to access, it's gonna check that context. And then you have the device-bound session credentials, or DBSC, and what they do is, they're tying each… 'Cause when you authenticate with a, a browser, that creates a session, and if you're using DBSC, those credentials only work in that session and they tie it to that device.

It's a little bit like passkeys in that regard, and Google is just talking about these especially for highly sensitive accounts. If you turn on these two features, it would tie, once you authenticate, it would tie that authentication to that session, and then context-aware access would check that, so the two work together hand-in-hand. And what that meant was if and when someone, like this malware, steals your session credentials, they wouldn't be able to use those anywhere else Yeah.

And specifically, this is the case where they're not just stealing your password, but they're actually stealing, say, your browser cookie your cookie and trying to replay and reuse your cookie on a different device. Right By using device-bound session credentials, you're guaranteed that even if they stole that session cookie, it's not gonna be usable anywhere else again, it's specifically something that, that they, talked about, or that Google talked about. so the next is this idea of compliance rules, and they're not… Th- this is, that's what Google Workspace calls it.

Every system has something like this, and the idea is look at any standing rules that are there. In a large company, you may have tons of them. This is where I think AI can be helpful here. Look for things that are created that are forwarding email out to especially external accounts, right?

Didn't you that's what's- anytime something gets touched in these compliance rules? yeah. A- agreed, right? Ha- have a compliance rule for the compliance rule.

that anything that's a new compliance rule, th- it should trigger some sort of event so that you can then go check that. That's a great, that's a great point. So then the next one they recommend is something you might think is obvious, but I guess a lot of people get this incorrect. It's separating your credentials across security domains.

So as we looked at this case, RedCap, one security domain, Google Workspace, another security domain, the two streams shall never cross, and yet someone used the same credentials across the two, and that's what sort of allowed a breach of RedCap to now impact their Google Workspace account. Right really should be keeping credentials separate if they need, if they don't need to be common, or using SSO or an identity provider which deals with all of this for you so you don't have to worry about it.

Yeah, or also PAM, so privileged access management, right? and go ahead Would… Here's a question for you, Curtis. Yeah this comes up a lot of times when we talked about backup systems, how keep backup systems separate than your production systems, from a credential management, active directory perspective. When a company is using SSO, say O- Okta or someone else like that, do you think it is important to still think about these security domains and so have a different SSO user for Red Cap versus their Google Workspace account, or is that overkill since you have Okta providing some of that control?

W- yeah, when it comes to admin accounts, I don't think there is such a thing as overkill, right? I think you, you need to treat admin accounts differently. perhaps you do one thing for, regular users. And again, going back to the client that I was talking about, that they, the reason why they were only, they used SSO for all the normal users, but admin accounts were separate.

so I think that is, that is important. You can use SSO to layer on top of that, right? So that they work together. But I think, I still think you need to, keep that separate for the, for all the reasons that we just talked about.

and then the next thing, we talk about here is get some logging, man. Some sort of XDR, some sort of SIEM, some sort of, some sort of monitoring going on, looking for the kind of things that were happening here, because they had to be sending a butt ton of sensitive e- email out to external accounts, for a really long time, and, nobody seemed to catch it. I don't, that, that's quite disconcerting did they say when they actually breached Google Workspace? 'Cause I know I, it was towards the end.

It was towards the end of the attack. Yeah. It was the last thing they did after they'd been harvesting credentials for over a year Yeah. Yeah.

Crazy Crazy. So here's a question. So we talked about this, right? And things you could do to prevent attacks.

Given the name of the podcast, is there anything that you would think Did we forget something? from a data protection, data recovery perspective, from a cyber recovery perspective they could have, should have done? Or is this even if you had the most amazing cyber recovery plan in place, you didn't do any of these basics, you're hosed I don't think, this is one of the situations where I'm not sure how much backups would have actually helped, right? Because it, you do need, obviously, you, you're never gonna be saying you don't need backups, right?

one of the things we talk a lot about is just please, for the love of everything, please make sure that you have regular, automated backups of everything that are on a sec- separate security domain, and also that use truly immutable backups, right? That i- you, 'cause the worst thing is when you hear something like this, and then you see that the backups were also impacted. The downside here is that backups quite possibly aren't going to help you because d- two things. One is, many people don't store their backups longer than the amount of time that this attack took, number one.

Number two, what are you going to recover? Are you going to literally, like you found out that they first compromised your system a year and a half ago. Are you going to restore your database to a year and a half ago? You're not going to do that.

so the related topic here is, from a system standpoint, one of the things that we came to in the book, which we haven't mentioned the book. if you're not familiar with the book, that should be over my, right shoulder in your view. if you're watching this, it's my left shoulder, but it's right in the video, right? I think I'm pointing right.

Yeah, it's to my right in the video. Anyway, the, is, Learning Ransomware Response and Recovery, which, I wrote with, Dr. Mike Saylor, who is a frequent podcast, guest. that, Dang it, I lost my train of thought.

We were talking about the… Oh. So you really have to approach backup a- as two things. We really came to the r- reality that th- there, there were three ways to restore after something like this. One is to just restore back to a point in time before you, you know that you were attacked, and that's like the last thing we want you to do, actually, i- in terms of restoring the operating system and the applications, because it's really hard to know when that point is, and it's really easy to reinfect your systems.

So that's one method. The other is this idea of, restore and then go in and surgically clean each system before you release it to the public, right? And that is better than just knowingly restoring, without doing any cleaning. But again, it still could be really difficult to find some of this very pesky malware.

You talk about the malware here that could- kept reinfecting. Sometimes the malware will be, in the firmware, and it will reinfect you after a reboot. So really, I think the only real method here isn't just, I guess it's technically backups, but it's really an automated, like infrastructure as code type environment, where what… you know what your… when we talk about VMs and applications, you know what version of the OS you're running, you know what version of the app you're running.

You should be able to push a button and boom, you have a new VM of that from a trusted source yeah that is, that was created when you first made that system. This isn't something you're backing up. So what you're backing up is the application data, the database and, and the documents and all of that stuff, and then you, when it's time to, to rebuild, you rebuild the OS and the application, and then you restore the actual data for that application. And generally speaking, the data itself won't be infected.

Go ahead. I have a question for you. Yeah Do you know, and maybe this is also a Dr. Mike question, any ransomware actors who have attacked the Golden Images?

I am not aware of any, right? I would I think I would argue… You mean the actual s- so where the images are stored? in the sense of if they infected the golden image, right? The, like you said in your Like a supply side type attack, basically.

Yeah right? Or it during the attack, yeah. If they, if they infected those images as you're trying to recover and rebuild, like you said, right? Rebuild the OS and everything else, you're basically reinfecting yourself.

Now, hopefully Yeah. testing those as well before they Yeah them during rebuild, but maybe that's something yeah, I, that's, I, it's a good risk to think about and I guess, in my mind, any golden image is stored either on worm media or on, immutable… Like worm media like a DVD-type situation or on truly immutable storage. But because again, you gotta think of it like a backup. you gotta make sure that there's no way they can attack it, right?

And so this is something that you're creating right in the very beginning. Yeah. ahead do wonder how many people actually think about that, 'cause I don't think they do until you were actually just talking through it and Yeah. Yeah, you have to think about it like a backup, and that backup has to be stored in a way that no one can ever get to it.

And again, my standard there is if you can't delete it even if you want to, then I'm happy. If it's anything less than that, I'm not so happy. and there are a bunch of ways to do that, right? You c- there are truly immutable storage platforms that even you can't delete.

There's different modes. sometimes you'll see compliance mode and retention mode. you want the more restrictive of the two. Again, the standard is i- if you can't delete it, even if you want to, then, then that's what you want, right?

and, you could also do that with worm tape, and you could do it with worm, optical media, one of which we'd covered here. M-Disk, right? the media that's meant to be around forever. I wanna see hackers attack that.

good luck with that. It's a, called a blowtorch. Which is what I was saying. Unless, like the only way you can this media is, like you said, if you have physical access, all bets are off Yeah, you, yeah.

That, that's true with any element of cybersecurity, right? so you, that's why physical access is, paramount. and I go back to the best that I've ever seen there, where it took multiple layers to, to get, to just to get in the building. I remember Yeah.

I did this a couple years ago with a client where it took a half hour just to get in the building, and then once you were in the building, you were monitored, because physical access is king, right? all thanks again for, chatting about, one of my favorite topics I know. I love these topics. Come on.

I was talking about woodworking Oh. I'm glad to see at least you're using the tool which I guilt-tripped you into buying, so Yeah, it was a lot of fun. I made, for a preschool that I work with, I made these, plexiglass windows that were… I m- I made window frames, out of two-by-threes. Yeah They were, yeah, two-by-four plexiglass window frames.

I made seven of them, and they were incredibly inexpensive because I did them with all of my own tools. And, yeah, it was very cool. All right. folks, thanks for listening.

if you didn't listen, why would we even do this? that is a wrap The Backup Wrap Up is written, recorded, and produced by me, W. Curtis Preston. If you need backup or DR consulting, content generation, or expert witness work, check out backupcentral.

com. You can also find links for my O'Reilly books on the same website. Remember, this is an independent podcast, and any opinions that you hear are those of the speaker and not necessarily an employer.