Stop Using VSS as a Backup Before Ransomware Deletes Your Shadow Copies

You found the backup wrap up your go-to podcast for all things backup recovery and cyber recovery. In this episode, we're talking about something that makes me want to flip tables upside down. People using Windows shadow copies or VSS as their actual backup system, and then ransomware comes along and then deletes those shadow copies using your own built-in Windows tools against you. And suddenly you have nothing.

I'm joined again by Dr. Mike Sailor, our intrepid cybersecurity expert and persona ndi of course. And we break down exactly why shadow copies are not a backup, how attackers are pulling off this move, and what you can do to protect yourself if you've been counting on VSS to save you when things go wrong. This is your episode.

Uh, just a quick note about me. I'm w Curtis Preston, AKA Mr. Backup, and I've been obsessing about backup recovery and now cyber recovery for over 30 years. If that's your bag, then I'm your guy.

Well, you're not gonna find anybody that cares about it more than me. Ever since 1993 when I had to tell my boss there were no backups of the database that we had just lost. Now I've written five books, a blog, and a podcast. And here we turn, unappreciated Admins and the Cyber Recovery Heroes.

This is the backup wrap up. Welcome to the backup wrap up. I'm your host, W. Curtis Preston, AKA, and Mr.

Backup. And I have with me a guy who couldn't bother to on time. Prasanna Malaiyandi, how's it going? Prasanna.

You've heard about Indian standard time, right? I am not making any comments about anybody's stuff based on where they may or may not from so I do have an excuse because I had to change into my shirt, which Curtis, I see that. shirt. I should get one of those.

Yeah. I think you have to just find yours is more what the. to God, I have turned this house upside down looking for that shirt that I paid good money for, I cannot find it. anyway, including yesterday's event where I literally took everything out of my closet.

have no excuse, but maybe it's being used as a cleaning rug somewhere. Anyway. Of course have our intrepid cybersecurity expert, Dr. Mike Saylor.

How's it going? Mike, It is going well guys. Thanks for having me. have you still have you yet?

I have not. Have you seen our book yet? I have not. It's killing me.

So we're of course referring to this book up here on my video, which if you're not watching us on YouTube, feel free. the same, this channel by the same name, the backup wrap up, and, learning ransomware response and recovery, available now, at, all the places that you buy books, although is. the. All the places, if you prefer Rainforest in South America, or if you prefer buying them directly from O'Reilly, or Barnes and Noble.

I was in a Barnes and Noble yesterday. apparently it's a building in which you can buy books. It's a fascinating. Fascinating idea.

maybe that'll take off. Anyway, so today, we're talking about, something a little bit up my alley. And I'm gonna just start out, imagine, if you will, you have developed a backup. I'm gonna make quotes in here, a backup system, because I don't think this is a backup system.

You have developed a backup system around something called shadow copies, which is a ver a feature in Windows that we're gonna talk about in a minute. And then you just come in and you've heard that something bad has happened from some sort of cyber attack perspective, and then you find out that, you know that those previous shadow copies are not available to you. I like the, this reminds me of, for those of you that have been around a minute, this is the ultimate sweep, the leg move, that is of course a karate kid.

Reference. and this also refers back to living off the Land, when we talked about living off the land a few episodes ago where they use your own tools against you. In this case, the tool would be VSS, admin, and they just delete this thing upon which you are counting for recovery. this sounds like a bad day.

What do you think, Mike? I agree. Yeah. I think one of the problems, that people have is they become complacent.

And I think that's, it's not just with your backups. I think in general with security. but for sure, whoever set up your backup strategy, whether that was you or your grandkid, or a volunteer or even someone been in it for a while, shadow copies have been around. and they come with, they come with the batteries, but they're, to, to your point, they're not, they're not a legit backup solution.

Yeah, A VSS volume, shadow services, AKA, shadow copies, and the, is a service within Windows where you're able to create a stable, consistent, image through which you can view the platform. Let me go back in time. There was a time when, VSS did not exist, and when you went to go back up. Windows.

the problem is you're backing up. When you have a large system, you're gonna, part of the backups are gonna be from this point in time, and part of your backups are gonna be from that point in time. And if the thing that you're backing up needs what we call referential integrity, meaning integrity between the different points of the system. You're not gonna have ref referential integrity and you're gonna have a bad backup.

And there used to be a product, I don't know. person, do you remember St. Bernard? Is that before your time?

That's. Or my time. Okay. Mike, do you remember St.

Bernard. I don't. Man, this is okay. By the way, St.

Bernard was what you bought something before VSS existed. And for the record, St. Bernard was a San Diego company. and, it came from the idea of the, that St.

Bernard's, the image of St. Bernard's with a thing of whiskey around the, they came to save you. So that's where the Saint Bernard name came from. The, so VSS was this method through which you could create a stable image, and do various things with it.

So Prasanna, why don't you talk about like the first, what, VSS was typically used for. Yeah, Because one of the things that you need to do with a Windows system, or if you're running say Microsoft SQL or Exchange is do your backup. And especially on physical machines, before we had virtualization and everything else. The only way to actually get a stable point from which to do your backup was to use VSS.

And so you'd have the backup apps call into and VSS. It's a framework that Microsoft provides. So you'd call in, you'd say, Hey, I wanna do a backup of this particular app. Microsoft would then.

quiesce the application. Take a volume snapshot, right? Shadow snapshot. And then it would unes, so then you can continue doing your stuff, right?

And then what the backup app would do is it would then read from that shadow snapshot in order to be able to do the actual backups. And so this is what it was useful for, and it was more of like a software snap snapshot implementation, pre VMware snapshots. but yes, it was their mechanism to get a consistent point for which you could do your backups from. And there, like you said, it was a framework, right?

So each application that wanted to be supported by VSS, they wrote what was called a VSS writer, and then VSS would, and then they just needed to plug into that world. And like you said, a backup app would just say, Hey, I'm here to do a backup. It would talk to V-S-S-V-S-S would talk to the writers. Everybody would do the right thing that worked for their application.

Then we would take the snapshot. it's great for that, right? It is the easiest way to get a backup of SQL Server or Exchange or any, or even Oracle running on top of Windows. Just use the VSS infrastructure.

but some people, Mike, have you seen this in the while where people use VSS, like as a backup mechanism? Have you ever actually seen that you have. I have. Yep.

You're killing me. What? any you? Why?

Why? Not that he's killing me, but, No? oh, you're saying why? Why would they Yeah.

Yeah. And what was the purpose? Bike. Could you elaborate a little?

I think it's just in the absence of an educated solution. so they, they tried to understand the manual or did a Google search or, saw the word, Or the idea that it was a backup of some sort, and they just went with that, Because you could create multiple shadow copies over time, right? and you could hold onto them. These are, a problem.

which is also a problem. It's a problem for multiple reasons, right? Prasanna, I'm pretty sure these are copy on write. Snapshots, right?

yes. Yeah, For those that don't know that there are multiple reasons why using VSS as a backup app are a bad idea. We're gonna primarily talk about one of them in the episode, but one of the other ones is if you've got 30 days of VSS snapshots, your performance is in the crapper, because of the way copy on write. Snapshots work.

I'll see if I can find them. We have a, we have an episode where we just. Delve into, copy on write, snapshots and what the differences between them and redirect on write snapshots. but because of the way VSS works underneath you are going to significantly hurt the performance of your system if you're using it as a backup app.

go ahead, Prasanna. one thing I do wanna. Clarify is my knowledge of VSS is slightly dated. They may have improved things over the years, but I doubt that they changed fundamental architecture from copy on write to redirect on write.

You agree? I would agree. I, think it's highly unlikely, but today. We're talking about the cyber aspects of this.

so Mike, why don't you talk, let me just look at the outline we have here. Mike, you wanna, you want to, just give a, again, a brief overview of the concept of living off the land and what, how that A applies here. Sure, yeah. Living off the land is a bad guy's strategy of utilizing data, applications and trust, already associated with a compromised host.

So if I. I'm able to get access to any degree, to a laptop or a server, what things are already there that I can utilize to further my attack, whether it's escalating my privileges or collecting data, or moving from this computer to another computer. those are the types of living off the land. techniques or tactics a bad guy would use.

So one of the first things that they will do, and this is the first phase in just about any attack, is reconnaissance. So I've got access to something. What is it and what do I have access to? what's here?

What tools do I have? one of the things that they will look for are shadow copies, for a couple of reasons. One is that potentially leverage? for extortion or something else.

and two, can I collect that data and potentially use that as information, to further understand the target? maybe it's, I've found out that this shadow copy is of, QuickBooks. so now there might be some financial information there, or, maybe there's a company name or a project name. something that's gonna make, the light bulb go off and help me determine if this is a valuable target.

and then whatever my next steps might be. Go ahead. When you're talking about that, I'm actually not sure what's done today, but when an application is reading from a shadow copy, does that get logged in all the other normal processes or is this sort of something that's hidden away and so the normal mechanisms don't necessarily apply and so it might be harder to figure out if someone is exfiltrating data. No, you can actually monitor for, not only the store, access to where the files, the data is, but you can also, monitor for the processes that would.

interact with the shadow copies, the, vs admin as an example. so yeah, those are all things that you can write detections for. and whether that's like anti-malware on the device or, much broader like network monitoring, cybersecurity monitoring platform. Yeah.

the challenge though, and again, this goes back to the, to the, the nature of living off the land is that the tools in question are commonly used by other tools, right? VSS admin and w and WMIC, which is a, the Windows management, interface, on the command line. that's one of the challenges of the, you can monitor for it, but you might end up getting a lot of false positives. the, sorry.

just looking at here. Ask another question while you're looking. Mike, sorry, I'm, I've never actually dug into, is it vs. Admin or VSS Admin, by the way.

Two S's. Okay. For VSS admin, I've never dug into it. Can you access control VSS admin?

I know you talked about you can monitor it for what files are being accessed, what processes can you also lock down who has access to it from a process perspective, or is that not really possible? That's a good question. I know you have to have a, you have to be a local admin. but I don't know if.

a scripted like PowerShell, could do it, remotely or, that's a good question, Okay, but but Okay. But yeah, once you have local admin, then all bets are off right at that point. Yep. So there are ways to use, again, just so we're not anti snapshot, I am definitely anti any kind of backup system that is easily deletable.

And this is one of them. There are ways to use snapshots and there are ways to use snapshots, perhaps storage array, snapshots, for example, that can be done in such a way that they're not deletable. basically if you get it to a place, if you get it to a version of the system that is, that is where you have worm turned on, that would be one. Write once read many, right?

There are, again, not anti snapshot. Just this particular snapshot being used as a backup method. go ahead. I wanna challenge you is based on your definition, Curtis, of what we've been talking about for the last six years, is that shadow copy, even a backup, It's not, okay?

that, Yeah. So what and why do you think it's, why do you think, I think it's not a backup. it doesn't follow your favorite rule in the world. it does not follow my favorite rule in the world.

which of course would be the 3, 2, 1, which is now the 3, 2, 1, 1 0, but basically three copies of the data on two different media, one of which is offsite. Arguably the way I understand Peter Krogh, the person who originally coined this term, is the idea, he would not think of three versions as three copies. He would think of three actual copies, right? And then he would want, and those need to be on separate pieces of media.

The two, the idea there is that you have it on different types of media. So that you have a disc drive and an SSD or an SSD and a tape or a disc, whatever. Cloud. that's what the two was meant to.

Basically different risk profiles. And then the one definitely needs to be somewhere else offsite would be the term that we would typically be used, right? It's a difficult term in the cloud, but, but. The, yeah, you're right.

this isn't any of the, this doesn't meet any of the three, parts of the definition of the 3, 2, 1 rule. So yeah, I would not consider this a backup, but based system can absolutely do that. yeah, so again, not anti snapshot, just anti using them like this. All right, Mike, you go ahead.

What. But even if you are using a snapshot system, so I know as some of my past employers, they would integrate with the VSS framework to then trigger a hardware based snapshot because that was better performant. And then they would do their backup off of the hardware based snapshot. Yes, question though, is.

I guess it depends how that snapshot is created. And is it possible to use the same VSS framework to manage those snapshots? Because I'm sure that those backup software has a retention life cycle that it manages. And so is there a risk that those snapshots could be also deleted if someone gains access to that Windows box?

I don't think the way VSS admin work is it's designed to control third party snapshots. I think it's only the designed to work the other way. I could be wrong. but regardless of how off host snapshots are created, they need to eventually be somewhere that is worm.

And if they're not, then I'm not a fan. and again, I'm not, it's not, I don't have a problem with VSS. I think VSS is a great tool. Use it for what it was designed for, which is to give you a stable, consistent image to create a backup from.

so let's talk about. What people can do to, protect against this. And I, Mike, you've already alluded to this one, is this idea of, complaining, have your EDR tools look at. Things like vss, admin so let's talk about the first thing here, this idea of, give us a little bit more information on the configuring your EDR to trigger on this stuff.

It's a process. And what I mean by that is, you've gotta your tools and your people to understand what's normal, so that. A normal process like this running in an abnormal fashion would be what you're detecting on. so with any good security tool or security service, security person, there's gonna be a lot of getting to know you the beginning.

so you know, the VSS admin, kicking off. Twice a week, once a day, whatever the case is, those are gonna show up, as a privileged, activity. and so when we're getting to know our clients, we want to be very strict at first so that we can understand all of those things and then we start to flag those, or, build an understanding. we build a wiki, that says these things happen on this.

on this frequency or in this manner. so that going forward as the, as we detect those things, we can determine if those are normal or not, or normal with some context like this will happen, but so will these other things. but if any of those normal things happen on their own, that in itself is an anomaly or abnormal. so creating a baseline and getting to know your, your environment is important.

because again, living, living off the land would indicate that it's stuff that's already there and that should be okay. Mike, then, I know you've brought this up, in previous episodes as well, about okay, you will need to have someone come in, understand the environment, Kind of fine tune it for your particular use case. It's not like a. outta the box, everything is good to go.

For you, how long does that process normally take? And I'm sure it's gonna vary significantly depending on the size of the organization, the number of devices, the applications, all the rest of that. But is this like something that'll take three years? It depends.

Yeah. what's interesting about that is, identifying the deficiencies or opportunities for improvement, actually happens fairly quickly. It's the remediation and the follow up that tends to take time. as an example, and I'll preface all of this with the adoption of security in most companies.

Has to be aligned with the culture or the compliance requirements. other words, if I'm a, if I'm a company that doesn't like security, but I want to do business in a regulated environment, I have to adopt security. how we adopt it, how we implement it is a little different. and may, it may take some time, but there are some companies out there, organizations that don't like security or their culture is security averse.

We're too friendly to have security type of thing. or, it's keeping me from doing my job, so we're not gonna do it. there are a lot of, if not every organization has one or two or many, Very blatant, observable, deficiencies, even with just best practice, basic cyber hygiene. One of those is allowing users to be local admins on their computer.

back in the day was very prevalent because it needed to be in order to, keep the overhead on already constrained IT support staff you have. let users do some of the basic stuff. I remember when one of the new versions of Windows came out and you had to be a local admin to change your wallpaper. that was ridiculous.

And, but it was culturally people were causing a big stink because they couldn't customize their computers and they were unhappy. And so it got overburdened with these local ad, these, basic local admin requests. So everybody got local admin. today, we still see that to a large degree, people, users have local admin, just because, because they, there's an exception because they're an engineer or a designer and their tools require it.

And there are better ways of going about that. Sure. Give them access to the local account, but don't let them run day to day as a local admin. run as administrator, not, Click on the thing and do run as administrator.

don't log in as an administrator. So there's things, the more things you implement, the more burden you're introducing into people's workflows and how they do their job. And so the easy thing to do is just. Give them local admin.

So there's that. but even in the, in those cases as a security, cybersecurity practitioner, so whether I'm teaching a tool to tell me what's weird or I'm monitoring an environment for what's weird, if I knew that about an organization, if I knew that somebody was a local admin, I could still, with that understanding detect anomalies, occurring as local admin. There are things that computers do, does bad guys do, that just are not normal user behavior. Yeah, the challenge I did a quick, quick look and it doesn't appear that with VSS admin that basically anyone with administrator rights can run it, and that I can't say, oh.

Only Steve. Only Steve. Or only administrator. Only you know the base administrator of account.

It doesn't appear that I'm able to restrict it by administrator, and so let's go back to your comment, Mike. I think we can agree that. The proper thing is to have an administrator account and not have everybody have admin privileges. But to go back to your earlier, thing, and by the way, this is the same in, in, in Unix world, right?

Is that you have root and then you have, we, and you use SDU to, to run things as root. The same thing here, you have administrator. Then you have Curtis and Curtis maybe an administrator, but doesn't have administrator rights, and so I need to run it as administrator. but to go back to your earlier comment, I couldn't agree more.

And it's the same in the backup world. If your culture is not one of cybersecurity. they're just gonna fight you the whole time, and I'm sure you've been in situations where the culture is the problem, right? I can think of a company, I can think of a, oh, this is a great story.

There's this company. on the East Coast and I used to work for a big consulting company and they had this particular company, and this is a company that literally everybody would know this. they actually make things. they're the rarest of all companies is an American company who actually makes things.

you have these things in your house, guaranteed. And we went to, they were having all these problems with their backups and I, I was asked to come in as the big guns and to come in and basically tell them, 'cause they were trying to get things done. And then the organization was fighting them and I came in to give a, presentation, which basically the title of the presentation was, are the 10 things that you can do to screw up your backup system. And congratulations.

You guys have managed to do every single one of them, But there was a person, there was a person that, that had power. Who was 100% against everything we were trying to do. She just, and when you have that, when you have a person, whether they're high up in the organization or down low in the organization, if they've got influence and they're actively fighting against you there, there's just not much you could do. And I'm sure you've.

been in that situation, Mike, where you're trying to do something like this where it's like you shouldn't be using your VSS snapshots as backups and you really shouldn't, and you should also be monitoring who's running VSS admin. What. What, by the way, lemme just stop for a second. If you ag, if you follow the suggestion from the first part of the call, which is don't use VSS admin as a method of doing snapshots, this concern becomes much, much less important, right?

Because the worst you could do with VSS admin, if you're not using it for backup, if you're not using them, as a backup method, the worst thing you could do with VSS admin is create a snapshot, right? Um, so. Um, no, but there's no snapshots to delete if you're not holding onto your snapshots right then who's running VSS admin becomes much less important. okay.

Let's move on to so Mike, I know we talked about okay, running VSS admin within the Windows box is bad. Keeping it long term is bad. we talked earlier about hardware snapshots are good. Could you talk about from a cybersecurity perspective, how this kind of might change with virtualization?

And if virtualization and VSS lead to less cybersecurity concerns. I know you talked earlier about like auditing and looking for anomalies and that sort of thing. Huh? Is that a loaded question?

Is that a hard. no. so on a virtual machine or a virtual host with multiple virtual machines, Again, it depends. It depends on what the bad guys were able to compromise.

If I compromise the host, then I've got access to all the virtual machines. If all those virtual machines are doing their own snapshots, I now have access to all of those snapshots, because if I've compromised the host, I'm also privileged on this box across all those virtual machines. So there's that. one of the things I do want to add though, even though the majority of, VSS.

Process and, data require admin privileges as a normal user on the machine, I can still read those shadow copies. So even if I'm a bad guy and have, I'm not admin yet, I can still do recon into those shadow copies to see what it is I have access to and potentially exfiltrate that leverage in an extortion attempt. Because it's already packaged and compressed, right? I don't have to redo that.

it's a nice little nugget, that I found. But in a virtual environment, just as virtualization makes things more efficient for us, it also makes things more efficient for bad guys. And back to a comment. or a thought, thread that Curtis was on?

As far as culture goes, I've seen it both ways. I've seen where majority of a user base drives culture. so even the leadership team may say, we need security, but then you've got this workforce that says, we will not use security. especially in a service or organization, where they're out, your user base is out delivering something and they're rebelling against you, you will often cave to them unless it's truly regulatory required.

and then I've definitely seen the opposite where leadership always wants the, they're gonna push it down to everybody else, but they've got the exception. And then everybody sees that, or maybe they become aware of it and they're like, Leadership's not doing it, I'm not gonna do it. setting the bad example type of thing. Yeah.

And this is one of those, I, I've been in environments where, I've got people in power that are telling me that raid is backup, right? We don't need backups. We're we have raid, and then they're actively lobbying against you. That's the situation where like you need to just walk away, and, let them leave.

Go ahead. there, there is an example of raid that is acceptable, but you have to unplug it and put it on a shelf. Okay, that's not just saying, not what we're talking about here. Mike.

You're killing me. am just saying it. I've seen it. They've plugged in a storage array.

They've done a backup. They unplugged it and put it back on the shelf. That's not braid, that's, you know what I'm saying? Dang it.

don't, you're trying to get my, trying to get my blood pressure up. It's so this is a little bit depressing of a podcast episode, but is the takeaway that VSS is evil and no one should use VSS. just don't use VSS as a backup. That's literally, we could, this could this episode could have been two minutes long if you're using VSS as a method of backup.

Not to be confused with using it to make your backup. I, don't you, you understand what, I don't know you're using it to facilitate whatever actual backups you're doing. In other words, the way we just, where we talked about where we said, Where you're creating a stable, consistent image that you're then creating an actual snapshot of or an actual backup of. I have no problem with VSS if you're using VSS to create multiple shadow copies you're storing those on your system, it's a bad idea for multiple reasons.

We've talked about performance. Mike talked about using them as a method to get historical information, and it's also bad because all it takes is one. Bad, one black hat with VSS admin rights, and then they just delete 'em all. So it's just bad.

So forget all the cyber stuff. We said, just stop. Just don't do this. get a real, go get a real, there, remember there, there was a, I used to have this, I don't know, it's probably bigger in my life than either of yours, but there used to be this Dilbert comic strip that we had, and it was, it was the, where the guy says.

There's a guy with suspenders in the beard he is oh, I recognize you. And he says, beard and whatever, and then smug expression on your face. You're a Unix admin. And then the guy flips him a quarter, and he says, here's a quarter kid.

Go buy yourself a real computer. That's the way I feel about people that are using their VSS, as backups. okay. I don't know.

Do we have any final thoughts on this? So I think you just said it. go ahead, Mike. VSS is often necessary as the guys talked about earlier.

it, it helps you, quote unquote lock files so that they can be prepared for. a backup. but that backup is some other solution. a lot of backup solutions today do that for you.

they put an agent on your machine. It does the, it does this function, in the absence of that capability, you're use, you're using some other backup solution. VSS is still some, is still often necessary, to prepare your data to be backed up. and then the last thing I'll add from a detection perspective is again, if I knew you were doing backups every night or certain days of the week or once a week, and that's often scheduled, right?

It's either a task or, a task on the local machine or a scheduled, activity across the network. If I knew that from a cybersecurity perspective, and then I saw. VSS or even some other privileged, backup related account, doing things outside that schedule, that anomaly, would be what I would respond to. Absolutely.

Yeah. I like by the way, there was a, it was an earlier topic. when you're in a new environment and you are trying to configure their EDR tools or XDR tools, can't you just put it in and let it watch for a while to see what normal is? You can.

and during that period, you're gonna get a ton of alerts. so there's your false positives. Okay. as you're monitoring, I found a new file that's not common across other user bases or, Bob logs in at 2:00 AM is, I don't know if that's normal yet.

during that baselining period, you're teaching those tools, tweaking them or whitelisting them. there's a ton of things that, that happens during that onboarding period. so that you're essentially creating the what's normal baseline? You can't just say, just watch for a month and don't tell me anything.

you could, it's not gonna you don't advise it. Okay. I see what you're saying. no, I don't advise it.

Okay. this is why you're the cybersecurity professional, and I'm Mr. Backup. All right.

All right. this has been fun. somewhat depressing, but yeah, just don't do that, doctor. It hurts when I, it hurts when I do this.

Don't do that. Yeah, I'd like to, that's a new t-shirt idea. Just don't do that. do that.

the antithesis of, Nike. Yeah, don't do it. Thanks guys for being on the show. Thanks, Prasanna, It was fun and I'm glad we gotta talk about something that was near and dear to my heart many years ago.

So. and thanks again. Mike, you're a little fuzzy all of a sudden. What happened with your camera there?

it's like an auto focus thing and it's got its own mind. So I'm not real sure, but, Anointing a few more of these with you guys, I would expect some logo. I'll wear a logo at some point. you'll get one when I get one.

That's what I'm saying. on. All right. Okay.

thanks to our listeners, we're nothing without you. That is a wrap.