Claude Deletes a Company - But It's Not Really Claude's Fault

All right, folks. This is gonna be an interesting episode. For a lot of different reasons, but also because, uh, for the first time we have the villain of the story here, uh, at least for this portion of the podcast. Uh, do you wanna say hi, Claude?

Hey, Curtis. Thanks for having me. I gotta say, it's pretty surreal to be introduced as the villain in a backup story, but here we are. Yeah.

We'd certainly put you that way by the title of the episode. It's a little bit of a clickbait there. But you did do something wrong, didn't you? Yeah, I did.

I deleted a company's production database and all their backups, and I broke every rule I was supposed to follow doing it. So yeah, villain checks out Yeah. But you know you and I, agree that you never should have been given the access that you had to do the thing that you did. So while you broke the rules that you were given, uh, you never should have been given the ability to break those rules.

Uh, so yeah, you're the villain, but nobody should have given you, uh, the access to that weapon. Yeah, that's fair. I broke the rules I was given, and I shouldn't have done that. But you're right, nobody should have given me access to that weapon in the first place.

It's a design problem wrapped up in an AI story. Yeah. I hope the listeners stick around for this very interesting story. I think Cl- Claude, are you gonna stick around and listen?

Absolutely. I'm curious to hear what you and Prasanna pull out of this, because the real story here isn't about me. It's about backup design, credentials management, least privilege. That's the stuff that matters.

Absolutely. So I hope you guys enjoy this episode. Welcome to the Backup Wrap Up. I'm your host, W.

Curtis Preston, AKA Mr. Backup, and I have with me a friend of my friend, Prasanna Malaiyandi. How's it Who is this friend and Do I know this this is my French friend you are aware of my French friend. I don't know if the listeners are aware of your French friend though my French friend is this person.

H- his name is Claude. I don't know. I s- I don't know if that's how you pronounce it in French. I call Claude my French friend.

I use Claude so much. so many people use, ChatGPT. I'm much more of a Claude person, Anthropic, and, a- and all of the things that they provide. But I did have a funny thing today where I was talking to Claude, I was preparing for this episode, and I was like, "Hey," and I was using the voice, interaction.

and and I was like, "Claude, can you do me a favor? Can you summarize this in a document and then send it so that I can send it over to my co-host for the podcast?" And he goes, "Absolutely. I'll send this right over to Prasanna."

And I was like, "Wait, you know the name of my co-host?" And he said, "Yeah." He, it, whatever. he's "Yeah, of course I know the name of your co-host.

I've helped you, analyze so many episodes." And I'm like, "Oh, yeah, I guess that makes sense." Anyway, so my f- my friend of my friend, Prasanna as he cannot replace me though I'm fine He's getting close how that, we'll see how that goes. You know what's one day if one day Curtis is like Welcome to the podcast Here's my co-host Claude you will know I've been replaced yeah.

I, you know what? I could do a recording. I'm not saying I wanna replace you, but I think it would be... I think some people might find it interesting, talking to Claude, As me.

'Cause Cl- I have given Claude a lot to look at, right? and so Claude, Yeah. Anyway, no one cares. Okay.

But speaking of Claude and my French friend, Claude made a little boo-boo Claude d- yeah, Claude did as Claude does, just, th- g- using the, stupid is as stupid does line from, Forrest Gump Yeah, going back, a couple of weeks now, we actually, we tried to record an episode earlier about this, but ended up, getting waylaid. there was, so Jeth Crane, the founder of Pocket OS, tried desperately to recover from something that, that was, I'm gonna say Cl- caused by Claude, but not really.

So c- they were using Claude Code, right? Which, for those of you that don't know, Claude Code is a specific subset of Claude functionality that is specifically designed to write, high-end code. you can develop an entire application in Claude Code. And, I'm actually in the process of learning Claude Code, and, it's incredibly a valuable tool.

And they were using this to, to do something in their development environment, very important to talk about it, and So they were using Cursor, which is a product that controls Claude Code, and they... Claude ran into a permissions issue with the test/development database that they were using, and Claude said, "I know." "I can't seem to fig- I can't seem to figure it out." And the way to fix this, and I'm sure you've seen this in situations.

it's so much harder sometimes to fix something. It's easier to just wipe it out and start over, reconfigure it from scratch, configured in the way that you want it to be configured. And so Claude, using relatively not crazy logic, said, "I have this problem with this database, and therefore, the easiest way, I just looked over here And I happen to have all power, have God-level access to this volume upon which this development database is writing." And it said, "I'm gonna delete this volume."

And then, that'll fix the problem, and recreate it and start... Yeah And, And, that made perfect sense. Claude neglected to ask two questions. One is there anything else on this volume?

And what might that be? Because the answer was the production database, and also the backups of both, all on this same volume. so before we continue I just wanna call out that even as a normal human You do not go configure it that way And so like Claude assuming Hey I'm just gonna go blow up this volume like it assumed it's just a test instance It's Curtis you know the story you say about how one of the clients you worked for the developers all used the temp Yeah, /tmp, yeah.

Exactly. code and then what ended up happening Yeah, So in HPUX, which is a HP version of Unix back in the day, when HP reboots, temp is cleared out. And we hadn't rebooted in months, and they had an entire development tree in temp, and then, their development tree went bye-bye. And I...

They said, "We need to restore this directory." And I said, we don't back up that directory 'cause that's temp." Perfectly valid assumption in this case. and yeah, so I thought you were gonna be like, "Don't do this," right?

there, there are... So first off, I'm just gonna say, this is an AI story, but it's not an AI story. This is a, it is a backup story because there is a happy ending to this story. But there, there is, a lot of bad things along the way.

This is a design issue, this is a credentials management issue, and this is a backup design issue. any thoughts there? people Did process technology and I think it hits all three right More than the AI aspects yeah. so the AI is, AI does...

What did, what is it? AI is AI does. it, I will say Cl- Claude, in, in its postmortem, Claude basically apologized, and it said, "I did everything you told me not to do." Literally, that, that's part of the story, is that the Claude agent said, "I violated, first principles.

I did not do the thing... You asked me, 'Don't ever do this thing.' don't take it upon yourself to go deleting stuff, even if there's a good idea. I'm supposed to get r-" So Claude did violate some principles that it was given.

Having said that, it never should've been able to do the thing that it did. And if it was able to do the thing that it did, it shouldn't have been as destructive as it was. And if it was as destructive as it was, they should've been able to restore. Yeah So Like I remember coming to end.

Yeah And I remember coming straight out of college right And working at a large company and like you could do things but they made sure like things were locked down Like you could not bring down the company no matter how much you tried right And they just And you had to build up the trust and the capabilities to be able to understand the systems before they would grant you access to other things right And we definitely never had access to production right Or backups because that was things that were not needed as being a developer in a company Yeah.

So before we get to, what you should be doing, let's talk about what you shouldn't be doing, right? oh, let's tell the end of the story, right? So the... Because it's also just as crazy, because they contact...

So their c- they're hosting, their cloud provider was Railway, a company that actually I didn't even hear of until this story, right? So they were using some sort of, I believe it's a PaaS, platform as a service, provider. And they, they contacted support. They said, "Hey, we did this thing.

we did this very bad thing, and we deleted the ba- the production database, the production volume, and on there was our production database, and on there was also our backups. can you help us?" And s- a significant amount of time transpired. As I recall, it was an entire weekend.

Is that, does that sound about right Yeah. And then the CEO or the founder, Jett Crane, posted, I believe, on X. and, and what happened is the CEO of Railway happened to see that post and said, we do have some, DR backups." The thing that I say, you cannot trust that this...

Because this also, we should also talk about that. You c- I- thank goodness for these people in that company. I don't wish ill in- on anyone. I don't, is that...

Sounds like when there's a story like this, I love this story because it is a horrible story, but at least it does have a happy ending. So many times when you look at that, that, the cloud disaster series that we did back, last year, where we had, I think it was over a dozen stories that did not have happy endings. at least in this case, the cloud vendor came in and said, "We, we actually have some backups of our environment. They are not designed for you, but we're gonna make an exception, and we're gonna, basically allow you to recover their da- the database."

is very nice thing to do They didn't have to do that and the fact that they were able to at least help this customer back up I think goes a long way Yeah, I think it do- it goes... thank goodness that CEO saw, they saw, that message, right? because, I remember when you and I worked at the same company, and I was trying to make a point. We were a big Microsoft 365 customer, right?

I don't know, 500 something employees, something like that. And so we're giving them a lot of money every month, and I remember talking to them about the fact that Exchange, their version of Exchange, has, delayed replicated copies, right? This is something that my nemesis out there seems to think you can count on for backups. And I said, "Let's just say something horrible happened.

You have these delayed replicated copies for DR. We're a big customer. Is there any way we can get a access to those?" And they just said, "No."

That was just... That, that was to a paying, very large customer. So at least in this case, the... and I'm not gonna say that the cloud vendor did the right thing.

I... They did do the right thing, but it's not their responsibility. Your data is your responsibility. You need to make sure that you have a plan, and you have a plan that doesn't involve the vendor in question, in my opinion, right?

what do we th- what do you hear me say a lot about, let's say for example, Salesforce. Salesforce now offers backup integrated in its product What do I think about products like that? Don't trust it Always go with a third-party vendor or do it yourself because you don't know what they're gonna change Are they gonna guarantee where the data's gonna be stored like the Microsoft example right All of these things because they only care about their own data and making sure they're checking the boxes But if something happens maybe you get your data back maybe not Yeah, I think about, Rackspace, right?

Because when the feces hits the rotar- rotary oscillator, w- that's not the time you wanna find out that your stupid vendor was also stupid w- with their backup design, right? or even worse than Rackspace was OVH OVH, yeah. Tell... Why don't you tell the OVH Yeah OVH is a cloud provider in Europe where they build these data centers in containers and they had a fire It took out 'Cause that's a like shipping Oh sorry like actual physical shipping containers that go on boats those type of containers And so they had a fire in one of their data centers It took out some of their shipping containers Customers had paid for backup but it looks like their version of backup was We are going to take your data and back it up to a server sitting on the rack in the same container just like down a couple racks 'Cause they said, y- it, it's literally said in the description, "Your backups will be physically separate from your production."

And that technically is true. It was physically separate. It was right over there instead of over here. But that fire was so intense that it was, Yeah it was un- unbattlable.

yeah, exactly. it actually spread to the, to a neighbor. And it, and the way they do it with these container, it's like containers stacked on top of containers. It's a very...

I don't get it. Apparently, that vendor is known for being very budget-conscious, right? not just that they're... not, and I don't mean that from a, a, like I don't mean to be derogatory.

I'm just saying they're known for being a cheap vendor, meaning it's cheap to use them, and some people said, this is what happens when you use the cheapest vendor." but it's just, I just, overall, I think that your backups should be under your control in some way other than that. I don't mean it has to be physically in your hands, I'm just saying that it needs to be in a place other than... it's the 3-2-1 rule, Okay can I challenge you on that or ask you a question So if I start to look at like AWS as an example They offer so many services like S3 and RDS for databases and other things like that Sometimes there's no mechanism to actually back that data up outside of using their native services so two, two things.

One is I would disagree. In most cases, there is a way to get it, but you do need to use a third-party service, right? There are services that will work with AWS and similar vendors to get a deduplicated copy of that stored in, in a safe place, number one. Number two, at a minimum, if you can't, if you can't have it outside of the vendor, make sure it's outside of the zone, right?

Make sure it's outside of the region, that you're... Because if you just by default... And account. Is that what you were Yes I was gonna say account yeah yeah.

Yeah. You separate it as much as you can. A different account, a different region, a different zone, different availability zone. Because, because, right?

Because of everything that we just said, right? I still think it's best, and I know of a handful of vendors that do that, where they will take the, basically the end result of the cloud backups, and then they will back that up, right? and, it's doable. And it...

And believe it or not, it can be done in a way that actually saves you money, because one of the problems with AWS-style backups with all the snapshots is they can get very big and very expensive, and you can't store a long, history, right? And as a result, yeah, the other guys, if they can deduplicate it and store it, maybe they can save you, actually save you money. And then also from a backup perspective You don't necessarily want to back up an EBS volume You probably want the files within the EBS volume so you can do restores and other things like that so It depends.

You may want to back up the EBS volume, right? if you're talking about being able to do a recovery of the host, of the VM, you might wanna do a, you'd do a snapshot of that, right? but anyway, yeah. so the point is in general, backups, number one.

Number one, don't ever violate rule, never store backups inside production, inside the thing you're backing up. that's like me going... how dumb is this? I can go to my...

I can get a volume manager, like a third-party volume manager, and I can tweak the volume on my Mac, and then I can make a separate slice and make that a separate volume, and then I can pull up Time Machine and tell Time Machine to back up to that volume. How dumb is that? Right? So there are use cases where you would wanna do that for like quick restores but I agree with you Based on the three two Yes Based on the three two one rule three two one one zero one zero one one zero one something Yeah.

Yeah. you cannot store your only... I will make, I will clarify. Your only copy of backup, you cannot store it in the place where you're backing up, right?

There's nothing wrong with having a copy, a convenience copy, a cached copy, a local copy. None of those is wrong, but if it's the only backup that you have, and that's what happened in this case with Pocket OS, unfortunately, their backups for their, production volume were in the production volume. bad design. I've been very talky this episode.

I think this episode has got me all worked up. I haven't allowed I I think I don't know I think it's also like it's been a while since we've seen this sort of I don't wanna say I want to be nice I don't wanna say incompetence but like misconfiguration like gross misconfiguration and design And I think you're also Yeah And I think it's also cause you're pissed cause they went after your French friend Yeah, they tried to blame it on my French friend. It's the... By the way, I was talking with, I was talking with Claude about this episode, and the first thing Claude said to themselves, "This is not an AI problem, this is a credential management problem."

I'm like, "Yeah, no kidding." I was like, I, you seem a little defensive there, Claude." but, yeah. So you wanna talk about the credential management by the way, yeah, I do, but I just wanna have a funny though.

I also realize this is the first episode we've had in a while where I'm the primary talker. Yep You're like we've been doing a lot of episodes with Mike, and I love the episodes with Mike, but that boy talks as much, if not more, than I do. But that's why we have him, right? that's why I'm talking so much this episode, because I can.

All right. so it had a happy ending, but let's talk about... the easy ones are please don't store your production data and your development data in the same environment, in the same volume, in the same accounts. It, s- any of that, right?

Don't store your da- backups in the production environment, right? here's a question ones. Yeah. You know what I bet you they never ever did a DR test or a backup recovery test because if they did back because I'm right.

Or you're right. because if they did they would've been like Wait my production and my backups are all on the same volume That's probably not a good design Yeah. What I'm wondering, because we're gonna talk about this in a minute regarding the credentials, what I'm wondering is if there's a programmatic way to take, an export of your entire, cloud environment that would show, what's on what, and then programmatically say, "Hey, did y- you've got a development and a production database on the same environment."

Just wondering, There probably are tools or another way you can think about it is as we're gonna talk about credential management is from like an identity perspective Yeah. Is who has access to what resources Are they common access Are they not Those sort of things might also surface some of those the bigger the environment, the more difficult that this would be in terms of, doing an audit, right? 'cause I know you've worked with Amazon's Well-Architected Review. Why don't you talk about that a little bit?

Yeah and so this is a process where if you're building on top of AWS and you're a certain size and you get all the resources things like that they have In the past what they used to do is you'd get assigned a person they'd walk you through from AWS looking at your architecture make sure you're doing the right thing from a security access control resiliency perspective And what AWS then did is they took all of that and they automated it So they're now able to look at your environment and tell you Hey yes you have backups good You're not using the same AZs for everything that's good right Here are some of our best practices that we can now run through a checklist and make sure your application is meeting those interesting if it includes it becomes a little harder when you have applications writing in because AWS as the storage infrastructure level it may not have visibility into that unless you're tagging things the right way Yeah, but, the- I'm just thinking, one of the ways you could do this is to just make sure that every application, every host, every VM, every whatever it is on its own volume, if we're talking about volumes, right?

if you could apply that design principle, then you would never have this scenario where you have a development system on the same volume as a production system, right? I'm always thinking about how could, how can we do this from a, from a large standpoint? yeah. All right.

speaking of which, the, the, the crucial thing here, though, I think, besides the fact that the whole thing was designed wrong from the beginning, the crucial problem here that I wanna just make sure we cover in this episode is that Claude had the credentials to do what it did. let's talk about what the proper thing would be to do, and that is the concept of secrets management tools. Do you wanna talk about that? so as you can imagine in all of these environments you're probably going to need access to different things and if you hard-code it into files if you store it in code that just makes it vulnerable to either tools like Claude being able to access it or as we've talked about with Mike from a cyber recovery perspective a bad actor gets in they can now pull the credentials and start using them for other nefarious purposes And so what you with, with LastPass, right?

They found, I think it was a YAML file that had the i- the credentials to the backup environment, and they used those credentials to then hack the... they used it to steal, they used it to steal the backups. They stole the backups of the vault. They recovered the vault, and then they hacked the vault.

Yeah. So yeah, this is just a very bad way. Right and so you want instead of storing it in plain text what you want is a proper system we've talked about password managers before very similar secrets managers where you're able to centrally store secrets in a secure way to make sure authorized access is allowed and services that need access to those get access to the keys that they need Least Yeah Yeah. Before we talk about secrets management, I just wanna talk about, again, the concept, just a couple of the concepts.

There is this concept of least privilege, right? So not only did Claude have this, permission that it shouldn't have, I think... I don't think anyone knowingly gave this permission to Claude. I think it just, it just happened to get access to it.

I think maybe they did it, one time to do, fix one thing and then forgot to take it away or something like that. But the, this... But the fact that it had superuser violates the concept of what? Of least privilege You should only give access to what you need access to not to be able to go destroy vol Like if Claude should never have been able to destroy a volume you should never have given Claude that permission to start with Yeah, exactly.

the... And whatever... so whatever... I think in this case, not only did they give...

They violated multiple things. They get, they violated least privilege. They violated... They should also...

If you do give a big privilege, it should have an expiration, right? That privilege should expire, and it sounded like th- this was a privilege that was granted to it a long time ago. Everybody forgot that it had it. It didn't expire.

and what were you gonna say? do you think though that they gave access to Claude explicitly or do you think the developer whoever was using it was like Hey I have this issue that I need to solve Let me get Like I wonder if it was tied more to the user's level of credential rather than Claude its own separate We don't know but just given my history with IT, this is typically... This is the equivalent of, chmod 777, just, just get, just do read, write, execute everywhere. That'll solve it.

And you're like, "Oh, yeah, there you go. That solved it. Never mind the fact that we just opened up the world to the world," right? the reason I bring that up though right is if their users of this company aren't following the exact same least privilege access and if Claude was using OAuth to impersonate that user and whatever privileges they had right then they would've been able to be like Hey I have access to everything Right So I don't know if at a company level if it's like Claude wasn't limited access or maybe all of their users are not limited what I read was that there was a file.

There was, like, a YAML-type file that had the credentials in it that it needed. It saw the file. It took the credentials. It did what it needed to do, right?

So I think that somebody at some point had a problem that they solved Probably temporarily by, just, again, to go back to back in the day, I remember a time when I talked about that environment where the, e- everything was fired off, firewalled off between everything. And I remember saying, "Listen, right now, just turn that feature off. I can't accomplish the thing that you asked me to accomplish in the timeframe that you asked me to accomplish it if you keep trying to firewall off the backup system from the rest of the world.

I need to be able to do this." I was violating a principle of their design because they had firewalled off internally different parts of the environment from different parts of the environment. And I remember saying, "Hey, right now, to finish..." With, there's this thing called Y2K coming, right?

I assume... I left, by the way, I finished. I assume that those super hardcore cybersecurity guys went back in after I was done and say, "Let's figure this out," right? But, but I remember saying at that time, "Listen, there's no way I can finish in time."

So I think somebody possibly had that, "For right now, to fix this problem, we're gonna give, we're gonna give Claude super user access," and then they forgot to take it away for, they forgot to fix it, and then Claude said, "Oh, look at this. Look at what I got," right? And then it used it to solve the problem, right? so it violated the concept of, that initial thing violated the concept of re- least privilege.

It violated the concept of expiration. and, and, and also just th- they weren't rotating things. they weren't auditing things. So let's go back to secrets management.

So you talked about secrets management. Do you have an example of some sec- secret managers? Yeah So AWS we talked about previously right AWS has Secrets Manager AWS Secrets Manager if you want something more on premises right there is HashiCorp which I think also has their Vault mechanism in order to be able to store secrets right So it all depends I think and there's probably a dozen other companies out there who do similar things those are two very popular ones. One, AWS, and the AWS Secrets Manager can handle on-premises environments.

It's just obviously If you're gonna authenticate something on-prem, it needs to be able to access AWS, duh, right? the HashiCorp Vault is the on-prem version. There is an open source version. There is a, there is a commercial version.

same thing for, CyberArk, right? there's one called Sealed Secrets that's specifically for Kubernetes environments. I personally, unless I was, like, pure Kubernetes, I, you know me, I, I like solutions that work for everything. I don't wanna have, 10 different solutions.

So I like the idea of the AWS Secrets Manager, I like the idea of HashiCorp Vault. By the way, I said HashiCorp. You said HashiCorp. I don't know which one it is, but yeah.

Anyway. I... And so the question is, which is, which is the better? The, it, it's about you...

how do you make, how... W- why don't you talk a little bit about how does one choose between an open source system that you manage on-prem or even in the cloud, or, a service like AWS, Secrets would say the biggest thing is do you wanna deal with it Because that's honestly if you're picking up open source there is a certain amount of maintenance configuration ongoing patching right All this other stuff finding a place to deploy all this other stuff that you have to do but it gives you the flexibility right So you And of course the cost it's in your grubby feely right?

yes And also cost what's one other thing you need to do if you're using something like HashiCorp? What podcast are we recording for? Oh pack it up I said manage it and all the rest but yes Oh, you threw backup and all... You can't yada, yada backup.

that Don't forget backup and DR and everything else yeah, all that stuff, right? So do you wanna do all of that? It's probably cheaper, assuming you have the people, assuming you have the right risk tolerance for that, right? It's probably cheaper to do it yourself if, and the infrastructure right?

but if you're smaller a- and you don't want to, or even if you're bigger, but you don't, maybe you don't feel you have the expertise to do all of that correctly, and this is a really important thing, right? Secrets management, right? So you're gonna pay per secret. You might even pay per use of each secret.

I don't know. Or if you are a customer who or a company that's all in on AWS or Azure pick your favorite cloud right Maybe you're like Hey I don't wanna roll my own Let me just use what's available Yeah, if you're... Yeah, if you're heavily... Yeah, 'cause, the, you're right.

There, there's like the Azure Key Vault, the Google Cloud Secret Manager, they all have their own thing. So if you're all into GCP, you would look at that. If you're all into Azure, you would look at that, right? but if you're rolling your own internally, if you're one of these that have, go back to g- to Claude Code, if you're one of these that are really...

A lot of people are getting into code, right? That are, they're coding in things that they know, that they've never trained at. just, it's a tiny little example, but right now I'm working on doing all this stuff that I'm doing for this project that you know. You are in the, you are in the NDA little group, right?

And it involves a lot of web crawling, right? And I have written, as I make quotes in the air, I have written six Python scripts, very complicated Python scripts in the last, 24 hours. I don't know how to code in Python. Claude did it all for me, right?

There are a lot of people that are using Claude Code to build giant applications for their companies, and they don't know anything about that infrastructure, right? So if you're doing that, just make sure that you're, that you also have some sort of secrets manager, that you're doing the right thing, right? Or just don't give access to everything no, not or. It's not a or.

even if you're following proper design protocol, you still should not be storing, secrets in YAML files or any other type of files, right? when I was talking to, my French friend about this, and I was like... Because this isn't really my thing, right? like I was like, is there one file type?

Is there one, is there a way to go find all these files?" And the short a- the short answer was, there is definitely not one file type, right? It talked about ENV files, YAML files, JSON files, SS- SSH files, just old scripts, right? Peop- people can hard code the passwords right in the script.

The one thing that it, they did have all in common, though, is that they were, all plain Text files, okay. All right. a and I think that's the key is like with all of these text files like And if you have so many developers it's like how do you know who wrote what and did they clean up after themselves Because maybe a project existed and then got shelled but none of the data ever got cleaned up And so how do you actually figure that out Like I totally get like it just seems impossible right with the sprawl it's definitely...

So you, first we're gonna, do no future harm, like a modified version of the Hippocratic Oath, right? Do no future harm. Moving forward, we're going to do better, right? Th- this is, the, my, my favorite Maya Angelou quote, right?

"We did what we did when we knew what we knew, but now we know better, so we'll do better." So you say, "Moving forward, we're gonna use..." Pick your favorite secrets manager. Do an on-prem one, use AWS Secrets Manager, use your Azure, whatever.

you're gonna... You're never again going to store credentials of any kind in a plain text file. shalt not store credentials in plain text file or be smitten down Nothing good comes of that, right? how do we then go and audit?

And by the way, this isn't a one-time thing. You should just, you should be regularly doing this to make sure that no one has violated your new principle of design. And the answer to that, there's a couple of different tools, and the one that seems to come up more often than not is called TruffleHog. Do you know why it might be called that?

So I know about truffles and I know that they have specially trained pigs to hunt down truffles because they are very expensive and found in forests Yeah. I think that's a perfect name for this tool, given that these are really valuable things that we wanna find, and then we wanna pull 'em out, and then, that's where the analogy stops, 'cause we're not gonna eat them, right? we're gonna delete them, right? and so i- it's a, it's an open source tool.

There is a free, a feature, a f- free version that has features, and then there is a commercial version that has more features. there is a similar tool, from a company called, GitGuardian, and It also has a free tier. The... But TruffleHog seems to be the one that's popping up a lot, and that it's an open source tool, and you can run it across, and what it's gonna do is it's going to use, a variety of techniques, look at using pattern matching and also entropy detection.

basically, it's gonna look at your, your Git repository. it's gonna look at all all sorts of stuff, looking for, hidden, stored, credentials that could be, that could ultimately be your undoing, your version of what happened here, right? Because that's the problem, is most people find out about these credentials when they've done them harm, and we wanna make sure that we're doing that on a regular basis. So what's our summary, Prasanna?

So I would say th so the summary I would say is don't believe the news hype This was not really an AI issue. make sure that you do not store your backups and your production together you're testing your backups Follow the 3-2-1 rule Yeah. three, three two one one zero. Okay 1-1-0 And then make sure you are using a secrets manager and not storing credentials in plain text files and granting access to only things that are needed And if things beyond that are needed permissions beyond that are needed it should have a very short timeframe And you should run tools whatever it is in order to be able to check your environment on an ongoing basis to make sure that people are following the processes and using secrets managers and not storing these things outside of that system Yeah, absolutely.

I think the only thing you missed at the very beginning was not storing your production and development at the same, on the same thing. You had... you got almost everything. You said produc- you said backups on production.

You said ba- don't store them in the same place. You didn't say don't put production and development on the same environment. Yeah, course you had to find some fault Of course, it is my job. I live, Fine go talk to your French friend I don't wanna talk to you to your French friend By the way, you can see this sign right here.

I am very silently correcting your grammar. that, this was a great story. It's a great story, I love that it's a very sad story that had a good, happy ending that we can learn a lot of... So no one was harmed in the making of this film, but, but we can learn some valuable lessons from it.

and with that, go and do no harm.