Hacked fridges and another Google Cloud outage

Speaker A: Halloween is upon us and October has already been full of it horror stories. French supermarket fridges filled with spoiled cheese after Internet connected temperature controls were hacked. And another hyperscale outage taking down banking services. While they are different and unique incidents, both raise questions of reliability and safety. Could they have been avoided? Will they happen again? And what can we learn from them? Hi, I'm Bobby Hellard.

Speaker B: And I'm Scott Becker.

Speaker A: And you're listening to the IT Pro Podcast. Joining us today is IT Pro news and analysis editor, Ross Kelly.

Speaker C: Hello.

Speaker A: So let's start then with the excuse the pun, Unbelievable incident of the French supermarket hack. What happened there, Ross?

Speaker C: Yeah, so this was described as an unprecedented attack on refrigerators at a, uh, series, uh, of supermarkets in France. Hackers were able to remotely take over the systems and reset their temperatures to 50 degrees centigrade. I shouldn't laugh. It's obviously, you know, a really concerning incident. Caused some fires, flooding primarily due to sprinkler systems and. Yeah, yeah, destroyed a lot of fresh and frozen food. Hundreds of supermarkets across France were impacted.

Speaker A: I was going to say it didn't just spoil the food, it did cause physical damage as well. Like you just had fires.

Speaker C: Yeah. I think this is a really great example in all the worst ways of cyber attacks having a direct impact on everyday life. You know, obviously you're talking about potentially tens of tons of food being spoiled, which in itself is a massive waste. But at the same time this has an impact on people, their ability to get their groceries that could have caused physical harm to people working in the stores as well.

Speaker B: Yeah, and so, uh, this wasn't just in France, but uh, other places in Europe as well.

Speaker C: Yeah, places in the Netherlands, A, uh, series of supermarkets there were impacted. There were other reported incidents in Spain and Poland where the sort of root cause here comes from the fact that it's a company called Danfoss, which operates essentially management systems for these refrigerators. This is a Danish company as well. So you can imagine this, you know, obviously spread through throughout a lot of European locations.

Speaker A: The question to ask here is, are we putting too much online? Did these fridges need to be online? Is that why they were mainly m hacked? Uh,

Speaker C: yeah. So Danfoss told AFP that this was due to essentially outdated firmware, legacy systems. The company obviously emphasized that, you know, they advise organizations, users to, you know, maintain these, keep them updated so that there's no incidents such as these unfold and just in terms of general maintenance and wear and tear. But yeah, in this situation it's definitely A case where, you know, outdated firmware gave an in for the attackers. And this obviously gave them, you know, quite significant access in terms of what they were able to do. And like we said, you know, they obviously tinkered with the temperatures. They tinkered with whether or not these systems were switched on and off. And I think it goes back to, again, basic patching, you know, keeping up to date with, with, with updates here. If particularly for, for these retail organizations being able to actually keep a lid on this and stay up to date, um, this will have been a big learning curve for a lot of organizations.

Speaker B: Yeah. And it'll be interesting, you know, kind of to, To Bobby's point, there is like, uh, the part of the learning curve is, do some things need to be connected to the Internet at all? You know, is it that important to have, you know, Internet access to your refrigerator temperatures? It also gives, uh, attackers a means to cause tens of millions of dollars of damage.

Speaker C: Yeah, I mean, it's one of those ones where, I mean, even in the consumer space, you have fridges in people's homes that are connected to the web. My fridge isn't personally, so I'm not in any position to be worrying about that. But at the same time, I'm very much of the opinion that some things, yeah, they don't need to be connected to the web. You don't need to have sprawling, Iot sort of web in your home, so to speak. That's basically opening the door to attackers potentially. But, I mean, I think it goes back to what I said at the beginning, that this is a really great example of the impact of a cyber attack on everyday lives. Particularly when you look at attacks on retailers in the UK in recent years as well. Obviously it's not a direct comparison, but when systems go down, particularly at, uh, say the Co Op, when they were hit by a cyber attack, shelves went empty in a lot of places around the UK because it impacted their logistics system and it impacted deliveries, particularly in remote locations like the northwest of Scotland, places like Skye, Harrison, Lewis, a lot of these places. The. The Co Op is like a very popular retailer in these areas. And so when shelves go empty, you know, that gives a real, A real indication of, you know, just how devastating these types of attacks can be.

Speaker B: It reminds me of an essay that I read recently too, just about. And I'm, um, blanking on who wrote it. But it was just about how you don't buy anything anymore. You enter into relationships like you buy a watch and you've got a, uh, relationship with responsibilities to pay match and things. And there are certain things you don't need a relationship. Yeah, you just need a product.

Speaker C: Yeah, absolutely. Sometimes you do just need a fridge and you don't need, you know, a long term relationship with a brand or, or a provider. And you don't need to be paying subscription fees for certain products. You know, you just want some form of household utility or equipment that you need. You, you really don't need that. Apologies to, you know, folks out there who do have connected fridges.

Speaker B: Oh yeah.

Speaker C: Part of it's, part of it's jealousy. I have a very basic integrated fridge. I think that's what you get with a new build home.

Speaker B: Ross, I'm with you. I've got a fridge that doesn't. In America it's a big deal. Doesn't even have an ice maker. Um, I've never had an ice maker on a fridge. I'm very, I'm very low end fridge person. So

Speaker C: that's what the little plastic ice cube things are for if you ask me. And you just fill, it's perfect. You fill it with water from the tap, pop it in the freezer and yeah, you're good to go after five or six hours.

Speaker A: Does anyone remember the story from years ago about the fish tank that was connected to the Internet and the hackers managed to get into the casino via the fish tank? It was a darktrace story, but that's the problem with these kind of random IoT devices is that they can be gateways into your wider business and network

Speaker B: and they can be so low profile that people m. Forget about, you know, that they've ever installed something or didn't check with the IT department. Switching gears a bit, you know, another month, another hyperscaler outage. This time Google. Ross, what's, what's the deal here?

Speaker C: It feels like uh, a very common occurrence at the moment in terms of the impact, not in terms of the frequency of hyperscaler outages, but when they do happen it's full systems go, so to speak, in terms of the impact on services. So with this particular incident, this occurred in the Europe West 2 region which you know, for those who don't know, that's basically the London region and it's linked to other European regions. The impact here was pretty significant in terms of how consumers and organizations were affected. So Revolut, so NeoBank in the UK, that was among the companies that went down, so to speak, due to the outage. Likewise with Bank Scotland, hsbc, Royal Mail and some retailers were also hit by this, some Revolut customers weren't able to withdraw cash, they weren't able to access basic banking services. Likewise with HSBC and um, bank of Scotland. So again you know we're talking about a real world impact here in terms of people being able to go out and pay for basic products and being able to check their bank balance. Another big outage as well. So that's when you look at the last say 15 months you've had Google Cloud, Microsoft Azure and now AWS. On this particular occasion it's obviously Google Cloud infrastructure based in Europe. A lot of the time when we talk about big hyperscaler outages the first thing that comes to your head is US East 1 because when something goes down across the Atlantic a lot of the time in Europe there's an impact there as well.

Speaker A: And this is quite a long time. It was an eight hour outage from Google's perspective.

Speaker C: Yeah, an eight hour outage. So a, ah, working day for pretty much most people, particularly in the UK and parts of Europe in terms of what caused it. Google's attributed this to a uh, power event. So in their postmortem of the incident they basically explained that power is supplied to this facility through two separate feeders which they describe as A and B. In this instance there was an outage on utility feed A and this was caused by an external underground cable fault on the utility network outside the data center facility. That's directly quoted from the um, the postmortem there. But in this case typically the facilities generator plant would pick up the slack but it failed to receive a signal automatically according to Google and then that prevented it from taking that load and then sparked the outage.

Speaker A: I think we've discussed uh, this before as well that every time we have an outage it seems like it's a different reason. There's so many variables to go wrong, so many things that can go wrong. It's very much something that's just going to continue happening year on year would you say?

Speaker C: Yeah, it goes back to what I said in terms of severity there. I think the frequency of these incidents isn't catastrophic at this point and perhaps as a fact of life, you know, um, we live in a very modern digital connected era. So when one particular hyperscaler has some form of incident it's going to hit headlines and it's going to cause nightmares for a lot of organizations, a lot of IT teams and a lot of consumers. The severity I think is what, you know, the big problem is for a lot of people the Big concern. When you go back to the AWS and Azure outages last year, these were very lengthy outages as well. Likewise with the Google one. More recently, eight hours, like I said, is that's a working day for a significant portion of the population. And if they're not able to access basic banking services, that's an issue. You know, it also has an impact on, on businesses as well. You know, if they are not able to access particular systems hosted through Google Cloud infrastructure, then you know, that has a financial impact on them. There's a big economic impact of any form of downtime, whether that's individually for the business or whether it's directly caused by the hyperscaler and then downstream to the businesses.

Speaker B: It kind of um, points up the trade off that we all made, you know, about a decade to 15 years ago to go to cloud infrastructure and where you're trusting Google, Microsoft, Amazon with your most critical infrastructure. And the positive parts of the trade off are they're able to hire the very best people, spend you know, incredible amounts of money on their infrastructure, you know, and redundancies. So the comparison is how much downtime would all of these customers have just in unreported small outages that just affect them and their customers. And how does that add up versus when Google has an incident like this and a lot of people go down, uh, because of it. And I'd be interested in, I don't know if you guys have ever seen a study on that, like what, how does this compare to what we would expect if everybody were maintaining their own infrastructure and just having sort of constant outages of their own, you know, on a regular basis, like, like we all used to.

Speaker C: It's definitely an interesting question. I mean if you were to go prior to sort of modern cloud era,

Speaker B: you

Speaker C: know, how often did these types of incidents occur? I think, I think you'd have to go really into the weeds there to actually work it out. But again, I think comes down to kind of an optics type situation for a lot of these companies. You know, they're very quick to respond to these now, you know, Google aws, Microsoft Azure, they've all responded very rapidly to these. I know when we sit here and say, you know, a lengthy outage or an eight hour outage, but you know, when you consider the complexity of these data center facilities and particularly the fact that in this Google Cloud one that was caused by an external power event, as Google described it, they're fighting with, you know, one hand tied behind their back, so to speak, of something external to their facility occurs, then they're beholden to that. Ultimately, if it's firmly within their facility, then they do have various redundancies and fallback sort of procedures to shift things over and to ensure that there's a degree of continuity there. But I think the fact that the response is very swift and they're very communicative as well. When you go through the status reports, the service health pages on AWS or Azure or Google Cloud continually, there are very regular updates there. So I think from that perspective, you know, in terms, if you're a business owner, you're going to be very frustrated with this, but they are communicating with you, they are letting you know this is what's going on, this is what we're doing. And I think that is a reassurance to some extent, or it would be for me personally. I can't speak for a small business owner that's not had access to certain services all day, but on that front, I think all of these hyperscalers definitely are very aware that these things do happen. And, um, I think an acknowledgement that these things do happen from consumers and end users, so to speak, is also perhaps required there. I'm sure there's people who will agree with me, particularly when you're talking about critical services. I think what is interesting in the wake of this Google Cloud one is that financial regulators in the UK and in the European Union, so, for example, the bank of England, they've taken a big interest in this primarily because, you know, millions of people were left without access to their financial service providers. The European Banking Authority has been in contact with them. Um, the bank of England also has been in touch, as well as a financial conduct authority. So, you know, when these things do happen, there are questions to be answered. And particularly in the UK and Europe, regulators, they aren't afraid to, you know, really clamp down on this, especially given the severity of the incident itself.

Speaker A: Unfortunately, that is all we have time for for this episode. But, Ross, thank you so much for joining us.

Speaker C: Good to be back.

Speaker A: You can find links to everything we've spoken about today in the Show Notes and even more on our website@itpro.com you can also follow us on LinkedIn and YouTube as well as subscribe to our daily newsletter. Don't forget to subscribe to the IT Pro podcast wherever you listen to your podcasts. And if you like what you hear, leave us a positive rating and review. We'll be back next week with more from the world of it, but until then, goodbye.