Rethinking Ransomware and Human Error in Industrial Security | OT Security Made Simple

Speaker A: Hello and welcome to a new episode, IoT Security Made Simple. I am Klaus Mohalski, founder of ReBo. My guest today is Daniel Ehrenreich. Uh, we've been following each other on LinkedIn for quite some while, but it's really the first time that you are on this podcast. So it's a great honor to have you here, Daniel, but, uh, maybe if you tell our audience a few words about yourself, uh, how you got into ot, what you have been working on lately and since when you have been in this field.

Speaker B: Thank you for having me. As mentioned, my Daniel Ehrreich, I am a self employed OT cybersecurity expert. I'm active in the field of engineering about 46 years. In the past 36 years I'm involved with industrial control systems, SCADA, uh, a broad range of industrial, industrial verticals. Water, oil and gas, electricity, transportation, manufacturing, and so on. And uh, during the past, uh, 16 years, actually since uh, May 2010, when the famous stocks that happened, I migrated my activity to cyber security, industrial control, cybersecurity. At that time I was working for Siemens and we got all we got but a major instruction from now on, you need to quickly learn cybersecurity for industrial control systems. So I did a quick migration and since that, uh, in the past 12 years, I'm already a self employed expert, acting as a trainer, workshop leader, conference keynote speaker, auditor, and so on.

Speaker A: Uh, very interesting, uh, thank you for that. Uh, so it gives us a very long perspective on the topic we're talking about. And I mean stuxnet was kind of a trigger and a wake up call for everyone in the industry, but it seems like for you even more, it acted as a career startup. Very interesting, very interesting story. Thank you for this. Uh, so when you reached out and when we discussed what we should talk about in this episode, um, you mentioned that you've listened to one of my previous episodes, uh, where I talked about something entirely different, where I talked with Dale Peterson about his, uh, uh, OTI impact score, which tries to put an objective metric, uh, to OT incidence. Uh, but we also spent uh, a little bit of time, uh, discussing about the perceived or observed number of serious incidents we are seeing, or more importantly, we are not seeing. And we were, we try to speculate why we see so little, so few incidents. And this is what really got our discussion going. So can you give us your perspective on this?

Speaker B: Actually, the topic of how many incidents occurred altogether is a very interesting, uh, topic because everybody viewed that topic from different angle. Some people are looking at only the published uh, incidents which had cyber physical consequences. Other people say wait a moment, but how about the incidents which actually did not happen because of uh, uh, all kind of uh, technical problems. Are you counting them or not? Other people say, wait a moment. You are in the control room, suddenly you see a black screen for 10 seconds and all return normal. Are you going to report about it? It's the failure. Maybe it's the beginning of a cyber attack. So a, uh, very difficult topic. Uh, I also follow a great expert, Ralph Langer from Germany, who is also in the opinion that there were not too many attacks which uh, actually caused physical damage and were published. Then we, then we have another topic, the ransomware. Many people claim, yeah, OT ransomware is growing and growing and growing. And uh, I had to say people, no, no, we need to be careful. OT ransomware never happened. Why? Because ransomware is a, is a hostile act where you expect to get money. If for whatever reason, attacker is causing harm, but he's not receiving money because nobody from the industrial area is going to pay for a decrypting key for sure. Then what, what you call it the famous, uh, colonial pipeline. Still, many people call it OT cyber attack. And I said no, it was a classic IT ransomware. They said, yes, but the supply was interrupted. I said, I agree, but it was interrupted as a management decision. So we cannot count it as OT cyber attack. So very interesting debate. I followed your discussion with Del Peterson, great expert, great speaker in person. And we all agree that we don't have a good tool to measure it and give a firm figure. I follow another expert, Jovies from the United States. He says a huge number of cyber attacks. Yes, because Joviz believe that everything must be counted. Yes, it's also true. But, but if I count something and I don't have any traces, what really happened, then how I'm going to protect myself. So very interesting topic and uh, let's uh, develop it to interesting discussion. I hope I will open the eyes of the audience with some different opinions.

Speaker A: Yeah, absolutely, absolutely. And I like your perspective, uh, to not say it's this way or that way, but it's difficult. Um, uh, we don't even seem to have agreement on how to count, um, uh, so this is where the problem really starts. And probably this is something we have to work on. And similar to what Dale Peterson tried to create with this OTI score, uh, trying to objectify the level, the seriousness of an incident, we need a similar agreement across the community to establish a uh, proper way to count these Things, and this will be complicated because we know that some incidents will never be published, even if they are real incidents. So there will always be this dark field. By the way, uh, Joe Weiss is scheduled to be on the show over the next couple of weeks, so I'll raise the question with him and see what answer he comes up with. So we'll have all the perspectives.

Speaker B: He will mention to you. My opinion for sure.

Speaker A: Yeah, yeah. I'll send him a, uh, recording. Send, uh, him the link to the episode that he can prepare, and be careful about what he's responding. But I think what you said is very important. Uh, so you gave one idea, um, let's call it idea at this point. You, uh, gave one idea why we are seeing so few OT incidents. And, uh, I'm also following Ralf Langner, and I know that he's adamant about, uh, that there are practically no incidents whatsoever. If you really are strict about what's an OT incident. Like, you also indicated the Colonial Pipeline thing was a management decision to, uh, turn the power off, uh, to infrastructure, which was probably or most likely not directly affected. So does it count as an OT incident or not? And if you don't count it, there are very few left. So, again, why don't we see more? You mentioned the ransomware, uh, attacks, which are, uh, they have a clear financial motivation. Uh, so it's mostly done by organized crime groups. And they want to make money. They run a business, and if they can't make money out of it, then you're not a potential customer. It's easy as that. Uh, but you also said, and I found this interesting, that, uh, someone, uh, managing an industrial environment would never pay rent somewhere. Uh, why do you say that?

Speaker B: I tell you why. Assuming that you are sitting in the front of the screen in the control room, and in the morning, you see the beautiful red screen about encrypting your system, and the attacker is proposing you to the decrypting key. And he will, even if he asks for a very little money, you ask yourself a question. Can I be confident that, uh, Industrial Control System PLC HMI Control server, which were encrypted, can be returned to a safe operation after conducting the decrypting process using the key that I purchased from the attacker? M. All people will say, wow, no way. I will never sign, uh, on that decision. In that case, okay, so don't pay. Or if you want to pay a small money and throw the decrypting key to the garbage. That's totally perfect. And the only choice that you have is uh, actually delete the installation, bring a golden image and reinstall it. But now we need to be careful because operational control systems there are many different types. Some of them are safety sensitive like a nuclear plant, a refinery, a uh, chemical plant and so on. They are all safety sensitive. On the other hand, if you look at the Amazon warehouse or any type of uh, energy saving system, they are not safety sensitive. So if something goes wrong, maybe you lose some money, maybe it will be inconvenient or maybe you ship the wrong package, but they are not. So we need to be careful in some cases. I said yes, I take the risk. Maybe if the system is working, that's fine. My system is not safety sensitive so I prefer to take the risk and not cause a three days shutdown to clean the entire system. In a chemical plant or a refinery you probably will not make such a decision. So this is the topic we need to differentiate between ransomware and the cyber attack. Yes, cyber attacks can happen. Ransomware attacks when the PLCHMI or the control server are encrypted probably will not be done by a professional attacker. I'm not speaking about attackers who are not professional. They can make a mistake. They can hope, okay, I get €100, that's fine. But the professional attacker will not do that.

Speaker A: So what are the professional attackers doing? Uh, when we look at OT infrastructures and uh, what tools are they using if it's not ransomware? So where is the biggest risk coming from these days and where are the potential perpetrators that we have to be uh, uh, careful about?

Speaker B: So I think that we need to differentiate between different scenarios. You may have a safety critical plant like a nuclear, chemical or a power plant. You can have a critical infrastructure like uh, water, sewage or energy distribution. Or you can have a small non critical SCADA system for small villages or uh, neighborhoods. Um, the attackers will already know that the critical systems are fairly well protected worldw now and it will be difficult. So if they can create a political or impact, maybe they will try to do it. I heard one major company in my country said we receive 100,000 attacks Ah, per day. And I said no, no, no, you get all kind of uh, pinks and scan. If you would receive 100,000 attacks per day, probably one would be successful at least every day. Fact of life, it's not happening. But now we have the biggest uh, uh problem that there are many small SCADA systems which are incorrectly designed, the architecture is incorrectly designed. They are exposed to the Internet because they are not protected. Why they are not protected. Because some people believe that having remote access and remote, uh, remote maintenance is so important for them that they are willing to take the risk. In fact, we had such incidents in the United States and I believe also in Israel, where water utilities suffered some attack. But what type of attack? Because someone decided that it's very important for them that that SCADA system will be exposed to the Internet. So if you expose to the Internet the SCADA system, what? Yes, you can expect that the non professional attacker, uh, even a ransomware attacker will attack your system. And in some cases these companies will pay because they are not, they are not professional the people and they may not be aware of the consequences. So that's the beauty about uh, industrial control and operation technology system. The broad range of different systems and each type, type of system must be specifically evaluated and considered. And this is fine. Okay.

Speaker A: Okay. So, uh, to make sure, uh, uh, and I want to reiterate this, you should never pay for ransomware tax. Uh, even outside of ot, you should never pay. And you gave just another good reason. So there are many good reasons why you should never pay, um, period. But an OT specifically because even if you get the keys and decrypt, you can never be sure that you have the same safe state as you had before. So that's an additional reason. So you should never do it, period. Um, so you also said remote access is a problem. I mean remote access in OT has been discussed many times over. Would you argue that, uh, it should be still today's best practices to not have remote access on the shop floor in critical infrastructures? Or would you differentiate what kind of infrastructures, how critical the specific network segment, how critical the entire um, uh, environment I'm trying to protect, or is it a general recommendation?

Speaker B: So I tell you very openly that uh, until about five years ago when the COVID 19 started, we clearly said, people do not allow remote access. If I have a problem, please fly from Germany, I pick you at the airport, come to fix the problem, and I return you bring you back to the airport and that's it. The COVID 19 changed the entire approach because we had no choice. We said if we, we don't allow remote access, then probably our plant will not operate for uh, two weeks. Okay, so things change since that several technologies were deployed which conduct some kind of authentication to the connecting entity. And then we are again coming back to this discussion. Are we talking about a safety critical plant or are we talking about the uh, small village water utility? Okay, so that's, that's the difference so in some cases I will be very reluctant to allow. Even today, this morning I had a discussion with the customer who said no, we not allow remote access unless we have an emergency, very complicated situation and then we have entire procedure. How do we allow a remote access? There are about, I believe about 10 vendors who different type of uh, solutions to allow secure, we call it sra Secure Remote Access. And recently uh, there is even a new technology introduced by the Data, uh, Diode company who are actually using a keyboard transmission in order to, to pretend like the keyboard is next to the system, but actually it's far away and connecting through the Internet. A very secured operation which was recently introduced. And I hear very good feedback from people that they are trusting that uh, solution. So again, coming back to the major issue, what type of system? How critical is that system? And then, then we are in a better position.

Speaker A: Okay, so you're saying the more critical the system is, the more careful I need to be in my selection process. There are good solutions out there, but you have to be careful not just selecting the best solution by looking through the latest garter reports. It's also a matter of deploying them and configuring them correctly. As always, as you mentioned earlier, uh, that you believe that most of the problems we are seeing today in OT infrastructures are not from cyber attacks but from uh, let's say misconfigured uh, PLCs and other devices in, or poor operational procedures. Uh, to put it more broadly actually

Speaker B: I tell you, even if you have the best technology, even if you know the person who is connecting, you know him by his grandmother M so well, you know that person who is connecting remotely, you still don't have an idea if someone is threatening that person

Speaker A: uh,

Speaker B: to take a uh, risky action. You don't know that. So to tell you that there is any kind of absolutely secure remote access. No, uh, we need to see what we are. And there are procedures you allow to do that. If you must do that primary because of financial or safety reasons, you allocate a very short 5, 10 minutes time slot. And once this process is completed, you disconnect the ability until next time you approve another connection. So yes, there are. So not only technology, but tracking procedures are very, very important to ensure operating safety and reliability and performance.

Speaker A: Okay, okay, very good, very good perspective. So you're very clear on secure remote access and I think this is understood. Uh, are there any other issues, problems uh, that you are seeing, uh, that you would say, uh, they are part of best practices, like things you should do or things you shouldn't do in critical OT environments.

Speaker B: I think that every organization must have internal procedures. We call it typically best practices. That's what you must do, or that's what you are not allowed to do. And it goes from daily maintenance, it goes to purchasing. How do you purchase a plc? Do you agree to purchase the PLC from a local uh, shop? Do you agree to accept it from a box that was already opened or maybe used or maybe manipulated? Many, many things can happen, but we need to focus about solutions that protect the system and, and I want to bravely say at an affordable cost because cybersecurity must not be expensive, must not. If we do it right, we can create a fairly inexpensive solution. For example, deployment of IDS intrusion detection system, um, is a very good solution, very important solution because you put your effort to detect a broad range of anomaly conditions, anomaly which you cannot define, you cannot define what happens, but you deploy it, you collect the data, uh, you present it on a dashboard and then it is fine. Then we need to look at how the entire system is working. What will be the role of the system? Security information event management system. What will be the role of the soc? Some people are talking about the term OT soc and I tell them no, OTISOC was never defined and it's not a good solution. It's not a good solution. Why? Because by the definition of the SOC for it, you allow intervention into the system in order to stop the attack, in order to minimize the damage or do any type of remediation. For industrial control systems you are not allowed to do that. You simply not allowed to do that. So I said no, never say otiso. You can bring us an alarm screen to the sock room to display the situation. But remember, in the sock room there are no people don't understand physical parameters, they don't know what is temperature pressure, except of the coffee temperature of the coffee. That's all what they know. Uh, so we need to present them a highly critical alarm, a critical or a low critical alarm, and then they know whom to call, what to act. But they are never allowed to access the industrial control system in order to take an action. Because I am well aware that damage can happen and the consequences might be even more severe than the consequences of that anomaly condition.

Speaker A: Absolutely, um, very uh, uh, interesting and contentious perspective on the OT soc, because especially as we speak there are uh, probably hundreds if not thousands of companies offering OT soc services. So how do you call, um, the functionality, uh, that an OT SOC promises to provide? How do you call this in the OT space, is it OT monitoring? Is this uh, condition monitoring? Is this process monitoring or is it something entirely different?

Speaker B: I tell you, if you live in an area where there are many villages, very small towns and those operators cannot afford ah, to have a 24, 7 supervisor for their water system, I say bring the screen, uh, the real time screen to a room, put 15 screens on the wall and there will be one person who can view the alarms and the conditions of each of the water companies in the village. And when they will see an anomaly condition or they will see an alarm, their role will be very simple. Example, call the assigned person and ask him either connect remotely up to their policy or just uh, go to the site and because there is an urgent problem.

Speaker A: Okay, so locality is very important. You need to do monitoring and the actions, the reactions, you need to localize locally.

Speaker B: I would like to call it SCADA Supervision Center. Okay. SCADA Supervision center where a person will be sitting in uh, a room and simultaneously watch about 15, 16 screens. And uh, I think that uh, he will have good chances to detect any type of uh, anomaly condition or a problematic situation and alert the right person. I think it's the reasonable, reasonable approach. Better than having nothing. You know, we always compare it. Should I spend a lot of money to put a person on three shifts or should I have nothing? So in between. Yes, it's the good solution. And also water in m. Most cases is not a highly, it's a critical infrastructure but in most cases it is not highly dangerous. Except of the old smart story which actually was not a cyber attack. It was a, it was a mistaken action by the operator and he claimed it was cyber attack. But no, but, but in most cases, no.

Speaker A: Okay. No, thank you. I, I think we need to do a follow up episode on the OT stock. Uh, it's too interesting a discussion so we'll definitely have to follow up here. Uh, so my final question that I give to most of my guests on the show. Uh, so we talked about all the challenges. Some of them have been discussed about for many many years, you could say even decades. And still we are struggling in many areas to do the right thing. So for someone who is listening to this and uh, who have neglected the area of OT security for the past uh, 10, 15, 20 years, what do you uh, recommend uh, to them, where to start and how to start in a single like 1, 2, 3 step action. What's the first step they should always take?

Speaker B: So first of all I would like. That's a good question, but uh, allow Me to say a comment that the biggest risk to industrial control systems are not the cyber attacks but the risk caused by people who were never trained on industrial control system CyberSecurity and the IT people who have completely different approach to what cybersecurity is and how it should be handled. And uh, so I say people start with a training program. There are many excellent training programs available. You can start uh, with a Internet type training programs. I myself conduct 4 hours to 40 hours session depending on what the organization is interested. I conducted the training program to Australia, to Hong Kong in Israel within the university I uh, conducted 40 hour sessions. So yeah, training programs are available including the famous SANS training center which are excellent uh, programs. So yes, invest in training because training has the highest return on investment that you can take to protect your organization from not only cyber attack but actually incidents which might harm the business operation continuity. So let's talk about two goals. Ensure safety and ensure business operation continuity and believe me, cyber attack. Cyber security will be included in this effort whether you like or not because the technology solutions that you need to implement will actually ensure also cyber resiliency.

Speaker A: Very good, very good. Thank you. So training is the answer and I really like this approach uh, because training always helps and I specifically like that you make it measurable that training generates the biggest return. Uh, Daniel, it was a great pleasure having you. Um, it's very interesting listening to you and we'll definitely need to do a follow up story on probably on the OT SoC, but let's talk about this separately. Thank you for being here. Uh, it was a pleasure talking to you.

Speaker B: Thank you very much. And I want to encourage people who have any question related to OT cybersecurity, please find me through the media and feel free to ask any question and I will do my best to reply in a very short time because I believe that this is a very important topic for protecting our critical infrastructure, manufacturing and so on. So important, important topic. Please feel free to contact me and I do my best to help you.

Speaker A: Thank you for this generous offer. Thank you.