The Nation-State Attacks Hiding in Your OT Network

Intro Welcome to the Industrial Cybersecurity Insider. In each episode, we dive into the world of industrial cybersecurity. Join us as we cover the latest trends, innovations, and practical insights as we talk with leaders and practitioners across the industry. Gear up and let's get into this week's episode. Craig Hi, I'm Craig Duckworth, and today with me I have Matthew Carr from Adam Sell. Matt, welcome to the show. Why don't you give the listeners a little bit of background about yourself, kind of how you got here and what you're doing? Matthew Thanks for having me, Craig. Excited to be here. My background is in penetration testing and red teaming of industrial systems. I've worked for, um, a number of governments and organizations, banks and utility companies, um, you name it, across the board. over the years, over, you know, almost fifteen years, the last fifteen years in securing operational technology. I mean, there's a lot of different names for it, right? Some people call it ICS or OT. some people actually, I think, wrongly call it SCADA security, where you're really actually talking about one specific part of supervisory control and data acquisition. Right. But in any case, um, you know, I've been involved in what I would call cyber physical security. So, what's really interesting about OT is, is that the cyber is really linked to the physical. Right? you can launch a number of attacks if you physically get access. and so, yeah, my career has been about, adversarial, simulation testing, those types of vulnerabilities, coming up with various hypothesis about what kind of attacks might work, and then going ahead and actually planning them, testing them safely, I might add. and then, you know, working with organizations or, you know, the government and to fix these issues and, make sure our critical infrastructure is more secure. Craig Yeah. No, I appreciate it. You know, and you mentioned cyber physical systems. That's a term that I hear more often than ot most of the technology vendors, the armistice Army, whoever they that are doing that, they're starting to refer to that. I think I even saw where Gartner and Forrester, the agencies are starting to refer to it as a cyber physical system, so that there's one consistent name because it's OT it's plant floor, it's control systems, it's pick a name that's kind of like industry four point zero No digital transformation, whatever you want to call it. It's how do we share data and really learn from each other. So that's good. So as you you know, currently today you are with Adam. So, you are one of the co-founders. So help. Just a real quick background, I guess, on kind of where you guys are, is it predominantly pentesting within the OTT space and how organizations can better understand help me understand really kind of where you guys are and what piece of this giant puzzle, which is very fragmented and not really clearly defined you guys are trying to help solve. Matthew Yeah. Happy to. we mainly focus on three areas when it comes to OTT security or cyber physical security. one of them certainly is Pentesting. So we have a specialist team. I'm the lead of that team. And, um, you know, we carry out pen tests safely. We quite often work with organisations or, you know, clients to replicate lines or replicate infrastructure safely away from production. So we can thoroughly test it without, potentially causing any issues. We are also experts at finding vulnerabilities in production networks. So we've actually spent a lot of time building, proprietary tools that talk the language of the devices and the, the equipment, um, as spent by the manufacturer. And so by doing that, we can safely enumerate what's on a network and identify vulnerabilities. That's something we do. So, ot pen testing and vulnerability discovery is a strong point of ours. And, you know, we've been doing that since before it was cool, so to speak. Um, in fact, I could tell many stories about that and maybe we'll come back and touch on that. But the second thing we do is, um, we do ot monitoring. We do that for a number of, select number of clients. But beyond the service, we actually write those rule sets ourselves. So we provide the service for a number of clients, just really to, you know, enable us to keep developing our monitoring rule sets. And so these are rule sets that you can, purchase from us and you can plug them into, uh, common monitoring, tools such as snort, snort three specifically for anyone listening, that uses that and a number of other, network security monitoring engines. So what we do is we have a team that looks for new vulnerabilities, whether it's CSRs put out an advisory or a CV that's come out. And we have a team that looks at that, and we either build a detection rule based on the advisory straight, straight off with the information we have. But in a lot of cases, we don't actually have a great deal of information. So our team will go and get firmware and we'll do something called patch Diffing, which is where we'll look at the latest version, the other version, and try and see the difference in the code. Maybe it's just the difference in assembly. And then we'll try and identify where that vulnerability lies. So we can create detection signatures and rule sets for those vulnerabilities. Because that's one area that we feel is, you know, massively underserved. There are a few companies that offer, you know, say they do AI, attack detection. But, you know, quite honestly, from a red team pen testing perspective, there's a lot of attacks that to, an I.T or an immature OT solution is just going to look like an industrial command. It's just going to look like normal traffic. at least, you know, a good attacker is going to make it look, as such. So, we specialize in rule sets that really minimize those false positive and catch the attacks. And then the third part of the work that we do, it might be my favorite, it's table top exercises. So, you know, we work with clients and we, we hypothesize a bunch of scenarios, and we work with the executives or the engineering team or the, industrial team to go through those scenarios and just gauge where they're at and identify the gaps so we can, you know, recommend to the client what they can do to improve their processes. And, you know, move that needle in terms of security, maturity. Craig No, I appreciate it. That's fantastic. And again, it's something that's, needed in the industry. so if I go back what most people don't just wake up and say, I'm going to do cyber physical systems, I'm going to do cyber security for OT. how did you get your start? what led your career or you to say, you know what that's interesting that I'm going to do? I'm going to focus on this and try and bring some relevance and some value to this part of the industry. where did you fall into that? Matthew It's a great question. So my background is in, vulnerability discovery and vulnerability research and exploit development. I did a lot of ctfs. Capture the flags. And, you know, it was very active, in research and presented a lot of conferences. So I've always been really into finding vulnerabilities and, crafting these types of tools and exploits, let's say, to, to prove the concept and, and help clients patch them or companies understand the vulnerability. And so that was that was really my background. And then I met, uh, a colleague, an old colleague of mine, a former company. his name is Mike. And we were working together doing some research. And he comes from, electromechanical engineering background. So he's qualified or he's a qualified engineer, works on industrial kit. and he was doing some really interesting research into, you know, over pressurizing gas units, you know, sort of real the physical attacks. And he brought me in, you know, he was like linking out safety gates and things like that, um, all sorts of, of physical based attacks. And he wanted to do some research, and we came together and I was on the, cyber side. And so my job was to find vulnerabilities in, like, the HMI or the SCADA system or just the regular network or, you know, the switch that that links the IT to the OT or something like that. And we're working together and we actually, were down at a large car manufacturer, a very, well known one, probably the oldest car manufacturer. And we were doing some research there, and we had a line to play with. they wanted to know, could we shut it down, what could we do? And, and me and Mike worked together and did a bunch of research. And, long story short, we actually came up with a way that we could shut this line down, and that was the first time I had seen software that I had written, or an exploit that I had written. Stop a physical process, you know. And during that engagement, also, the exploit that I had written could take over a robotic arm. And to me, that was the, first I mean, my background is in, uh, before cyber was software development. So, you know, I've spent my life looking at code and, low level bits and bytes and CPUs and whatnot. And what kind of wowed me and still does, really, is the fact that you can affect things in the real world with that code. And that was the first time that ever really happened. and so from there, I was hooked because it's, the sort of it's that bridge where, you know, a vulnerability in the cyber realm can come out and affect the physical world. And for me, that was mind blowing at the time. Craig and we hear that often, you know, cyber gets in the way of production. You know, it's an easy concept when, when you really think about the physical output of these devices, you know, it could have loss of life environmental impacts. And it has real world consequences. And we've seen that you look back at, you know, some of the history from Stuxnet to various, you know, pneumonia or ammonia being released into the environment and killing towns and, you know, everything gas coming out and getting ignited and it, you know, devastation, the physical aspect of of what you just described can have real world implications. And, you know, as as we look to try and bring awareness to the to the industry that we're in, helping organizations transition and really understand their industrial posturing, if you want to call it that, how do they better protect secure their industrial networks the same way they are? Their enterprise is is really it's a balancing act. And, you know, as you know, from from the board to the executive level down to the operators, and where is that engagement needed to try and drive that narrative? You know, and I would say from, from your standpoint, because it's a little different. You talked about One of the three pillars that you guys do at Adamsville is is pen testing on the OT side. You know, you ask one hundred different OT professionals. They're going to be like pen testing is absolutely hard. No thank you, but no thank you. We don't want to do it because of the implications and the severity, the delicacy. Maybe it's the right word. Within that OT network, you know. So so what you play is a pivotal role in how to bring that awareness delicately to that OT environment. But as you guys look at the market and as you're looking at organizations that are trying to figure this out, where are you seeing the biggest opportunities and where do you see the biggest gaps? Maybe, maybe the the gaps that organizations should really be focused on. Matthew Yeah. Great question. I think the biggest gap. would be related to what you're talking about. So firstly, a lot of organisations are afraid to test these things. and maybe for good reason. and I can touch on that in a moment. but they're, you know, can be afraid to test these things. And all I can say is, is an adversary isn't afraid to touch those. Craig Totally agreed. Yeah. Matthew Um, so, you know, you have to you have to consider that. I'd also say that from a lot of, client sites I've been on and and jobs, a lot of organizations don't know the lay of the land. They don't know what they have. So they, you know, a network diagram was created, you know, when they commissioned the site or. Yeah, you know, when the consultants came in and you get a diagram and it looks great, but, you know, when you actually start to enumerate the network and figure out what's where, you know, they don't really they don't really match up. And, you know, it's drifted quite a lot. And, you know, you realize some engineer had to, had to have some kind of access to this remotely. So they put in a sort of SIM based access point, you know, um, and now they just piggyback off that and, you know, someone had trouble here sometimes they disabled that firewall. In fact, they swapped it out. They don't even like it. So it's gone now. and, you know, all these changes have been made over time. And so that that to me is the biggest gap, because one thing attackers do is they know you're they generally know your network and your, your infrastructure better than you do. and secondly, you know, they're actually willing to touch these things and exploit them. So if from the flip side, if you don't know what you've got or where it is, you can't really manage it. There's no quote from the founder of ABB, which is, you can't manage what you can't measure, and you can't measure what you don't gather. Right. And so it's like when it comes to cyber security, if I go into an organization and say, you know, how are you doing with your OT cybersecurity? They're great. You know, we've done this compliance standard or NIST and whatever. But, you know, they don't really know what's on their network. And they haven't they haven't done any pen tests or vulnerability analysis. Well, I, I would dare to argue that they're not really in a position to say they are managing it because you can't manage what you can't see. And so to me, that's the that's the biggest gap. And we actually as part of our vulnerability discovery, we do actually have the ability to map networks for some clients. and this really leads on to the opportunity, I guess. So the, you know, the opportunity that I see for a lot of people is to map their networks so we can do that in a safe way. As I said, we do that in a way that speaks the language of the equipment. And, you know, we talk, the right protocols to the right devices. because really, the issue you have is the reason why Kit gets taken down is, is for a number of reasons. But either because you're talking, you know, you're quite literally talking the wrong language to this kind of device. And they're not good at taking that. They're not like internet servers that are built to take hits and sort of take anything. Or you might be dealing with a bit of kit that, has very low bandwidth, is running on an SD card, has been for the last ten years. The person who wrote the program is is no longer with us. you know, so that's precarious. Carious, um, and, you know, that could fall over at any minute. but by doing it the way that we do it, we can map these networks safely. And I think that's where the opportunity is because, um, you're right. A lot of clients or organizations wouldn't just go straight for a pen test. And I think that's right, because I think you actually you want to trust the vendor that you bring in. That's a big part of why organizations don't pen test their OT environments is, I think, trust. Right. Because you do have to give people access to sensitive things, sensitive parts of your network. For a lot of companies, it's actually even, you know, the secret sauce. It's it's what makes the company great or efficient or able to manufacture at scale and, have the edge over the competitor. So, understandably so. Companies don't want to let just anyone in. So I think you want to build that relationship. You want to be a trusted advisor. And a good way to start is to, start with mapping. Maybe you map some non-sensitive parts or perhaps even map for the sake of insurance. We're seeing that more these days. Insurance providers are actually asking for network maps and OT pentests. Craig So yeah. Matthew Hopefully for our company's sake the opportunity is in OT Pentesting. Yeah, absolutely, absolutely. Craig And we see the same thing we have. We go into organizations and, two things that you mentioned are really kind of ring true with kind of what we're seeing as well. One, you got to trust the partnership that you have with with whoever that is. and we've preached this all the time as a very large systems integrator that's working on the plant floor within these environments every single day you find out who they are. Most CISOs are it. They don't know who these organizations and clients are. If there was an event in an organization, who would they call? They don't even know that. So you absolutely need to have that relationship and you need to trust them. And who better than someone that understands intimately your industrial network that potentially built it, that's there to support it when something goes wrong? That troubleshoots it and assesses it on a daily basis. And the other thing that we see is the same thing you mentioned clients do not have visibility into their network. They don't know what they have. And you're a thousand percent correct. You can't protect and secure what you don't know. You know you're going into it blind and you know you've got to have that visibility, you know. So so there's that piece of it. You know, also, you know, as you look at the different layers within the network and you think the traditional old assets, the Siemens, the Schneider Electric's, the Rockwell PLC's drives, the Io's, all those lower level devices. And then as you climb up that network stack to more of an IoT device, the cameras, the printers, the scanners, the the things that are maybe a little easier to manipulate, that got put on the network four or five years ago, never had a password change, never had any security protocols done to it. that's different than a, PLC that's been out there for thirty years that was designed, built, implemented before the internet even started and had no security in mind because it wasn't designed to be on the internet at all, you know? So as you look at these assets and you look at these devices, it's becoming more prevalent and it's becoming more required to figure this out and understand what you have and how to protect it. Is that the same thing you guys are seeing? Matthew Yeah, one hundred percent. you know, generic passwords, default passwords, are extremely common. These IoT or industrial IoT devices are actually a favorite of mine from a Pentesting red team point of view, because they quite often have default passwords, or they actually just have really vulnerable web applications that you use to administer the device. Because from a manufacturer's point of view, They're just trying to get devices to market and quick. they're not they're not pentesting their software. They should be, but they're not. and actually, you know, you could put your tinfoil hat on and say, you know, maybe some of these that are coming out of some countries that are, you know, on some export control lists, maybe have intentional vulnerabilities, who knows? but, you know, it's a possibility because the amount of vulnerabilities that I see in these devices are unreal. just recently I just, uh, working with the team to type up a report just yesterday, and it was, in fact, a smart TV that, we were able to get root access to and actually launch attacks from the TV. sit inside the TV and launch attacks from the TV. and so and you actually also the added advantage is we did a proof of concept code where we can actually see what's being watched on the TV. So in this case it was actually in a conference room. So if you know, this sort of leads to other types of attacks, because I find what's really interesting about cyber security is, um, ransomware did a lot for cyber security, right? A lot for awareness. but the trouble is, that most people think that's what a cyber attack is. So people assume I'm going to turn up for work one day. You know, I can't access my computer. You know, the the line is down and there is skull and crossbones on my laptop, and they're asking for Bitcoin or some other kind of cryptocurrency. Um, yeah. Ensure that happens. But people are people are forgetting a massive area of cyber attacks. And that's espionage. That could be, um, it could be nation states, looking to gain some capabilities, especially when it comes to, power grids and critical infrastructure. You know, every the holy grail of every nation states, fifth domain of warfare capabilities is shutting down critical infrastructure. Right. and so they're not going to have skull and cross. They're not going to ask you for Bitcoin. Their goal is to remain undetected. In fact, there's a term called fud fud mauer fully undetectable. So their goal is to remain fully undetectable. And so that's one area. Then you have corporate espionage. So maybe up against a pretty, you know, Machiavellian, company, you know, and they're going after you and, and and this does happen. I mean, I've had companies approach me for this type of work. For the record, I've never taken it. Um, you know, but it happens. I am a reputable cybersecurity researcher with a credible background, and people still approach me and ask me to do those sorts of things. You know, they were classed it as sort of grey hat, things. I still classed it as unethical. you know, but this this does happen. And so again, you wouldn't if I did take a job like that. my goal wouldn't be to be detected. You know, in this case, my goal for this company would be to gain access to the TV and to siphon off every meeting and, you know, confidential slide deck that they put up there. Give it to my client. Good payday. Craig Absolutely. You think so? Think of the implications of that. If every executive meeting you turn on and you record the slide decks and in the conversation. The insider capabilities to either take advantage on the market or to ruin that organization in a different manner from either the positive or the negative is is crazy. the trade secrets that come out of executive board meetings that could be taken and manipulated for whatever that is worse than here's, you know, the skull and crossbones and pay me my five million in crypto because now I have information that on your roadmap, one year, three year, five year, ten year plan. And if I beat you to market on every one of those, it could crush an organization. That's unbelievable. Yeah. Yeah. And I think organizations think of that. Matthew No, they definitely don't. And most people, you know, most risk assessment meetings you sit in, they won't incorporate that type of of cyber attack. And from the world I come from. That was the original. That's the original type of cyber attack. You know, I come from, you know, the days where I learned my craft. Hackers were supposed to not be known that you were good. If you weren't caught and you weren't known, you definitely didn't claim any responsibility for any attacks. You know, you went under the radar. Your goal was mostly to siphon off information and to increase your capabilities, maybe add new infrastructure to your arsenal or, you know, gather information for whatever reason. So, yeah, it's like ransomware. As I said, it did amazing things for awareness and getting people to care, but it almost got people to only focus on that and forget that there's this whole other world and that and, just one more thing to add in terms of whole other worlds, because I could I could talk about many of them. You have hackers who all they do is they breach an organization to simply have that access, and then they go on the darknet and sell it. So that's all they do. And again, like, if you don't know what you have and you don't know where it is and you're not monitoring it, are you going to detect that kind of activity? I know you're not right. You're not. And so it's there's I would say that's the majority of cyber attacks in this field because, you know, as I said, including nation state, capabilities in there because most of the ransomware people, they're going after soft it targets. Craig Yeah, they. Matthew Are, you know, and also like a lot of ransomware groups, they don't necessarily want to take down, a defense, network or grid. You know, they don't want, you know, kinetic warfare to be used to. As you know, you're not going to get hacked back. You're going to have a predator drone coming for you. Craig So exactly. Matthew There's there they've got their head screwed on. but the OT is, is definitely full of seriously capable, well resourced adversaries, nation states, serious organized crime groups. So these aren't we're in the world here where, you know, you sit in a risk assessment, you say, oh, you know, script kids in the basement. No, not in cyber physical world. You know, you're dealing with well-resourced, capable adversaries. and so you have to treat it seriously, and you have to be proactive and get ahead of get ahead of the risk. Craig Well, you know, the adversaries have figured out real quickly that the industrial networks Works. Manage the cash register for the organization. If I can make that cash register, stop and bring that organization to its knees, they can do whatever I want. They're going to give me what I want. They're going to pay me what I want. If it's a ransomware situation I've got, I have all the cards in my hand and they're really understanding that. But it gives those organizations that that false sense of security of all my stuff is really tightly secured and I'm all good. I got this warm fuzzy when they really don't, especially when they don't have that full visibility back to you have to understand what you have in order to protect it. And a lot of clients don't have it. So if thinking of the conversation we had and kind of where you guys are, is there a piece of information, something that the listeners could take back And if they're trying to figure this out, or maybe they're partly down this journey. Where would you see real value in a client investing money or trying to focus on to better secure their environment? What would you what would you say to a client today? That's that is trying to go down this path? Matthew Yeah. So if there's starting out down the path and I can't believe I'm saying this because given I started in technical field and, you know, used to sort of, not be the biggest fan of the infosec department and, um, you know, compliance standards, I would say as cybersecurity roadmap for, for the cyber physical. So start by plotting that out. Start by getting buy in from senior leadership. Start by identifying gaps. Just, you know, start from a real sort of project planning perspective and a very high level. And, even if that's where you start and you just make those first steps, quite honestly, you're in a better position than a lot of companies out there who just like, tend to ignore, the problem and hope it'll go away. Craig Yeah, that is so. Trust me, a prayer and a cup of coffee is not going to fix anything. That's not the answer. A lot of organizations that's that's their that's their approach. So yeah. You know what Matthew? Great. Great conversation. Hopefully the listeners have have gained some information. incredible story that you have in in your your journey so far. I appreciate the information and you know, wish you the very best of luck. Matthew Thank you very much, Craig. It's been a pleasure to come on and talk with you. And for anyone listening, we actually do have an OT security white paper that talks about one of our red team engagements. It's a nice bit of reading. It's quite interesting. it's a little bit of a, you know, tales from the Front line, so to speak. So for anyone listening that wants to download that and have a good read, feel free. It's available on our website at Amazon.com. Craig Yeah, perfect. And we'll add it to the podcast as well so that the listeners have access to it. it's a great piece of advice I appreciate that. Thank you. Matthew Brilliant. Thanks, Craig. Intro Thanks for tuning in to the Industrial Cybersecurity Insider. To stay up to date with our latest episodes, be sure to click the follow or Subscribe button now. And if you found this podcast helpful or have a topic you'd like us to discuss, please leave us a review or let us know. Thanks again for listening. See you next time.