Reflections from the Front Lines of Industrial Cyber Failures

[00:00:00] LuRae: Cyber threats are moving faster than traditional patching can handle. In this rewind episode, we're revisiting the key insights on virtual patching, OT security, and practical approaches that help protect critical infrastructure and manufacturing plant floors when legacy systems can't keep up.

[00:00:22] Debbie: ​Yeah, I think that once you get past the network side of things with virtual patching, microsegmentation, those types of technology, deep packet inspection, and being able to set up which machines can talk to which ones really come in now.

[00:00:36] Debbie: So that kind of deals with the external thread. Coming through from that IT to OT convergence, whether that, and we know number one is ransomware, right? And so the tools that we put in place for the network perspective really help address that. But now you gotta worry about the insider threat. What comes in behind the network, who has physical access to that OT asset?

[00:00:56] Debbie: So when you talk about protecting that asset, we wanna make sure that it's free of malware, maybe that it stays within, nobody's using the USB ports on that machine itself. That we want application lockdown so that I don't care if it's good or bad, I don't wanna change this. Machine's been doing the same thing for 10 years and I don't want it to change.

[00:01:18] Debbie: So for those types of use cases, we have an agent based solution that we call Stellar. That is, again, that gets installed and it provides all those features to make sure that we're locking down that asset, protecting that asset from change, protecting it from malware. Making sure that the USB device only certified or um, authorized USB devices can be used on that system itself.

[00:01:42] Debbie: It doesn't require cloud dependency. That's very key. I. So all the updates and changes happen when you say updates and changes can happen,

[00:01:51] Dino: and we'll circle back on that one because we'll talk about the events from last week, but the impact of CrowdStrike and what we've learned and what we've witnessed with some of our clients on the impact of that due to.

[00:02:02] Dino: A cloud based technology that's made its way to the plant floor and the disruption to when that event happened. People would say, well, that was an IT event. It wasn't an attack, it was just a human error. Right? It was a screw up. Plants went down. Because those plants rely on Windows operating systems to be able to run their SCADA applications or the DCS applications in order to run that plant floor.

[00:02:29] Dino: And so a lot of manufacturers try to shy away and do not want cloud-based solutions because of those types of back doors that can happen through the cloud. It can be very impactful to them. And you know that it jammed CrowdStrike down on that plant floor. They did, and it's difficult in standalone machines running older Windows operating systems, trying to put CrowdStrike out there.

[00:02:53] Dino: It's too heavy. The overhead is too much. Those machines won't even run. If you're running in a virtual environment and you're running your HMIS on virtual machines, you have a better chance of using those type of agents. You still got the heavy blip, but now you have this vulnerability from an external activity being disruptive to your plant floor and plant people.

[00:03:11] Dino: As they learn this, what do you think they're gonna do? They're not gonna wait for the next one to happen. What they're gonna say is, get that stuff out of here. I don't want it on these machines because look what happened. And I was down for three days.

[00:03:24] Debbie: And lemme just say from the IT perspective, in selling a cloud-based agent solution for six years.

[00:03:30] Debbie: There are definitely value in having that in the cloud from identifying zero days and behavioral and not having to maintain, but it doesn't mean that was the right thing to do in an OT environment. Correct. To allow any change to happen in your OT environment from a security perspective that you don't know about is not good.

[00:03:51] Dino: And there comes the rub. I was having a little bit of a debate with an OT IDS partner here just recently, and you could tell his entire focus, his entire background, where he came from was from the IT side, right? He had very little. Really recognition and understanding of the value that tools that his company sells.

[00:04:14] Dino: Who are they really for? Meaning that you have a, an OT IDS system, for example, 80 plus percent, 85% of the data produced by that system is for the OT teams. And I'll find companies where the IT security team doesn't even let the OT teams have access to an OT IDS. Now, how crazy is that? That the people that really need that data don't have access to it because some group with an IT from a security perspective is doing the cloak and dagger thing, and at the same time, their cloak and dagger mentality is just what shuts you down last week based on the CrowdStrike stuff.

[00:04:50] Dino: So we have this issue between, I know better than you on what it takes to provide security for that environment. I hear it and I witness it a lot. To your point, you know, you think about your background, where you came from on the IT side, and you've moved into that OT cybersecurity arena as some of these other IT security professionals have crossed over into that OT environment.

[00:05:12] Dino: But I would argue, well, how many of 'em actually worked on the plant floor? I. Literally worked on the plant floor and done work to put in a network, to put in security, to work on a control system startup. Put a new packaging line in my company. Are you part of the startup? Did you live in the field for six months while they were putting this $25 million piece of equipment in, in motion?

[00:05:31] Dino: And if the answer is no, then you can't sit in an ivory tower somewhere and expect to dictate what a cybersecurity, um, strategy should look like on the plant floor.

[00:05:42] Debbie: You better be a sponge. And take it all in. Yeah. Well, what I would say

[00:05:48] Dino: is you need new friends, right? Because if you think you're the smartest cat in the room, maybe you are in your circles, you need new friends, right?

[00:05:55] Dino: Right. Is is what I would say. And, and that's where the OT ecosystem of suppliers, automation technology vendors, uh, the system integrators who use these technologies to build these control systems and run these machine centers, they have to get in the game. Right. I mean, that's been my position since, since starting up the prior company, that the only way this is really gonna work is to get OT partners, providers into this space and start asking better questions and bringing better tools, working with partners who specialize in this space, and then sit out and bring it into it.

[00:06:32] Dino: 'cause I, what I witnessed is the closer it gets to this. The more they tend to back away from it because at the end of the day, they don't wanna be responsible for production, real production.

[00:06:43] Debbie: Mm-hmm.

[00:06:44] Dino: And, and part of it is educating some of our partners, especially those that are moving into the space, which is to me is the best part of the job.

[00:06:51] Dino: But that's just my view on it. Right. And there may be some CISOs in it, executives out there and infrastructure directors are probably rolling their eyes, listen to me talk. But at the end of the day. Walk a day in my shoes in a plan and then let conversation.

[00:07:05] Debbie: Yeah. And we've said that multiple times.

[00:07:07] Debbie: Collaboration, right? It's a team sport. You gotta get in the game to your point. And by being I just meant be a listener. Active. Listen, listen to what. The OT side is struggling with and what their challenges are and be able to adapt and change to that. \

[00:07:30] Craig: It's the not knowing the unknown. The ignorance of just purely not being educated is a risk in itself. And if you look at a risk matrix, that should be a big piece of that is what we don't know should really scare us. Yeah. And how do we get that information? Who do we partner with? Where do we go for that?

[00:07:55] Craig: Let's call it source of truth. You go to straight to a vendor, you get. Architecture left and right. You go to, you know, a true systems integrator like us that has breadth across all of the plans. What does that look like? How do you get best practices in your environment to truly understand organizations need to start looking at that.

[00:08:18] Dino: Yeah. And they should, I mean that CIO or CISO should be asking those integrators, here's our OT cybersecurity strategy. We wanna continue to move forward in, in this environment. Or what are you doing integrator to improve the cybersecurity posture within our environment. And what are your capabilities?

[00:08:34] Dino: Do you have a practice around this stuff? You have partnerships with some of the leading OT cybersecurity technology companies. And so that's the kind of discussion that needs to be had within these environments. And it should be back and forth two way, and we just don't see it. Not at the level that it should be at.

[00:08:52] Dino: You know, we don't, I don't wanna say, I don't know why, because I think it's just both sides to, to our point earlier, doing what they've always done. Putting their heads down and doing the work that they do, and nobody's really checking their homework, so to speak on that front cybersecurity perspective.

[00:09:09] Craig: Yeah, true. You hear how many million open recs in OT for cybersecurity, for infrastructure, and how's that play? The skill sets aren't there. The OT teams are already over tasked, overworked, underpaid. You look at it that way. They don't have the tape capabilities to continue to take on everything that's there from an organizational standpoint.

[00:09:33] Craig: Pick a partner. Pick someone that you trust that is intimate with your environment. Go through, start with basics, tabletop exercise. Who owns this risk? Who owns these assets? Where are you at from an organizational standpoint all the way through to a vulnerability assessment, to a risk mitigation, to where does your path?

[00:09:55] Craig: Forward, do you, where do you wanna be in one year, three year, five year? Have you started the journey yet? So many questions to ask. I think it's overwhelming for organizations and it's, they're looking at it, how do I eat this elephant? Well, you can't do it. One bite, you gotta chunk it up in a little piece is you gotta step back and say, okay, where do we begin in a crawl, walk, run, fashion, and really start checking that off of the to-do list.

[00:10:26] Dino: And the other thing too is for you folks out there that have chosen a tool from an O-T-I-D-S perspective, understand that those software companies are not in the service business. That's not where they wanna be. So they rely on partners. And so now you have to start interviewing their partners and to find out which partners specifically are actually people that truly work in the industrial environment.

[00:10:46] Dino: Yeah. Versus buying from a partner who's an IT var, who's got a small service group of some sort. Maybe they got a networking crew, but they are not OT professionals, right? They don't design and build control systems. Right. And so you have to bridge that gap in order to recognize I'm going to buy. Clarity.

[00:11:05] Dino: I'm gonna buy Armes. I'm gonna buy Zomi, whatever. Take your pick. Dragos. Great. Now, who is the servicing organization for them? Who, what partner would they bring to the table? Yeah. Besides just to sell you the stuff. Anybody can sell it to you.

[00:11:21] Craig: Yeah.

[00:11:22] Dino: Real question is, is now that you bought this thing. What happens next?

[00:11:26] Craig: Well, you need a practitioner that is fully versed in that tool or technology that, like you said, understands control automation that can go into that plant, know how it's gonna interact, how to. Install, deploy the sensors, the taps, the vlan, configurations, the switching configuration to get those assets, which are gonna provide the vulnerability management into that tool to, to just sell it, drop at the doorstep and walk away.

[00:12:01] Craig: Everybody loses in the entire ecosystem. That's a horrible model.

[00:12:04] Dino: And we see the shelfware, right? We see the alert fatigue for the clients that haven't really done a really good job of understanding what this technology means within their environment and how to deal with all this new information that's coming out up out of that plant floor.

[00:12:19] Dino: And to be able to operationalize it. The OT adoption, as we call it, to tune it, you know, to make it really valuable and actionable information benefit to not only the IT guys, but mainly the ot, right? I've said this many times in these podcasts, 80 plus percent of the information provided by these OT cybersecurity technology platforms is geared for OT to solve right's problem to solve it.

[00:12:44] Dino: It's problem to solve it. It's OTs problem to solve.

[00:12:47] Craig: Yeah.

[00:12:47] Dino: Being uncovered by these technologies because it's never gonna be able to come back and go, well, you need to upgrade that PLC 'cause it's got some serious risk and vulnerabilities to it. What's it take to do that? He goes, I have no idea. Well, it, you can't recommend doing a formal update without fully understanding the impact, what that far more update means of that controller.

[00:13:06] Dino: Right. Yeah, you, you need to know that. And if you don't know that, then there's a gap. There's a miss.

[00:13:12] Craig: Yep. Agreed.

[00:13:14] Dino: So long story short, you gotta look for the disruption, potentially the, the blind spots that you have between those IT and OT environments to determine if ot, if it has a material breach and it's making its way to the organization, if it's getting to the plant floor, how and why and how can you stop that?

[00:13:34] Dino: And you need to be doing that now. Totally agree how you can shut those doors and improve this without allowing that, that breach to, to drift down into your plant floor and shut you down, because that's real money, right? And your cybersecurity insurance isn't gonna pay for business disruption, right? So if you lose millions of dollars due to downtime, that's just money you've lost.

[00:13:56] Craig: When you look at the risk matrix, take a look at that IT OT dynamic. What's the relationship of the two of them? Like Dino said, what are the doors that are open that would allow that to drift into the OT environment? Start closing those doors, start evaluating that true risk and better protecting yourselves.

[00:14:16] Craig: Find yourself a partner like. Design group that understands it. Have a conversation collectively with IT and OT, and make sure that everyone's on the same page and start really making some strides toward better hygiene. Be my recommendation. I. So great conversation, Dino. Hopefully somebody out there learn from this can go back and have a conversation internally with their teams and, and do the right things.

[00:14:49] Dino: I get back to everybody that's been breached, a hundred percent of them, and I can say that for a fact, have had firewalls in the organization and it did not stop breach. What are you doing to get better? And so best practices is get an asset inventory. Continuously monitor, understand your vulnerabilities and start putting some metrics around that thing.

[00:15:10] Dino: Look at your remote access into that environment and what does that look like? Because OT comes into those plants seven ways to Sunday to get in there, right? They just do. And COVID kicked that door wide open because people couldn't travel. And now, and now this air gap conversation, even for those that believe that their machines are still air gap 'cause they're controlling the cable in and out of the network.

[00:15:31] Dino: When I want 'em in, I connect it in or I turn the switch, they get in, do their thing, I turn the switch off or I disconnect the cable. Well, how do you know that gets done? Because everybody that says that, there's always those one or two or three machine centers in there that are still hot, still connected, right?

[00:15:46] Dino: Right. Somebody didn't pull the cable, somebody didn't turn the switch, or when they were remoted in via cellular modem, which we find those two and it's still and connected. Again, nobody's watching it.

[00:15:57] Craig: Yeah. Couple things, I'm kind of trying wrap this up. I would say, what are some. Significant consequence to potential consequences of not patching.

[00:16:08] Craig: We hear OT doesn't patch. OT doesn't patch, it's gonna cause downtime. We don't wanna do it. We can't test it. So are there significant consequences that could impact for not testing that we could say, Hey, this is the case. Do it safely in the right manner with the appropriate technologies versus just don't do it

[00:16:32] Dino: well.

[00:16:33] Dino: There's also insurance impact because I can't get cybersecurity insurance 'cause I don't have an asset inventory. I don't have vulnerability management metrics in these manufacturing facilities. So the insurance companies have gotten wiser and said if stuff. Or your coverages are going way down and your cost for policies are going through the root.

[00:16:52] Dino: So you have that scenario going on. You have those that have been breached who aren't necessarily communicating the fact that they have to their investors, you know, even though they've been forced to do so, regulatory requirements are saying, Hey, if you're a publicly traded company, you're supposed to communicate to your investors within a few days that you've been ahead of material breach now, and the government has come and been lax on that, depending on who you are.

[00:17:17] Dino: Because they recognize it's like, I don't have several millions of dollars to go solve this problem. But at the same time, it's. I say most, I say smoking ears is what some of them are doing. Say, ah, I bought a tool, we're doing this again. You're just, you're just scratching the surface. It's back to that

[00:17:35] Craig: 60, 30, 10 rule, right?

[00:17:37] Craig: 90% of organizations are just beginning to try and figure it out. You got 10% that are really doing something and making the effort and are on the journey. The rest are not

[00:17:52] Dino: so. Couple times, why do they fall back when they start seeing the cost Yes. Of what it takes to do this. And so they didn't either budget for it appropriately, or they haven't, haven't been able to articulate to the executives and the owners of an organization on why this is relevant and important.

[00:18:12] Dino: So they, they continue to wanna roll the dice. They'll say that's an acceptable amount of risk. Well, what's an acceptable amount of risk when you don't even know what that acceptable amount of risk is?

[00:18:22] Craig: That's where organizations like ours come in. We can help them with that process. We can help them with budgetary guidelines.

[00:18:30] Craig: We can help them. You've mentioned that asset inventory, we can help them with that asset inventory. There are ways to manage it. You can't just blindly say, we are not going to do it because of X, Y, and Z. You can't. We will build that case for them, help them down road so that they can go in. Educated and present that to the board of directors, the C-suite, to whoever's making that financial decision.

[00:18:58] Dino: It it's, well, in order to build that case, you gotta get all the wires in the same room, right? Yeah. Number one. You do and, and it's not just internal, it's external. Yeah. A lot of external players involved in your manufacturing capabilities. Again, back to the OEMs, back to the, that are also. Perform activity and work for you to help keep those machines running.

[00:19:20] Dino: So what is their role in this game? And again, it's an old term I've, I've known for, for decades. You've gotta get all the liars in the same room to help solve this problem. You can't just kick it over to it and expect them to do it. Even where we are today for the organizations we work for, and I witnessed this when I worked for Rockwell.

[00:19:39] Dino: You have to get both sides to start having a strategic, serious conversation around this in order to move forward.

[00:19:48] Craig: Great points all the way around. This conversation could be days. This is not gonna go away. This is something that's gonna take Collaboration from both sides can take strategically understanding the way that it works.

[00:20:02] Craig: But great food for thought for the listeners out there. We hope everybody, let's start taking steps to. The next

[00:20:12] Dino: You gotta get better.

[00:20:13] Craig: Yeah, you gotta get better. Period over period.

[00:20:15] Dino: Most sides have to agree. Are we getting better? It can't be just one side saying, oh, we're getting better. Would OT say that because it bought a tool?

[00:20:23] Dino: 'cause it put a firewall in because even OT understand the ramifications of the fact. Yeah, you put in a firewall, but I don't manage it. You manage it. So when you update and patch that firewall on a pretty regular basis now, what's changed to the environment? Because I'm not watching it on the bottom side to see what holes you've poked into my environment that before.

[00:20:43] Dino: Or an o eem who comes in, or an SI who comes in and provides vulnerabilities in the environment. 'cause you're not mapping and studying what they're doing either. So again, you gotta get all the right folks in the same room and have a serious conversation about this. And sometimes you have to do it by.

[00:20:58] Dino: You can throw out a general blanket strategy and say, here's what we're generally gonna do, but it's gotta, it's drift all the way down to the person that's actually standing in front of the interface on that machine center out there at the plant floor.