The #1 Myth Putting Your Industrial OT Assets at Risk

[00:00:00] Dino: you can't do it in a vacuum IT, you can't do this from your ivory tower from states away in order to determine that this cybersecurity technology process tool that you might be implementing or wanting to put into place and what that really means to these control systems.

[00:00:14] Dino: Which means you have to really understand the operation of those control systems, the technology that's in there, and really get a good feel for that, right? The back to the think globally, act locally. And so the question then becomes is, how do we work together in order to recognize the disruption that can be caused by these IT vulnerabilities and or actions that they take?

[00:01:03] Craig: Hi, I'm Craig Duckworth.

[00:01:04] Dino: And I'm Dino Boussalaki.

[00:01:05] Craig: So, Dino, today, let's talk about ITOT, which is a very common topic that we do, and how the OT can better protect themselves against the unknown for misconfigurations or not knowing the implications of some of the IT tools and what they have on the OT environment. Is that something we can talk through, you think?

[00:01:29] Dino: Yeah, and it's not just tools, sometimes it's just IT's behavior,

[00:01:32] Craig: True. Yes. It's everything.

[00:01:33] Dino: And their processes, that can be very disruptive, right? And what we're kind of wanting to talk about today is, I don't think too many people would argue that most of the impact when it comes to material breaches, 99 percent of them come through IT.

[00:01:46] Dino: They just happen to make their way to the plant floor. And why is that? Why is it that if IT gets a breach that comes in through email, why is it making its way to impact my plant and its ability to make goods, right?

[00:01:57] Craig: Yeah, and part of it is, there are applications, email on HMIs, on engineering workstations that they don't need to be there, potentially. There are misconfigurations where someone would come in and make a change and forget to revert that change. And the ever loving myth of we're air gap, which is we know is not the case in 99. 5 percent of the situations and the IT approach not working as well in the OT space. It is a big portion.

[00:02:26] Dino: All right. So, let's talk about some real world examples. Let's talk about EER, for example, right? Endpoint Detection Response. We have technologies like CrowdStrike, and Sentinel 1, and Symantec, and there's others, right? Now, we know that most engineering groups that develop SCADA systems, for example, they don't put EDR tools on their HMIs, or their historians, or their application servers.

[00:02:49] Dino: And the question you have to ask is, why is that. And do we find EDR on those systems? Yes, we do. And it's usually because IT has made the determination that we need to have this endpoint protection on those devices. What IT doesn't really take the time to figure out is what those processes are, those activities are, and what those HMIs, for example, are doing to keep that EDR from impacting that.

[00:03:11] Dino: So, if you all remember last year, CrowdStrike had this issue where there was a change within their environment that broke CrowdStrike. Well, it also broke CrowdStrike at their clients. And then you had CrowdStrike running on OT assets, that were impeded and impacted by that CrowdStrike failure, right?

[00:03:28] Dino: And anybody that use this CrowdStrike, they went through this, know that, right? And this was an afterthought, right? This is IET installing EDR under the premise of governance and policy and trying to improve cybersecurity processes without fully understanding what that EDR tool is going to do to those control system applications that are managing and running those kinetic physical connected systems.

[00:03:51] Dino: And so we created a blind spot, right? And so why did it do that? Right? Why is it making decisions from a security perspective without really doing their due diligence in understanding the impact of when antivirus fires up, for example, or scans are going on within that environment because of that tool, or updates that they push down to it, that all of a sudden my packaging line and my filler.

[00:04:13] Dino: Are starting to slow down or stop because the HMI is out of service, so to speak, right? So it's not servicing the application and or the control systems that are required to communicate to have those machine centers on optimally, Right?

[00:04:27] Dino: We see it in changes to the network. We see it with patches that they push out, right? We see these blind spots that nobody's really going through and reviewing that to determine how do I close these doors, right? And you can't do it in a vacuum IT, you can't do this from your ivory tower from states away in order to determine that this cybersecurity technology process tool that you might be implementing or wanting to put into place and what that really means to these control systems.

[00:04:52] Dino: Which means you have to really understand the operation of those control systems, the technology that's in there, and really get a good feel for that, right? The back to the think globally, act locally. And so the question then becomes is, how do we work together in order to recognize the disruption that can be caused by these IT vulnerabilities and or actions that they take? Hey, in their mind, they're doing their due diligence, but at the same time, that due diligence is impacting and impeding my plant floor.

[00:05:17] Craig: Well, again, I think it goes back to the very core values of who owns the asset and what tools, technologies and processes work best for that asset and the process that asset is performing. Again, it's difficult for someone that has no responsibility, I wouldn't say it's difficult, it shouldn't be allowed for someone that has no responsibility for the outcome to interject something that has an effect on that outcome into the environment.

[00:05:44] Dino: Well, and the voting power is off, meaning that, you know, if you got 15 electoral votes, and OT got 12 of them, and he's got three of them, that balance is off. So somehow we're going to make it so that OT is also participating in this, has the same amount of influence, and responsibility, and is a key decision maker and stakeholder in the decision making of the cybersecurity initiatives that IT is driving to say, well, how's this going to impact my plant floor?

[00:06:10] Dino: If you're going to put EDR on my infrastructure, you're going to put EDR on my infrastructure. Down here that's running a hundred of my HMIs. Are you going to keep from messing those things up? What are you doing to understand how those things work and run so that when I'm in a particular run, I'm making we're packaging beer today at 2000 cans a minute, and I'm going to do this for eight solid hours. How do you keep them, IT, for firing off a scan or pushing out an update onto those HMIs?

[00:06:36] Craig: Well, I'd tell you how you do that. You put an IDMZ in and you don't let IT into your network at all, but that really is not how we fix the problem. But that's one way.

[00:06:46] Dino: Yeah, but think of it this way. If you go back to whoever, whoever was the system integrator of the OEM that's providing that machine center who did the development of the applications running those HMIs, what conversation is IET having with them in order to put EDR on there, number one.

[00:07:00] Craig: Probably don't even know who they are.

[00:07:01] Dino: Right. Well, the system integrator is not putting EDR on there, and there's a reason why they don't do that, because they know it's disruptive, and somebody's got to maintain it, and somebody has to administer it. And, I was using the analogy in a conversation earlier. It's like overburdening the weight on the plane, right? If you got to stay within certain weight limits for that plane to fly, however long it's going to fly. And all of a sudden, now, IT's at an extra thousand pounds on to that plan.

[00:07:25] Craig: Right?

[00:07:25] Dino: Right? Without anybody knowing it.

[00:07:26] Craig: Well, then you got to go back, you got to say, dude did I load enough fuel to get me where I need to go? Do I have...

[00:07:31] Dino: Is the plane big enough?

[00:07:33] Craig: Do I have enough horsepower in my engines to get it off the ground? There's all the things that go into that and I don't believe that, again, someone that's not responsible for the operations and the production is interjecting things whether that's EDR in this situation that is, other...

[00:07:51] Dino: Network management?

[00:07:52] Craig: Anything that has a negative outcome on the devices and assets in that environment and how they perform.

[00:08:00] Craig: You talk about production, talk about latency, you talk about critical processes, you have to maintain certain levels and you don't always know how it's going to behave.

[00:08:09] Dino: Well, IT needs to recognize that the tool and technologies that they're using that could potentially bleed over or be implemented and used in an OT environment, is it live fire? Are you firing live rounds into my environment into this production space? You need to know whether you're doing that or not, right?

[00:08:25] Craig: It makes me wonder how many organizations have a true OT test environment where they are validating anything that they want to put into the production cycle. What's the implications on that production environment? Is it going to break something? You can only do part of it. What's the server structure? You know, there's so many things you have to consider, does it even make sense?

[00:08:49] Dino: Well, and whose priorities are, right? Is it IT's priorities? Is it OT's priorities? I mean, at the end of the day, and we've had this discussion before, CISO or CIO, who would find and learn that Windows 7 is still running in their environment, their first reaction is get that stuff out of here and out of the environment, take it off the network, without recognizing, it's like, well, taking that off the out of the environment shuts down my plan, or shut down portions of my plan, right? So we're not going to do that. What else you got to deal with that? But that's IT position, get it off.

[00:09:18] Dino: They're not looking for virtual patching. They're not trying to determine how they can better micro segment that network. They're not looking at a potential E.DR Solution that is purposely built for those types of HMI. And they exist, so what homework are you doing, right?

[00:09:32] Craig: Well, I think part of it goes back to the IT environment is rich, mature, been around 30 years. The corporate structure, the enterprise have longstanding relationships with the IT vendors that are now saying we can do that, too. And since they're buying millions of dollars from them and partner, and they're saying we can do it, we have that under control, and they're trying to frame a square peg into a round hole, and it doesn't work.

[00:10:01] Dino: Well, and if you take an organization like ours, if we're responsible for 70 percent of the control systems in your plant, we designed them, we built them, we installed them, we're maintaining them and supporting them. When's the last time a CIO or C Sorting was people have called you to find out specifically what's going on inside all that technology that you've been managing and running and supporting and implementing that plan over the last three to five years?

[00:10:23] Craig: They don't. We know that. They don't. That doesn't happen.

[00:10:26] Dino: So how do you expect to solve this problem?

[00:10:27] Craig: It should. Again, it's conflicting IT, OT security priorities. Back to the conversation of where does the org chart overlap with OT and IT? And it's normally in the CIO's office or the CEO's office. They don't have the same task and they're different.

[00:10:45] Dino: Well, and OTs shouldn't be let off the hook here either, right? They have a more significant role they need to play in this space.

[00:10:51] Craig: They do.

[00:10:51] Dino: Not only internally amongst the organization, but also with their supply chain, with their vendors that are supplying these technologies and services into their environment. They need to start asking a better line of questions. At the same time they have to recognize that they have accountability and responsibility for this also on the OT side. And they should be searching for those vulnerabilities, just like we do. When we go out there and put in one of these OT IDS tools, what do we invariably always find?

[00:11:15] Dino: Every single time, and there's a couple of them, right? One, we find IT meddling in that OT space in a lot of different ways. Whether it be EDR, or scanning the network, or making changes to it, or pushing patches to it. The other thing is, we find things in the environment that neither one of them know anything about.

[00:11:33] Craig: Ture.

[00:11:33] Dino: Right? Road assets, the perception that it's air gap is like, yeah, try again. This is not, and we can tell you that it's not right. Keep telling everybody that it is OT.

[00:11:44] Craig: Oh, yeah! For sure.

[00:11:45] Dino: The fact of the matter is you're connected, whether it's some remote access piece that's going in there, whether it's Craig showing up with his laptop to go down there and plug into

[00:11:52] Dino: that environment, or it's something connected to the network that's giving a path up and out, some VPN concentrator that's in there, or something that's completely missed altogether, right?

[00:12:00] Dino: A wap in the ceiling tiles that we find. I mean, all kinds of rogue assets, things that they're unaware of. And so they both have cool ability. The question is, what are they doing to solve it? And not every group is well suited to do this all by themselves, either side. You need to start bringing in organizations.

[00:12:16] Dino: One, especially the people who've designed and built the machines that you have, and gain a better understanding of that. The OT folks have a pretty good idea, but under the hood digitally, not so much.

[00:12:25] Craig: Right.

[00:12:26] Dino: They know that that machine is running, and doing what it's supposed to be doing, and it's running optimally. But under the hood, do they really understand all of the flows that are in there? All of the protocols that are being used? If we go into ten plants, we'll find two, two and a half, they'll have malware then either side knew that they have.

[00:12:43] Craig: Right.

[00:12:43] Dino: That's in the environment. Even organizations that have been hit with ransomware tried to unravel and untangle it themselves, or wrote the check to pay for it. Did you really get rid of it? And what are you doing to keep an eye on it to make sure that there's nothing out there that's still got a signature in it, right?

[00:12:57] Craig: They're not looking at that, It goes back to the collaboration, working together, the skill sets, the staffing and or staffing shortages on the OT side, how does it look on each side it's not being addressed.

[00:13:10] Dino: No, it's not. And I go back to what I said earlier. Organizations like, or even our competitors and our peers out there on the OT side who are in that supply chain for these manufacturing clients, we are not being tasked or asked by IT about what we're doing. What are you designing? What are you building? What are you using?

[00:13:26] Craig: No.

[00:13:27] Dino: Right? Not, never.

[00:13:28] Craig: No.

[00:13:28] Dino: Never.

[00:13:29] Craig: I don't think that they would. And they should be asking every time. I look back and I say, okay, if they sat down at the table and they asked the questions, what are they going to learn from the conversation?

[00:13:40] Dino: Well, I think one of the key things they should be asking is, okay, I see you're bringing in a new packaging line. It's going to have six HMIs on it. It's going to have a dozen PLCs. What are the risks and vulnerabilities associated with those assets? Day one, they landed my plan.

[00:13:53] Craig: Well, they're all outdated to start with because the OEM froze that firmware and froze those stuff.

[00:13:58] Dino: But what is, it?

[00:13:59] Craig: Yeah,

[00:13:59] Dino: Where are you starting from? Day one. And so if IT is doing their job, or even OT for that matter, either one of them could be asking is, what are you bringing in here, and where are you at right now from a risk and vulnerability standpoint with those assets?

[00:14:11] Craig: Mm

[00:14:12] Dino: Alright? that vendor design group should turn around and say, here is exactly where we are and why we're here. And here's our path forward, over the next few months in order to upgrade the operating system, upgrade, SCADA application to upgrade the firmware on those PLCs or on those drives in order to try to keep you as current as we can with the information that we have in order to reduce your risk and improve your security posture. And then measure that, right? You can't manage what you're not measuring.

[00:14:39] Craig: Right I get they need to freeze it for Production cycles, but once it's in it's validated. It's commissioned Then you're 100 correct lay out that plan that 90 126 month plan Whatever it is to bring that asset to full Cyber hygiene if you want to call it that so that it matches what the rest of the environment is

[00:15:04] Dino: should be poking and probing, right? When these OT, suppliers are engaged and involved in manufacturing facility, well, what devices are they bringing? How many BYODs do they have? They come in with their own laptops, they come in with their own tablets or coming in with their phones,

[00:15:19] Craig: Yeah, most systems integrators bring their own laptops in because they have the proprietary software Whether that is rockwell's Factory talk or it's siemens or it's Inductive automations, ignition, whatever that is, they're bringing that in because the end user OT teams don't have spare licenses laying around if they need anything done.

[00:15:46] Dino: it's expensive,

[00:15:47] Craig: Very expensive.

[00:15:48] Dino: and that engineering group that's bringing that software in has multiple projects on it for not only other, , clients of theirs, or it could be other plants of yours, and now you've got this mismatch of this workstation in there that's got project files on it from clients.

[00:16:07] Craig: Yeah, it's the not knowing the unknown the ignorance of just purely not being educated is a risk in itself. and if you look at a risk matrix, that's should be a big piece of that is what we don't know should really scare us. And how do we get that information? Who do we partner with?

[00:16:32] Craig: Where do we go for that? Let's call it source of truth. you go to straight to a vendor, you get Architecture left and right you go to You know a true systems integrator like us that has breadth across all of the plans What does that look like? How do you get best practices in your environment to truly understand?

[00:16:54] Craig: Organizations need to start looking at that

[00:16:57] Dino: and they should. I mean, that CIO or CISO should be asking those integrators, here's our OT cybersecurity strategy. We want to continue to move forward in this environment, or what are you doing, integrator, to improve the cybersecurity posture within our environment? and what are your capabilities?

[00:17:13] Dino: do you have a practice around this stuff? You have partnerships with some of the leading OT cybersecurity technology companies, and so that's the kind of discussion that needs to be had within these environments, and, it should be back and forth, two way, and we just don't see it, not at the level that it should be at.

[00:17:30] Dino: You know, I don't want to say, I don't know why, because I think it's just both sides to our point earlier, doing what they've always done, Putting their heads down doing the work that they do, and nobody's really checking their homework, so to speak, on

[00:17:45] Craig: Yeah.

[00:17:46] Dino: cybersecurity perspective.

[00:17:47] Craig: Yeah, true. you hear, how many million open recs in O. T. for cyber security for infrastructure and how's that play? the skill sets aren't there. The O. T. Teams are already overtaxed overworked underpaid. You look at it that way. They don't have the capabilities to continue to take on everything that's there, from an organizational standpoint. pick a partner. Pick someone that you trust, that is intimate with your environment. Go through start with basics. Tabletop exercise. Who owns this risk? Who owns these assets? Where are you at from an organizational standpoint? All the way through to a vulnerability assessment, to a risk mitigation, to where is your path?

[00:18:33] Craig: Forward do you where do you want to be in one year three year five year? Have you started the journey yet? So many questions to ask. I think it's Overwhelming for organizations and it's they're looking at it. How do I eat this elephant? Well, you can't do it one bite you got to chunk it up into little pieces.

[00:18:53] Craig: You got to step back and say, okay Where do we begin in a crawl walk run fashion? And really start checking that off of the to do list.

[00:19:03] Dino: Well, and the other thing, too, is for you folks out there that have, uh, chosen a tool. from an O. T. I. D. S. Perspective, understand that those software companies are not in the service business. That's not where they want to be. So they rely on partners. And so now you have to start interviewing their partners and to find out which partners specifically are actually people that truly work in the industrial environment.

[00:19:25] Dino: Versus buying from a partner who's an ITVAR, who's got a small service group of some sort, maybe they've got a networking crew, but they are not OT professionals, right? They don't design and build control systems, and so you have to bridge that gap in order to recognize I'm going to buy. Clarity. I'm going to buy Arbus.

[00:19:44] Dino: I'm going to buy Zomi. Whatever. Take your pick. Dragos. Great. Now, who is the servicing organization for them? Who, what partner would they bring to the table? Besides just to sell you the stuff. Anybody can sell it to you. Real question is, is now that you bought this thing, what happens next?

[00:20:05] Craig: you need a practitioner that is fully versed in that tool or technology that, like you said, understands control automation that can go into that plant, know how it's going to interact, how to Install, deploy the sensors, the taps, the VLAN configurations, the switching configuration to get those assets, which are going to provide the vulnerability management, into that tool.

[00:20:34] Craig: To just sell it, drop it at the doorstep and walk away, everybody loses in the entire ecosystem. That's

[00:20:42] Dino: Next piece. and we see the showfare, right? We see the alert fatigue for the clients that haven't really done a really good job of understanding what this technology means within their environment and how to deal with all this new information that's coming up out of that plant floor. And to be able to operationalize it, the OT adoption, as we call it, to tune in, you know, to make it really valuable and actionable information benefit to not only the IT guys, but mainly the OT, right?

[00:21:09] Dino: I've said this many times in these podcasts, 80 plus percent of the information provided by these OT cybersecurity technology platforms is geared for OT to solve.

[00:21:19] Craig: Right.

[00:21:20] Dino: He's problem to solve it. It's, uh, I think he's problem to solve it. It's O. T. S. Problem to solve being uncovered by these technologies because I.

[00:21:28] Dino: T. Never going to be able to come back and go. Well, you need to upgrade that P. L. C. Because it's got some serious risk and vulnerabilities to it. What's it take to do that? He goes. I have no idea. Well, I. T. You can't recommend doing a firmware update without fully understanding the impact with that firmware update means of that controller, right? You need to know that. And if you don't know that, then there's a gap. There's a mess.

[00:21:51] Craig: Agreed.

[00:21:52] Dino: So long story short, you gotta look for the disruption, potentially the, the blind spots that you have between those IT and OT environments to determine if ot, if it has a material breach and it's making its way to the organization, if it's getting to the plant floor, how and why and how can you stop that?

[00:22:13] Dino: And you need to be doing that now,

[00:22:15] Dino: right.

[00:22:15] Craig: agreed. Yep.

[00:22:16] Dino: You shut those doors and improve this without allowing that, that breach to drift down into your plant floor and shut you down because that's real money, right? And your cybersecurity insurance isn't going to pay for business disruption, right? So if you lose millions of dollars due to downtime, that's just money you've lost.

[00:22:35] Craig: Well, when you look at the risk matrix, take a look at that ITOT dynamic. What's the relationship of the two of them? Like Dino said, what are the doors that are open that would allow that to drift into the OT environment? Start closing those doors. Start evaluating that true risk and better protecting yourselves.

[00:22:54] Craig: Find yourself a partner like Design group that understands it, have a conversation collectively with I. T. and O. T. and make sure that everyone's on the same page and start really making some strides toward better hygiene. Be my recommendation. So great conversation, Dino. Hopefully somebody out there learned from this and go back and have a conversation internally with their teams and, , and do the right things.

[00:23:22] Dino: But I guess what I would say in summary is IT organizations, find out who your OEMs and system integrators are, who your OT people are working for in that ecosystem. And start making building relationships with those groups. Thanks,

[00:23:34] Craig: agree. Totally agree. Support that 100%. Great conversation.