OT Patching vs IT Patching: What's Commonly Misunderstood

Dino [00:00:00] it's a huge effort in order to determine the current state of the cybersecurity, , posture within that manufacturing environment.

Dino and they don't have good asset visibility. You know, when they go in there, they're not getting a really good, asset inventory of the environment to determine the status of those machines. And getting back to the research piece, there's technologies out there, for example, , 'cause you can't run agents per se on a lot of these OT assets.

Intro Welcome to the Industrial Cybersecurity Insider. In each episode, we dive into the world of industrial cybersecurity. Join us as we cover the latest trends, innovations, and practical insights as we talk with leaders and practitioners across the industry. Gear. And let's get into this week's episode.

Craig My name is Craig Duckworth.

Dino And I'm [00:01:00] Dina Bus Locking. And today we're gonna talk about it versus OT patching and software updates. So.

Craig Given that Dino, how would you say patching on the OT side differs from that on the it.

Dino Yeah, well, you know, it patches everything all the time pretty much. Right? So patch, patch, patch, that's their standard way of staying ahead of the curve and keeping up with, uh, the patches that put out by the software vendors. Right. On the OT side, they don't patch very often, hardly at all.

Dino So it's creating a big gap of vulnerabilities that you find in an OT environment, which is connected to an IT environment. So you have one organization that's consistently and constantly patching, right? Patch Tuesday, as we still call it, you know, the Microsoft patches that come out and those get pushed out pretty quickly and immediately for most organizations.

Dino And then on the [00:02:00] OT side, , it's a rare occurrence.

Craig Yeah, I've heard definitely Patch Tuesday. Everybody in the Enterprise IT organization is familiar with that. It affects all of us when we come in and log onto our laptop, you know, recognizing that there are Microsoft devices down in that ot. Network, whether they are engineering workstations, they are hmis.

Craig How different are unique challenges that would present as you attempt to, or completely ignore, maybe that's not the right word, but don't patch as you would say in that OT space.

Dino Now there's a couple of things to note there. One on, you know, and everybody's experienced this, it's taken a patch and it's changed something or broke something, and in the office environment is it really a big deal , for that disruption, that minor disruption, you know, in it can be pretty quick sometimes about rolling the patch [00:03:00] back out if, the patch was too disruptive.

Dino But on the OT space, those disruptions will shut down your plant floor. Which is very costly, which is a big difference between somebody not being able to load their email or get to their email versus , I got a packaging line that's not producing goods. Right. It's very expensive and a lot of organizations.

Dino For example, don't have virtual machines running their hmis, for example, , and so the rebuild of that to uninstall and reinstall can take time, can take hours if not days I've witnessed, especially if you have machines that have older windows operating systems on them. , So that's challenge number one.

Dino The other challenge , in it is that, you know, the, a lot of the OT assets are out there for decades, right? Their life cycle is much different. And in order to patch, you may not have the physical resources on the equipment to be able to even do the patches, right? Memory requirements, process requirements, and there's also patching has to come at [00:04:00] certain, .

Dino Levels, meaning that it's gotta be, you can't just jump from patch one to 10 to get all the other nine in. Sometimes you gotta patch one, two, then seven, and then eight to get to 10. And also in a lot of cases, the SCADA applications or the control system applications have to move with the operating system.

Dino So if you're a Rockwell shop, if you're gonna move the Windows operating system up a few patch levels, you also have to move the SCADA software up a few patch. Patch levels also, right? So it's gotta, they gotta sometimes go hand in hand. And so that work in order to determine that can, again, costly downtime, the work to get it done.

Dino And so in the OT world, they just tend to ignore it because of the disruption to get from one level to the next.

Craig I can see that as if I were, you know, controls engineer, our plant manager, and I'm being metriced and, bonused on uptime. I would be hesitant to, to introduce something into my environment that's potentially could be a downtime.[00:05:00]

Craig Issue. That's a completely different with, you know, if I come in in the morning and I take a patch and my laptop doesn't boot up and I don't have emails for two or three hours, it's no big deal. I can catch up.

Dino You find something else to do?

Craig Yeah. I'll go to the break room and I'll have a coffee or two and sit there and chat with my buddies.

Craig But if I'm on the plant i, can't produce goods for three or four hours or that same time window. There's a hard cost to the organization for that downtime.

Dino Well, there's also the OEMs that sometimes that have the responsibility to maintain and support that machine center won't allow you to patch those systems too.

Dino So you got what we call the OEM blockade, right? Where the OEM is don't, if you touch the machine center, you're gonna avoid the warranty. And a lot of clients haven't done a very good job of negotiating those terms upfront in order to say, look, we have cybersecurity initiatives in here and OT cybersecurity is, is a fairly new phenomenon, right?

Dino I mean, [00:06:00] even though it's been around for several years, and we've said this multiple times through multiple PO podcasts, 60% are still. From the unaware to aware phase of these OT cybersecurity tools that provide them vulnerability management information to give them the advisory bulletins required for the patching strategy in order to patch up.

Dino So,, that's another, , deflection point was OEM saying you can't touch the machine. So that's another barrier that, that the manufacturers are up against.

Craig And, then you take and introduce the safety aspect into it. Let's say that a machine is operating and you introduce a patch that kills a safety PLC or kills a safety program and the machine potential harm to individuals in the plant or the environment or other themes.

Dino Yeah. Quality to choose. Right. So it is definitely a different. Approach that needs to be had , and it is not, is not [00:07:00] aligned, you know, with that situation per se. They just know the plan floor doesn't patch and they just kind of leave it alone, which is why we, see this. This break a lot of times between IT and OT from a security strategy because of scenarios like this, right?

Dino And organizations aren't really doing a really good job of, studying their plant floor situation. And we see it all the time. You know, when you all these mergers and acquisitions, for example, you know, a manufacturer goes and buys a plant, but they're really not doing their due diligence on the situation.

Dino The computing platforms. The networking platforms status on their industrial side. Right. You know what I mean? There could be malware already in there within the plant. Right. A lot of vulnerabilities that already exist. And so the new manufacturer, the new who's buying the, this plant, they don't really don't know what they're getting at the end of the day, you know, in a lot of ways.

Craig Yeah, no, for sure. I'm really surprised that we don't see more, you know. Equity firms that are doing [00:08:00] mergers and acquisitions, leveraging the technology today that we see in the industrial space, to go in and get that true vulnerabilities and asset inventory to say, you know, you're trying to sell me a hundred million dollar plant.

Craig I run my analysis and it's old, it's outdated. Everything you've got, it is dilapidated. I'm gonna give you 30 million 'cause it's gonna cost me 80 million to fix it all.

Dino About 80 million. But it's definitely, it's a huge effort in order to determine the current state of the cybersecurity, , posture within that manufacturing environment.

Dino and they don't have good asset visibility. You know, when they go in there, they're not getting a really good, asset inventory of the environment to determine the status of those machines. And getting back to the research piece, you know, there's technologies out there, for example, you know, 'cause you can't run agents per se on a lot of these OT assets.

Dino And you know, as of today, anybody, so Mark this date, July [00:09:00] 19th, CrowdStrike. Right is having an issue. Right? And it's an EDR in a lot of cases where you got, a lot of businesses are being impacted by a problem between Microsoft and CrowdStrike. Right? It's causing, causing downtime. Imagine having that on your plant floor.

Dino Today, not even having visibility of what's happened in your environment. And so when you think about virtual patching, where you have the ability to take a machine center and put technology in there that knows the vulnerability of that asset and based on that is able to, virtual patch and virtual patching means that if there's remote code execution.

Dino Denial of service, you know, or stack overflow kind of activity that can happen because you don't have the proper patching level in your cvs. There's technology that you can put in front of that machine center that will provide patching for you. You know? So like on a Windows machine, you put patching on , the Windows operating system running that [00:10:00] HMI.

Dino Well, how do you do that with A PLC, right, for example. You can't, you can't put an agent on there and so you have to put, you have to put something in front of it. And a firewall is not sufficient for this, right? So you have to have something that's doing virtual patching. And one of our strategic partners does that and does it really, really well.

Dino And it's a good, fast, quick way to say, look, I can't patch. But yeah, you can't, you know, if you looked at virtual patching solutions, and I guarantee you this, there's so few organizations out there, one that even know what that is, or even looked at technologies to determine how that might be a benefit to them.

Craig Yeah, there, there are. And you made a, great point. , There are other methods. There's not a one silver bullet fixes everything. There are different ways to approach this. You know, some organizations. Put it in a, lab environment and they run, you know, let's call it smoke test, and they screen test and they make sure that it's gonna work with the rev, the revolutions, and the versions that they're gonna put it on before they go into a [00:11:00] production environment.

Craig Some organizations don't have that lab or test environment and they don't have that luxury, let's call it, to do that. So it's really a gamble when they go into a live environment to try and do this.

Dino And again, you gotta look at the vintage. I was at a, manufacturing facility yesterday. They had equipment that's been in there since the thirties and forties, you know, and people would say, oh, smokes.

Dino , How would that be even applicable to today? Well, even though the equipment is that old, the control systems that are in front of it. Or newer, they might be 15 years old, you know? And so the question then is, you know, what can you do? And we have the capability today to put things like virtual patching in front of those machines.

Dino , And my take on it is, you know, it. It's very standardized, right? They run the same computing platforms. They have this typically the same networking PLA infrastructure. They have the same databases, right? They have applications that [00:12:00] are very, , the standardized as far as their ERPs and their CRMs and their email system, et cetera.

Dino You go on the plant floor. None of that exists. Right? And I don't wanna call it the, but you're gonna find a whole host of different technologies spanning 20, 30 years, 10, 15 years in that plant,, and knowing that. Right. It's like, well, how am I gonna even be able to begin to start this, this journey? And you know, it's, it starts with the first step, right?

Dino It starts with the first plan. Again, back to that 60, 30, 10 that we talk about, you know, where are you in this journey? Have you even begun doing anything? And it's a resource issue, you know, because most organizations, because it is so standardized, they can get away with a couple people, you know, potentially running, a particular technology within the organization.

Dino But when you get on the plant floor, you can have hundreds. Of these different technologies, and you don't have enough people to be able to look at one plan compared to another plant. Compared to a third plant. So even if you're a, , let's say you're a beverage plant, right? [00:13:00] And you've got 30 plants in your fleet.

Dino Well, none of those, no. 30 of them are the same per se, right? And they were built at different times. They have different technologies in there, so you have to get back to who are the boots on the ground that are closest to that equipment to give you the most valuable information you can to determine what strategic direction you wanna go to start tackling with these new technologies.

Dino Where you can start from the level of standardization for OT cybersecurity, right? You know, intrusion detection systems, they're agnostic. They look at these things that are 20 and 30 years old, right? And provide you rich content and device information around those machines that you have in your environment that are running on networks, that are running firmware, that are running operating systems.

Dino That are running applications, and start building a baseline. Determine how you might patch what you can't patch, and well, if I can't physically take Rockwell patches and Microsoft patches, well maybe I can put virtual patches around that system right in front of it.

Craig Are we out in the field or [00:14:00] how much vendor.

Craig Participation or support , are we seeing out there meaning, you know, from the automation vendor technology partners, the Rockwell, the Siemens, the Schneiders, the Honeywells to the technology partners that we. Interact with, are you seeing a lot of collaboration between the two of, you mentioned earlier, if you can only go up so many revs until you have to change the operating systems and things, are we seeing publications and notifications from the automation vendors saying, Hey, here's how we should do that, or we recommend this, or what are we seeing from that side?

Craig Anything?

Dino Well, some are better than others. You know, like Rockwell had a lot of. You that out year or so, right? And so they publish advisory.

Dino Some do, right? Some can't because , of the, the particular [00:15:00] situation with the equipment they have without having to go through what I want to call a rewrite. You know, where I gotta, I can't use that PLC in order to get to that level. I gotta take that one out, put another one in. I can't use that version of the SCADA software.

Dino I gotta take that one out, put a new one in, and now you're talking about hundreds of thousands of millions of dollars. For a particular machine center, , to able to get it to a level of cybersecurity posture that, that would be acceptable. Right? And so most, you know, they accept risk.

Dino The question is, well, how much risk are you willing to accept? Right? And look what's going on today. Everybody that's being impacted today by the issue that's going on with CrowdStrike, they have firewalls. That's not preventing this, right? They got a really good EDR tool, CrowdStrike, that, that's having a problem.

Dino We've seen this with SolarWinds, right? We have seen some huge, breaches and vulnerabilities associated with technologies that are used by these clients. You know, most will fall back and say, well, nobody's attacking the control systems. And it's like, well, how do you know that if you're not watching it?

Dino Number one. [00:16:00] Right. How do you know they're not in there researching and discovery in your environment, trying to learn it? Right. We find malware in a lot of cases in the work that we do when we go out there and start doing, you know, our visibility studies , and, , asset inventories within these organizations.

Dino And we start finding this weird stuff, , I was at a plant here again just recently. They used a lot of, , tablets, iPads as part of their, production environment. And they had three wifi networks and there's a production network and there's a guest network. And then there was, , another network for the warehouse.

Dino Well, some of those tablets were sitting on the guest network. They're down below on the control system side that's running the applications within the plan, you know, and the IT security guy, when he saw that, you know, was taken back, it's like, why are they on the guest network? Well, one first question you ask, yourself is why does that tablet even have access to the guest network?

Dino Right. And again, it's because they're not watching it. They don't see the fact that you've got, think of that, that tablet might as well be an [00:17:00] HMI think of an HMI sitting on the GU network. , Is anybody attacking it? No. But is that allowable? Well, the answer to that is also no.

Dino Right? And then what are you doing to solve that problem? You can send out an edict and say, nobody should put the tablets on the wifi network. And then the operator that comes in at midnight and works on the third shift decides that he or she wants to read, look at their personal email. Or maybe they want to access Netflix on the on it, right?

Dino Because they know they can get to Netflix through the guest network. And nobody, yeah. Well, because it doesn't not recognize that's a tablet, that's a production machine that's sitting, that's going through the network to get to Netflix to watch, you know, game of Thrones while the guy's sitting there in the control room working at midnight.

Craig there's a lot of considerations that organizations need to make sure they are looking at as they try and develop strategies. I hear you talk about, you know, in the past where, you know, the plant shut down for one week every year at. Between [00:18:00] Christmas and New Year's, and you've got six weeks worth of stuff trying to cram into one week's worth of time, and everybody wants their stuff put at the same time.

Craig How do organizations try to prioritize, strategize, and make sure that they're taking the appropriate steps , to get the right? Patches in place that are critical or needed versus a luxury or a nice to have.

Dino Well, that's the two sides coming together, right? That's it. And OT trying to come together to make sense , of what's going on in their environment, right?

Dino So trying to learn as much about as of it as they can. And that, again, if you got 30 plants in your fleet. How many people does it take to do that and also you have to get the ecosystem, as I call it, of OT partners, have to also be engaged in this conversation. So whether it's the OEM, who's providing you this machine center or the si who's building out the control systems around that machine center, what is their role?[00:19:00]

Dino Because I can tell you right now, and coming from that, that OT side of the house, from a systems development integration standpoint, it is not normally in the room hardly at all as being part of the narrative of what technology is being brought in or what technology is there, or what kind of remote access do these folks have, and what do they do when they're there to start figuring out how can we start doing a better job.

Dino Providing patching strategy and to provide, you know, a security apparatus around these machines that we have in our plant floor, you know, and have visibility into it. Again, you've heard me say this many times, are you exercising the same amount of due diligence to secure, protect that plant floor as you do the enterprise?

Dino You know, the answer to that is no. We know the answer to that is no. And so the question is, you know, do you lay at the CISO and CIO's feet and say, Hey, you need to figure this out. They're not gonna do this by themselves. They can lead, they can be a part of it. They can help with the narrative, they can help with the budgeting.

Dino They can help describe the [00:20:00] situation to the directors and the C-suite and the owners of a company. But you just can't have it turn, tail and run and go look. I just wanna put a firewall in and try to ignore that best I can. Again, I get back to everybody that's been breached, a hundred percent of them, and I can say that for a fact.

Dino Have had firewalls. Organization. Organization. And it did not stop the breach, right? So what are you doing to get better? And so best practices is get an asset inventory, continuously monitor, understand your vulnerabilities, and start putting some metrics around that thing. Look at your remote access into that environment and what does that look like?

Dino Because OT comes into those plants seven ways to Sunday to get in there, right? They just do. And COVID kicked that door wide open. Because people couldn't travel , and now this air gap conversation, even for those that believe that their machines are still air gap, 'cause they're controlling the cable in and out of the network, right?

Dino When I want 'em in, I, [00:21:00] connect it in or I turn the switch, they get in, do their thing, I turn the switch off or I disconnect the cable. Well, how do you know that gets done? Because everybody that says that, there's always those one or two or three machine centers in there that are still hot, still connected.

Dino Right. Somebody didn't pull the cable, somebody didn't turn the switch. Or when they remote it in via cellular modem, which we find those two, and it's still hot and connected again. Nobody's watching it.

Craig Couple things as we're kind of trying to wrap this up. I would say, what are some.

Craig Significant consequences or potential consequences of not patching. We hear OT doesn't patch. OT doesn't patch, it's gonna cause downtime. We don't wanna do it. We can't test it. So are there significant consequences that could impact for not testing that we could say, Hey. This is the case to do it safely in the right manner with the appropriate technologies [00:22:00] versus just don't do it.

Dino Well, there's also insurance impact, right? Because I can't get cybersecurity insurance 'cause I don't have an asset inventory, I don't have,, vulnerability management metrics in these manufacturing facilities. So cyber, the insurance companies have gotten wiser and said, well, if you don't have that stuff, you're not getting it.

Dino Or your coverages are going way down and your costs for , policies are going through the root, right? So you have that scenario going on. You have those that have been breached who aren't necessarily communicating the fact that they have to their investors, you know, even though they've been forced to do so, regulatory requirements are saying, Hey, if you're a publicly traded company, you're supposed to communicate to your investors within a few days that you've been had a material breach.

Dino And the government has come and been laxed on that, depending on who you are, because they recognize, it's like, I don't have several millions of dollars to go solve this problem. But at the same time, it's like, well, what are you doing to get better? Right. Have you done anything I say smoking years is what [00:23:00] some of them are doing.

Dino Say, ah, I bought a tool. We're doing this again. , You're just scratching the surface. Right.

Craig It's back to that 60, 30, 10 rule. 90% of organizations are just beginning to try and figure it out. You got 10% that are really doing something and making the effort and are on the journey. The rest are not.

Dino Well, and what's the other thing? We heard it again this morning. We've heard it a couple times. What? Why do they fall back when they start seeing the cost? Of what it takes to do this. Right? And so they didn't either budget for it appropriately, or they haven't, , they haven't been able to articulate to the executives and the owners of an organization on why this is relevant and important.

Dino they continue to wanna roll the dice. they'll say That's an acceptable amount of risk. An acceptable amount of risk when you don't even know what that acceptable amount of risk is?

Craig But , that's where organizations like ours come in. We can help them with that [00:24:00] process.

Craig We can help them with budgetary guidelines. We can help them. You've mentioned that asset inventory, we can help them with that asset inventory. There are ways to manage it. You can't just blindly say, we're not going to do it because of X, Y, and Z. You can't. We will build that case for them, help them down that road so that they can.

Craig Go in educated , and present that to the board of directors, the C-suite, to whoever's making that financial decision.

Dino Well, in order to build that case, you gotta get all the wires in the same room, right? Number one. You do, and it's not just internal, it's external. Yeah, there's a lot of external, players involved in your manufacturing capabilities.

Dino Again, back to the OEMs, back to the sis, you know, that are also performing activity and work for you to help keep those machines running. Right? So what is their role in this game? And again, it's an old term I've, [00:25:00] known for decades. You've gotta get all the liars in the same room. Right to help solve this problem.

Dino And you can't just kick it over to it and expect them to do it. Even where we are today for the organizations that we work for, it's, and I witnessed this when I worked for Rockwell, you have to get both sides to start having, a, uh, strategic, serious conversation around this in order to move forward.

Dino Right.

Craig Great points all the way around this conversation could be days we, this is not gonna go away. This is something that's gonna take collaboration from both sides can take strategically understanding the way that it works. But great food for thought for the listeners out there. We hope everybody, you know, you gotta start taking steps to protect the organizations.

Craig You have to make the next. Yeah, you gotta get better period over period.

Dino Both sides have to agree are we getting better? You can't [00:26:00] be just one side saying, oh, we're getting better. Would OT say that because it bought a tool? 'cause it put a firewall in because even OT understand the ramifications of the fact.

Dino Yeah, you put it in a firewall, but I don't manage it. You manage it. You update and patch that firewall on a pretty regular basis, not what's changed to the environment because I'm not watching it on the bottom side to see what holes, you've poked into my environment that weren't there before, that are there now.

Dino Or an OEM who comes in or an SI who comes in and provides vulnerabilities in the environment. 'cause you're not mapping and studying what they're doing either. So again, you gotta get all the right folks in the same room and have a serious conversation about this. And sometimes you have to do it by plant.

Dino Or plant, you can throw out a general blanket strategy and say, here's what we're generally gonna do. But it's gotta, it's gotta drift all the way down to the person that's actually standing in front of the interface on that machine center. Out there at the plant floor.

Craig Correct. Well, great conversation though, [00:27:00]

Dino until next time.

Intro Thanks for tuning in to the Industrial Cybersecurity Insider to stay up to date with our latest episodes. Be sure to click the follow or subscribe button now, and if you found this podcast helpful or have a topic you'd like us to discuss, please leave us a review or let us know. Thanks again for listening.

Intro See you next time.