From Shelfware to Security: Operationalizing OT Industrial Cybersecurity Tools

[00:00:00] Dino: You spent this money, you've been doing it for years, right? I'm not talking about something that just started up. We're talking five plus years of, of actively doing this and you're missing. 70% of your control system. I guess the question you have to ask yourself, why are you doing that?

[00:00:17] Dino: What's the point? What's the purpose of going through this exercise and not delivering what you said you were gonna deliver?

[00:00:53] Craig: Hi, I'm Craig Duckworth.

[00:00:55] Dino: And I'm Dino Bus.

[00:00:56] Craig: What would you like to talk about today? Dino?

[00:00:59] Dino: I think a common theme that I'm seeing in these manufacturing facilities I've been touring this past year, and I want to call it the false sense of security.

[00:01:07] Craig: A lot of that out there.

[00:01:09] Dino: And what I mean by that is we're coming across a lot of clients who have bought cybersecurity tools, some type of OT intrusion detection system platform that they've had for years and spent a lot of money on it.

[00:01:22] Dino: And they've trained people, they've spun up , a resource organization. And what I find is that. The lack of visibility they have into the, true assets that are in their environment. And I mean, by missing, not missing a little, but missing a lot over 50% sometimes as much as 70% of the OT industrial control system assets are not being collected into the baseline on these OT ID tools, and you have ask yourself.

[00:01:54] Craig: so guess there's a lot there to unpack. So I would say one is if somebody say, Ching, done hands washed, we got what we're doing, we're all, it's not our problem, is it? They've got the wrong. People looking at it.

[00:02:10] Dino: They don't have enough of the right people,

[00:02:12] Craig: Okay.

[00:02:13] Dino: right? Sometimes you have to work with electricians.

[00:02:16] Craig: Yeah.

[00:02:17] Dino: People, you know the people who know where the skeletons are buried in these plants, right? you really, you gotta connect with them 'cause they know where that stuff is at. And what I mean by what are they, what stuff are we talking about?

[00:02:29] Dino: The panels that have the PLCs and the hmis and the network switches and the drives, and a whole bunch of other operational technology control systems in them that are connected to the network or to a network.

[00:02:42] Craig: Going through the motion, they're believing that they have this visibility at a very high percentage. Maybe they don't believe they have a hundred, but maybe they're thinking they have 80, 85% and the reality is they're seeing 30 to 35% maybe.

[00:03:00] Dino: Yeah, well, they're skimming across the top right. , and part of it I think is probably the belief is that, well, there's certainly not much you could do. I mean, some of these systems, you know, the ones that I, just recently, , , worked with project team on in the field, we're talking about stuff that was put in the nineties, right?

[00:03:17] Dino: PLCs, right. Rockwell PLC five. For those of you out there that know what those things are and how long they've been around and finding hundreds of them, thousands of them. Right. and yet you're not monitoring them. You bought the tool, now you put a team together.

[00:03:33] Dino: You spent this money, you've been doing it for years, right? I'm not talking about something that just started up. We're talking five plus years of, of actively doing this and you're missing. 70% of your control system. I guess the question you have to ask yourself, why are you doing that?

[00:03:50] Dino: What's the point? What's the purpose of going through this exercise and not delivering what you said you were gonna deliver?

[00:03:58] Craig: Yeah, so part of it is, in that environment is the, let's call it the north south traffic that we talked about all the time. Is that good enough or is the executive leadership, the CISO being. Reported to that they are seeing everything when in actuality that's not the case

[00:04:18] Dino: I think there is, I think there's some miscommunications going on without a doubt, right? Because who's validating the data, If you're not working with the local plant teams, and let's talk about the local plant teams and what's, , comprises that organization on average you're gonna find a dozen or more. Sis or OEMs working inside your manufacturing facilities on a monthly basis, Whether they're working remotely to gain access to that environment or whether they show up on site,

[00:04:46] Dino: And the work that they're doing and the activities, , that they're pursuing and doing whatever they might be doing in life.

[00:04:52] Dino: , Line efficiencies, lifecycle replacement. , Maybe we'll do MES, , expansions, whatever it might be. Product new products coming in. . Nobody's tracking what they're doing from a cybersecurity perspective, and these tools, as we know, these O-T-I-D-S tools are really, really good at, , an audit record of activity, not only showing you vulnerabilities and risk malware, it may be in there, remote access, in and out of that environment.

[00:05:16] Dino: , Assets coming and going, applications coming and going, ports being opened up, changes in the firewall, all these different dynamics that are going in this environment. It's not being conveyed down to the people at the plant level to know why this is important. You know, if I'm sitting there talking to an electrician, and most people would say, what does an electrician have to do with cybersecurity?

[00:05:34] Dino: Well, it could be a lot because they're responsible maybe for the control system, and there's changes in control system. For example, if a filler that's putting liquid in a bottle or a can, if it stops working. That electrician is gonna be brought in to try to figure out what's going on.

[00:05:50] Dino: , and if they don't have access to the data that these tools provide, it just makes their job harder, as it would make the job harder for a system integrator. And OEM who should be asking these questions by the way, you know, trying to understand the telemetry around the machine centers that they're working on, especially if the clients got this tool

[00:06:07] Craig: it's

[00:06:07] Craig: like, saying, Hey, my filler in your example stopped working and I need you to come fix it, but before you come in, I'm gonna tie this blindfold around your head and I'm gonna tie one hand behind your back and expect you to do miracle work. Versus saying, Hey, by the way, here's a tool Maybe they haven't operationalized yet, but here's a tool that gives you everything that you need to forensically investigate and understand.

[00:06:39] Dino: And we have those conversations when, when we go in and we implement, , you know, our visibility studies, right? We, when we go in there and do this work, and we have to work with somebody locally at the plant in order to find all of these machine centers and these panels and these control systems. And so we're kind of explaining to them why we're here.

[00:06:56] Dino: What we're doing, why it's relevant, why it's important, what this information, , pertains to. And it's like the first time they've ever heard it. Right. And it's like, we've had this tool for five years. I've never heard of it. How do you not know that you've got this technology in your plant that's supposed to be providing Overwatch for your control systems and you guys dunno anything about it. that just seems insane to me.

[00:07:18] Craig: Well, they've purchased a tool and they have not operationalized a tool. They're not getting value from their investment. There's no ROI at that point. So if I were. Board of directors or C-Suite or whoever that was responsible for the budget spend, and I saw no ROI after five or six years or however long I'd be question the skills and capabilities of the people responsible for implementing.

[00:07:45] Dino: Right. But again, there's a disconnect between whoever is the owner of the tool. The folks that should be on the receiving end of the data that the tool is providing. and that's what's missing. Not only, there's a cyber piece to it, but there's also the process integrity piece that we talk about, you know, on efficiencies, right?

[00:08:05] Dino: When you go in there and you look at the board and they've got their OEE number, it's 62%, you know, in a discreet manufacturer. Like full well that you know a good percentage somewhere in the neighborhood of two to five, six 8% of your OE number could be better if you leverage this technology to determine what are the things that are being disrupted within your environment that are causing you unplanned, unscheduled downtime, which impacts that OEE number.

[00:08:31] Craig: It does two things. doesn't. Allow them to run efficiently as they should be because they're not tracking what's causing that. Getting down to root cause analysis is providing a very false sense of security to the leadership that they may be in a better position than they actually are.

[00:08:51] Dino: I think the people that are buying the tools are not doing a very good job of explaining it to the organization. and the organization I'm talking about are the people that operate, run the plants of why you're doing what you're doing. we're buying this tool, what its capabilities are, here's the data that's gonna come out of it.

[00:09:08] Dino: This is how it's gonna help you do your job better. They're not doing that. That conversation is not being had.

[00:09:14] Craig: It's almost like reminds me of being a kid when mom would say, Hey Craig, I need you to do this. I would say, well, how come? Because I said so well to me that meant nothing. Explain to me why am I doing it? What's the reason behind it? Is it, am I gonna get hurt? Am I gonna get burnt? is it bad for me or is it.

[00:09:31] Craig: Because I said so,

[00:09:33] Dino: my point is we see this a lot, right? And these very, especially in these very large organizations, these large manufacturers who have, you know, the pocketbook to be able to go spend the millions of dollars to go buy this technology. But then they're just lighting the money on fire.

[00:09:49] Dino: and not taking advantage of it. You know, I equate, it's like shooting a BB at a freight train, you know, and you're not gonna stop the freight train. How do you get in front of it, right? And how, are you gonna make this stuff actionable and valuable to the people that needs to be actionable, valuable to, that's, really the key question.

[00:10:04] Dino: Who's this supposed to be actionable and valuable for? Is it for somebody in some IT organization in an ivory tower, states away. Know little to nothing about what's going on inside the plant, you know, from a day to day operational perspective, or should be the people are in day.

[00:10:23] Craig: Well, it should be. It should be both. To be honest with you, both there's, there's two pieces of this technology. There is the. You know, operational integrity piece that we talk about so often. And then there's the cyber aspect, vulnerability management piece, which again applies to InfoSec to the security teams.

[00:10:41] Craig: And both of them should have access to the data for their respective areas and what they're trying to do. This is not something that should be guarded like it was a material breach or something. And we don't wanna share because everybody's got an NDA, this should be. Shared amongst the organization so that everybody gets better.

[00:11:00] Craig: It's designed to help both them.

[00:11:03] Dino: Well, and it's like they, they try to sprinkle things on top to make it look better than it is, you know? And what I mean by that is like, oh, we put sentinel. The mix. We got firewall in the mix, we've got our TIDF platform. Great. whatcha giving to the plant people to help them do their job with the data that those tools are, uncovering.

[00:11:22] Dino: Right,

[00:11:22] Craig: They don't have a login.

[00:11:24] Dino: I will tell you that CrowdStrike or Sentinel one is, is disruptive in the environment. When it fires off its scans, you know, or it's running AV on the HMI, it's distracting an HMI from its intended purpose, which is to run the filler. again, there's an example of my filler slower over stopping because the HMI wasn't available to communicate with the PLC and the PLC's.

[00:11:45] Dino: Like, okay, if I can't hear from you, then I'm just gonna, I'm gonna go to a safe place, , and then all of a sudden now I've got stoppage or I've got a slow down in my plan right? Then all of a sudden, and it's a domino effect. So you know, at some point they're gonna have to come to the realization that you have to engage the people in the plant to, , help them understand the technology that you've brought to them and how it can help them with the jobs that they do.

[00:12:13] Dino: And it's not that you have to train 'em on the tool, but they should have access to the data. And then somebody help them interpret that data, right? And chances are that the people that are, providing the, technology don't do a very good job or don't know how to interpret the data like A PLC change.

[00:12:28] Dino: If somebody makes an edit to A PLC, to, , put more glue on a box because there's not enough glue going on the boxes that are going down through the palletizer and they make that change from an IT perspective, do you really care? The plant manager might care because now you might be putting too much glue on the box.

[00:12:47] Dino: Now you've got a quality issue and you've, you've got resource consumption issue. You know, that you're using too much glue on the box and nobody's gonna catch it until it gets to qa or it gets to distribution or gets to the store. Somebody calls and complaining, saying, you know how hard it is to open up your box because you got too much damn glue on them.

[00:13:05] Dino: Right? And so the point being is that. Changes need to be in the hands of the people, that they can actually do something about it and understand what amount of glue needs to be applied to a box as it's going through and through the packaging process.

[00:13:22] Craig: absolutely. and even at that, with all of the system integrators, the OEMs that are in the plants helping. Discover and helping fit and helping to troubleshoot all these assets. You and I both know that they're allowed to walk in, plug right into wherever they want to without any sort of controls, change, changes or nothing.

[00:13:47] Craig: they're just like thankful for the help, but Again, that should be part of the process that I think is often overlooked of what is the control for your vendors, your ecosystem, to come in and help support when they're just walking in, it never sees that they don't know anything.

[00:14:03] Craig: They don't even know what happens,

[00:14:04] Dino: Again, operationally is, it's not under their realm of, responsibility or control.

[00:14:09] Dino: Them, it doesn't matter, is the trust to verify, right? You gotta validate those types of changes and activity going on within your environment. But those people within control of that plan can take corrective action, right? And there's a lot of players that are involved, you know, getting in these bigger companies and there's a lot of players between InfoSec network, some global cybersecurity team, some stock somewhere. You've got administrators that are local and far removed from it. Right. like I said, and you have all these different vendors coming into your manufacturing environment and you've stated in, a prior, , podcast that we've done.

[00:14:44] Dino: You know, CISOs and CIOs really need, need to take the time and energy to determine who their largest outsides and OEMs are that they're spending. Your company's probably spending tens of millions of dollars with annually working in plants, and you should know who they are and what their capabilities are and what their practices look like and who are their partners, you know, and what do they have for a cybersecurity practice or a network infrastructure practice within that, their environment, because all the OEMs and system integrators are not the same.

[00:15:12] Dino: , They have varying degrees of capabilities within what they do, and networking skills aren't necessary at the top of the list. Cybersecurity's definitely not at the top of the list,

[00:15:22] Dino: right? but yet they're the ones in there implementing and supporting and changing technology within your manufacturing environment. And so use the cso. Do you care? Is it your responsibility to care? Or is it fault the plant manager or the gm or the local project team working on the project? Whose responsibility is it in that scenario when you have people in your plant working, can tell you that the plant manager and those people that work for that person from a safety perspective or making sure that you are dotting your i's and crossing your T's, that people are still up to work in their plant from a safety perspective. That's great because we go through that all the time. Every single plant we go into, we have to have the proper PPE, we might have to take training. There's documents and videos we gotta read and assign. so there is a responsibility, but when it comes to the messing with digital footprint side of it, there's nothing.

[00:16:15] Craig: Well, again, that goes back to what I've been arguing for months now, or longer. It's. Authority versus responsibility. I have the responsibility as the speak though to ensure that the enterprise, IT and OT is secure. I don't have the authority to tell an OEM. This, you can't come into my plant and connect your laptop and fix this equipment because the plant manager's gonna bump me on the head and say, yes, he is.

[00:16:49] Craig: Go away because we have to get this operation back up and running, and I've got 30 minutes to get it done. And now I've done a 28. So please move out of the way. So again, it goes back to that I may have responsibility, but if I don't have authority to affect change and do actionable items, my title is useless.

[00:17:09] Dino: Right. And, and I think that in that particular case, which I think happens a lot, if everybody's supposed to be doing it, nobody's doing it. and when I'm talking to, you know, one of the key plant personnel whose responsibility is to do just that, to keep that plant up and running, and then you're completely unaware that there's technology within this environment that can help them do their jobs.

[00:17:32] Craig: Shame on them.

[00:17:33] Craig: it increases the attack surface when they're not aware. It makes their jobs, let's call it handicapped, because now they're not given the tools to properly be able to investigate and remediate concerns that happen. it makes it more difficult across the board.

[00:17:52] Dino: Yeah. Well, I mean, and some would probably sit there and say, yeah, but we're running. Right. We still make product. Yeah. But how good.

[00:17:58] Dino: At what cost, You know, if your costs are going up and your downtime's going up, you know, and you can't quite put your fingers on things that are being disruptive in the environment.

[00:18:06] Dino: Because if you can't find something physically, mechanically broken that needs to be fixed, that's an easy one. Right? Versus this intermittent problem that you can't quite put your fingers on. It happens every morning on Mondays between eight and nine, and it comes back on Fridays between four and five.

[00:18:24] Dino: It costs me hours of downtime, and I'm not, I don't know what it, what caused it. By the time we clean everything up, straighten everything up, you know, get ready to, run again and get approval to do that, I've lost 3, 4, 5, 6 hours, you know, at a hundred grand, up top, 50 grand to pop whatever it is.

[00:18:40] Dino: , and I've gotta push it over to the next shift potentially. And then I, they've gotta come up to speed on whatever was causing the problem, you know, versus somebody could sit back and say, well, I can tell you exactly why your filler quit working at three 15 today. we can tell you exactly what happened, and don't, and don't assume it's in the control system, that the control system has this data.

[00:19:00] Dino: It doesn't. especially it's networking related, operating system related, , , remote access related. , Somebody's scanning the network at the wrong time, somebody making a change to the firewall, you know, , a server firing up and a, a new application coming on, a new services being applied. You know, they don't know that it's causing the grief because they're not watching it.

[00:19:22] Craig: and all that kind of ties back to that, false sense of security in. Miscommunication of what the reality is versus somebody going to the plant. Physically doing again, the walk, walking around, validating, doing a gap analysis with what's there and truly getting down to what's it gonna take if, the standard of the organization is, we wanna see 85% pick a number, whatever, it's hundred unrealistically.

[00:19:50] Craig: You can say hundred if you wanna, but what's that number? What is it gonna take? Get to that, and then somebody physically go out there, do a gap analysis of where you are today and what is it gonna take from an effort, time, money, material to get to that number. it's always different. There's nothing, it's, people, organizations are not doing it and they have to start doing it.

[00:20:17] Dino: You have organizations that don't have the capital, right? that's One argument you'll hear. They don't have the capital. Well, but if that's true, then why did you invest in this Coolidge bot? and to get to like your point, I want 90% visibility. Do you know what that takes?

[00:20:30] Dino: There's people out there that can tell you exactly what it takes to do that. The question is, are you willing to spend the money to get there to do that?

[00:20:37] Dino: Right now, you're just doing a small percentage of it. you're taking the easy path, you know, water, half the least resistance.

[00:20:44] Dino: I'm gonna drop a box in, call it good, and it's only seeing half the OT assets in my plant. What about the other half? Do they not matter?

[00:20:51] Craig: Yeah, if you look at K ones, you look at, stock reports. You look at quarterly and annual reporting organizations today. Year over year. Record revenue, record increase as numbers are going up, margins are through the roof. One, I don't buy the, we don't have capital.

[00:21:09] Craig: The reality is you're not allocating capital. You have money. Two. When I look at it and their cost to do business is a hundred, 200, $300,000 an hour a day, or whatever that number needs to be, how can they afford not to do it? That again, you can't afford not to do this. It's gonna be cheaper. Now, back to the preventative is much better and preventative.

[00:21:40] Craig: The long run than reaction

[00:21:43] Dino: Well, the question is, when they start down this path, what are you trying to resolve? realistically, what are you trying to resolve?

[00:21:49] Craig: most of them have not done a business analysis and have not created a use case.

[00:21:54] Dino: 'cause you're doing it right, as I stated earlier, you would have a pretty good idea of who the vendors are working in your plan and what their capabilities are. You'd be able to convey to them what your plan and your strategy is to improve your cybersecurity process and improve process integrity within your plan.

[00:22:10] Dino: And I invariably, I so rarely find anybody in all the plants that, we go into that have put together, , a really good strategy along those lines because an organization such as ours would be at the tip of the spear in those conversations if we're putting in a $20 million packaging system for you, or working in 10 of your plants you know, doing 10 millions of dollars worth of work.

[00:22:33] Dino: Your manufacturing facilities, and we have no idea what you're doing for asset inventory around cybersecurity, around how you're improving your cyber hygiene. we don't know any of that stuff, you know, in the traditional CFI, , a control system integration mindset.

[00:22:48] Dino: Right. so whose job is it to inform these groups? And I would say that, you know, back to that I have the responsibility, but not the authority. as a CSO or a CIO, you 100% have the authority to sit down and talk to those outside and OEMs working in your plant.

[00:23:05] Dino: You don't need the general manager, plant manager, engineering team's approval to sit down and talk to those guys.

[00:23:11] Craig: you could absolutely have the conversation and it should be had, that si should be on speed dial for that ciso, without question. However, if he or he are. On the hook from a, I have responsibility. You and I both know that if they try to stop a system integrator from doing something to bring that plant back online, that'd go over like a lead balloon. Again, they don't have authority to do it.

[00:23:41] Dino: No they don't, but it doesn't mean that they can't put a plan together and start working

[00:23:45] Dino: towards don't wait for the fire to ert. you gotta plan ahead have these conversations ahead of time, you know, at the startups or the kickoff or the commissioning, of bringing somebody in.

[00:23:56] Dino: And then same thing with , support responsibilities, right?

[00:23:59] Craig: Organizations they have to start doing better. I know we talk about this constantly. Over and over. You can't just check a box. You can't buy a tool, not operationalize it, not give access to the departments that need the technology, meaning the OT teams and just keep it for infoset or

[00:24:25] Dino: Right, and nothing in the plant of their gap. You know, somebody's gonna go connect to that device, whether it truly is not connected to anything. Somebody goes in there and plugs into that device and then they take that machine and they go plug it into another machine in your plant.

[00:24:41] Dino: Or they connect it to the plant network while they're doing their work, or it's got cellular or some type of VPN concentrator in, in that machine center. So anybody who says that that machine is air capped is just trying to, to blow you off,

[00:24:54] Craig: Yeah. Air gap is a myth, like kinda like a unicorn.

[00:24:57] Dino: the position you should take for somebody who has that responsibility is. then prove it to me. Right. Trust would verify you're telling me it's there yet. Let's go take a look and I'm gonna ask you certain questions about that machine center. And I'm gonna tell you right now, it's like, yeah, every day at two o'clock, Joe comes in and plugs his laptop out there to make some, some modification or to pull some data off, or,

[00:25:21] Dino: you know, is adding a new feature into the machine.

[00:25:24] Dino: something is being done to that piece of equipment.

[00:25:26] Craig: Yeah. wrap up today, you know, let me, kind of take it back to the beginning. Your example in the plant today, last week they thought they were seeing 70, 80% whatever the number that they thought they were doing. You guys go in, you're, doing this assessment, you're trying to filter through all these panels, and you're looking in, you're walking around your electrician, how do organizations start to try and figure out, how do I get from 30% to 80 or a hundred or whatever they wanna get to? What's the best path How can they start?

[00:25:58] Dino: Documenting what you have, right? Asset inventory is the number one. Pillar in any security framework, asset inventory, and you can look at it one of two ways. You can do it in a snapshot in time, or you can do it continuously, And why do something in a snapshot in time when you can do it continuously, especially if you've already invested in a tool that does it continuously, Because that asset inventory is gonna give you a wealth of information in regards to the current security postured state of those assets, You're gonna have stuff that has been patched in years or a decade, and you might say, well, what can we do about that? Well, how do you know you can't update it? Just because the vendor hasn't done it or the local plant hasn't done it, doesn't mean that it can't be done, number one. Number two, what things are in that environment? It could be disruptive to that stuff that you can start calling out of there. Why is the packaging system seeing stuff from mixing and blending or from the brew house or from utility?

[00:26:52] Dino: There's no reason for data from those systems to be over here or from it, you know, from HR or from some third parties somewhere out there in the world. You can see the traffic flows of information in and out of that particular machine center. Why is that there? The number of applications running on the HMI. Why does it have four 50 applications on it? When should it have 50? Right? Why are there accounts on here on this machine that have been, been there for years and not used

[00:27:21] Craig: Great conversation. Thank you very much. Hopefully you guys listen out there, learn something today. Continue to like and listen. Follow us for more advice and you gotta do something to get better every period, whether that's month over month, quarter, over quarter, year over year. gotta do something.

[00:27:37] Craig: You gotta get started.