Cybersecurity by Design: Building OT Security Into Your Manufacturing Plant Floor

[00:00:00] Craig: In an automobile, you pre-designed in the seat belts, the backup cameras, you don't go and build the vehicle and come back in and retrofit because of cost, because of time, because of constructions related incidents.

[00:00:17] Craig: So again, you, you do the work upfront, you end up with a better product that. Is functionable. So as organizations are going through brownfield replacement, Greenfield build, how do you start taking steps to proactively begin implementing these procedures into your environment and not being reactive? It's more of a shift in, in mindset, I would call it.

[00:01:13] Craig: I'm Craig Duckworth.

[00:01:15] Dino: I

[00:01:16] Craig: What's on your mind today?

[00:01:18] Dino: manufacturing resiliency. How do you build and design cyber resiliency into your manufacturing plant floor environment? Right.

[00:01:31] Craig: Is very relevant, is very pertinent, and I think that as organizations are looking to walk down this path or journey of cyber resilience, they should begin practicing and figuring out how to begin implementing this upfront versus retrofitting on the backside.

[00:01:52] Dino: I think it, it kind of goes back to those sands critical controls points of which for the plant floor, the OT environment, we have five.

[00:02:02] Dino: Right? And, and trying to understand what those, um. critical control points are right for cybersecurity in ot, whether it be your incident response plan, your defensible architecture, your network monitoring. You know, how are you monitoring that ICS environment? , your remote access into that environment and your risk and vulnerability management, right?

[00:02:24] Dino: And so if you, if you follow those five. OT critical control points, you know, that are outlined in the sands playbook, the framework, you know, it has what, 15 of those, maybe ot, we have five. Is how do you, how do you implement that into your OT cybersecurity strategy? Again, overall manufacturing resiliency, but then how do you implement, you know, in a proactive manner, cybersecurity into your plant floor?

[00:02:53] Craig: Well, you know, and I've, I've heard you say this probably more times than I can think that it's, it's like safety. In an automobile, you pre-designed in the seat belts, the backup cameras, you don't go and build the vehicle and come back in and retrofit because of cost, because of time, because of constructions related incidents.

[00:03:18] Craig: So again, you, you do the work upfront, you end up with a better product that. Is functionable. So as organizations are going through brownfield replacement, Greenfield build, how do you start taking steps to proactively begin implementing these procedures into your environment and not being reactive? It's more of a shift in, in mindset, I would call it.

[00:03:45] Dino: Yeah, and I think that organizations need to. A, a, a more comprehensive interviewing process with their OEMs and their system integrators. Right. Because, you know, there's a, I don't want to, silos is the word I'm gonna use, because when you think of the plant floor, a lot of those machine centers, those skids, you know, those OEMs and, and control systems, integration people bring to the table very.

[00:04:10] Dino: Pretty widely, right? It's, it's rare for us to go into a manufacturing facility and say that it's 100% Rockwell or it's 100% Siemens, right? There's usually a blend of different technologies that are being used within that manufacturing facility, right, from equipment that comes in potentially from all over the world.

[00:04:28] Dino: And design and built by different organizations all over the world. So how are you gonna get those seat belts and those backup cameras and those sensors and those airbags, and from a safety perspective, consistent throughout your design, within your plant floor, And so you have to have this, 30,000 foot view of what is our OT cybersecurity strategy gonna look like, right? If you get back to those five critical controls that I was talking about, you know, that stands outlines in that security framework.

[00:04:58] Craig: Yeah, all valid points that during this OT IT Convergence during this journey that organizations take, those are all considerations that need to be thought through. On this approach. It's, it's not a a, a silver bullet. There's not a, you know, one size fits all. There's not a one technology fixes everything.

[00:05:25] Craig: It's that back to the core, basic foundations of defense, in depth architecture. How do you best apply organizations have to look at this as the same way they do safety. It's, and it's gotta get better.

[00:05:40] Dino: which is non-negotiable, right? Safety is non-negotiable,

[00:05:43] Craig: No, it's not.

[00:05:44] Dino: and OT can't take the position that cybersecurity is somebody else's problem. IEI it. Right? Can't say Well, it's got that responsibility. and, and many times and, and the work that we engage with, we'll find OT cybersecurity technologies that have been bought by it, and OT is completely unaware.

[00:06:08] Dino: Of what they, and at the same time, it is not driving a strategy, again, back through that OT ecosystem to make sure that there's some standards in place in order to ensure that you have for instance, ICS continuous monitoring capability within your plant. And, and if so, at what depth? And when we say what, at what depth, what does, how much visibility do you want to have?

[00:06:33] Dino: And if you, if you, if you're familiar with the Purdue model, and if you don't know what it is, please look it up. You know, think of the University of Purdue, right? Not the chicken guys. And look at the Purdue model and determine how far down in those levels do you want to go to gain visibility into those assets.

[00:06:50] Dino: And do not make the assumption that that stuff is air gapped and it's not connected to your network. It is. It you, you will find that it is connected. It, it, it just, it's the nature of the beast, right? If you don't understand how the in industrial control system architecture is set up for the infrastructure, but those PLCs are on those hmis, those drives, those motors, those pumps, those robots, all of that, OT and IOT equipment.

[00:07:21] Dino: You know, everything and remote access, you know, how do people get in there and fix problems? Right? They come on site with laptops. Yes, they do. Do they come in remotely from somewhere in the world? Yes, they do. Right? Do we find cradle points, cellular modems out there? Yes. We, we find those. VPN can, you know, concentrators out there in those panels.

[00:07:39] Dino: Yes, we do. So understand that, right? And if, if you, you know, if you're a medium sized manufacturer, you should find several hundred to several thousand assets within your manufacturing environment. Not a handful, but literally probably thousands.

[00:07:55] Craig: we see organizations, Dino, and this is, this is very common that an IT will will come in, they'll drop a technology in, and they're seeing what's often referred to as that north south traffic, just the, the traffic that's going into the enterprise through a firewall that. They're being, let's call it, allowed to see, because that's the way that the OT teams architected that infrastructure when they built it.

[00:08:22] Craig: There are many layers deep in a lot of cases below that PLC, based on additional NIC cards and other networks that are performing that controls later or often return referred to as the east west traffic. When you look at how do I gather all the data? You, you made a perfect point. What does a hundred percent visibility look like?

[00:08:47] Craig: One, is that important to the organization? Do you need to see a hundred percent? Do you know what effort, time, material that's gonna take to get a hundred percent? Do you know what that's gonna cost you? And it's, it's different for every facility, but. To stand back on a, a pedestal and say, we want a hundred percent, give me a hundred percent.

[00:09:11] Craig: That's a bold statement, and a lot of organizations don't understand what that means.

[00:09:17] Dino: That's the disconnect between 19 ot, right? When you talk about east, west traffic, we're talking about control traffic, right? And that's then, that's very critical, right? The criticality of that east west traffic or that control traffic, it means two things. One is safety, the other is process integrity, right?

[00:09:34] Dino: Because. And so OT and, and their ecosystem, the design and build these systems, go to great links to secure those comms within that control system environment. The question is the integrity of those comms, right? And if you're not watching them, right, it's the old, the old adage, you can't manage what you're not measuring.

[00:09:56] Dino: , and we've been able, the OT environment is all, and I think it is, is allowed this to happen, fall back and say it's air gap, it's not connected. Well, you have to demonstrate that that's really true. And even if it is, how do you make sure it stays that way? Right? Because over time things get connected, and when we're talking about ethernet and, and how easy it is to start connecting devices up, you know, within that environment, it happens pretty quickly.

[00:10:25] Dino: It happens all the time. These environments are very dynamic. They're not as static as you think. And so then that visibility that you thought that you had a good. Good grasp of, all of a sudden you're missing 30, 40, 50% or more of the connected devices inside that manufacturing facility that have software, right?

[00:10:44] Dino: They have firmware, they have applications, they have operating systems, there's networks down there, and nobody is paying attention to what's going on in that space because the control side does not fall into it's realm of responsibility for, again, those two reasons, safety and process integrity, So the only way to really fix this is this OT ecosystem has to be engaged and involved in that. What that cybersecurity strategy is that you're driving. What's your roadmap, right? and, and if you've chosen a tool and you pick, picked a top OT intrusion detection system of some sort and there's, there's many out there, some are good, some are okay.

[00:11:22] Dino: Point is. If you've chosen one already and you've got new machines coming in, new packaging line, back to Craig's Point, brownfield Greenfield expansion, lifecycle replacement activities in that plant, you should be thinking of sitting down with that OEM or that system integrator that's, you know, designing that machine or working on designing your control systems and saying, I need to have visibility down inside that control system space.

[00:11:50] Dino: Right. Design that seatbelt in. Design those backup cameras in those airbags right into that machine center before it gets to your plant.

[00:12:02] Craig: it's easier. You build it into the project, you build it into the budget. You allow for the space, the real estate, the power, the connectivity inside those panels, inside that machine center that we're talking about upfront. It adds. Minimal to the cost of the project, but the value that the organization gets on the backside is 20, 40, 50 fold.

[00:12:29] Craig: What the upfront cost is to do that, and you make it a standard, you say, okay, going forward, this is our standard. This is what we're doing within our OT control space, in our industrial control area, as we. Begin modifying. It goes through data or, you know, technology refresh every three to five years. It's a fact on the OT side, it's, it's a lot longer cycle, so those assets are out there for.

[00:13:03] Craig: 15, 20, 30 years at times. So if you're going to replace something, if you're gonna go through that effort to update that, do it right, do it correct the first time, and build it in so that it's there.

[00:13:18] Dino: It needs to be non-negotiable, like safety,

[00:13:20] Craig: Yeah, for sure.

[00:13:21] Dino: if you have an OEM that's pushing back that doesn't wanna put a sensor inside that machine center to passively collect information, not actively in that environment, passively in that environment to get that metadata down there that's relevant to determine those five.

[00:13:37] Dino: Ot, sand critical controls. Defensible architecture, continuous monitoring, vulnerability management, risk assessment, right? Remote access in there, you know, and what in your incident response plan, those guys need to be part of that conversation. If you've got machine builders that are pushing back, then you have to question, are we using the right machine builder? Is what you should be thinking, right? Because OT shouldn't just be able to run rough shot and say, we really want this machine, right? I want, I want that. You know that Ford F-150, I don't want the Tesla truck, right? I don't want that. I want the Ford F-150. Well, if, if it means that you gotta move towards the, the Tesla, which is probably a bad analogy these days, but my point is, is the technology in that compared to the F-150 is, is significantly different.

[00:14:27] Dino: Right,

[00:14:27] Craig: but I would say there are exceptions to every rule.

[00:14:31] Dino: and it doesn't have to be the cost. Cost is not the override. It, it's much cheaper to do this upfront than to come back to the retro and try to put that stuff in after the fact.

[00:14:40] Craig: Cost should be number three or four on the decision matrix of, of what I'm procuring, what am I buying, or lower

[00:14:48] Dino: But it's a rounding air. You're talking about a, let's say a $2 million packaging

[00:14:52] Craig: yeah. Process selling.

[00:14:53] Dino: of some sort, right? And you're talking about what several grand in technology, you know, on something that's a couple million dollars. It, it's a rounding error, you know, and it's not disrupted to put it in upfront versus trying to do it on the retro because it's a little disruptive because of the downtime and some of the engineering that needs to be done in order to get that, that technology retrofitted in.

[00:15:16] Craig: Well, There's always a, an exception to every rule. If you have one OEM that is the only manufacturer of something that is unique. Purpose built for your industry and no one else in the world makes this widget and you have to use them.

[00:15:32] Craig: Then there may be some negotiations, some Com communications back and forth with that vendor. But if there are five or six different provider OEMs of the equipment you use and the one that you're favored is taking a hard line no of, we are not going to. Then look at the other four. See what you can find out.

[00:15:51] Craig: As long as the feature functionality's the same. But you gotta figure this out.

[00:15:56] Dino: Well, I think that's where the crux of the problem is, is because you gotta think about who the buyer is, who's the influencer on that machine, right? And it's COT guys and it influence on that is, is pretty weak. Is meaning that it, it's almost non-existent, right? Because the OT team doesn't have the knowledge and experience and understanding of why this is important.

[00:16:19] Dino: It does.

[00:16:20] Craig: right,

[00:16:21] Dino: Does not. The problem is, is the people that are driving the decision on the machines that you're, you're, you're procuring and having built and designed for your business. There's no IT involvement. It's primarily 99.99%, if not a hundred percent driven by the OT organization.

[00:16:40] Craig: Again, they're using it.

[00:16:41] Dino: influence.

[00:16:42] Dino: There's no influence from it in that, in that discussion. Right. And at the same time, OT needs to, would have to, to be able to. You know, make the effort to understand that machine's operation its process, you know, how it's put together in order to determine, to assess why they want to get cybersecurity features and functionalities and solutions built into that machine center.

[00:17:07] Dino: 'cause OT isn't going to, they're not in a position from, from a knowledge standpoint or, or the, position of even it being their responsibility. But safety's their responsibility, right? Is, is safety's it's responsibility. Maybe if they're on the, at the plant and they're in their working, they better be thinking of how operating safely.

[00:17:26] Dino: But typically, traditionally, safety's not their job.

[00:17:30] Craig: Well, you know, OT will tell you that security gets in the way of making product. You know, and if you, if you're looking at safety safety's first, and you take the safety approach, then it's an easier conversation.

[00:17:47] Dino: Yeah. And you know, it's always, it, the, the other old adage, it's, it's easier to ask for forgiveness than permission. Right. And you're only gonna get so many of those on a safety violation, you know,

[00:17:58] Craig: and like that's like one or less.

[00:18:00] Dino: One or right. It might depends on the, the, the event and how bad it is, what that incident is.

[00:18:05] Dino: Right. True statement and this, this needs to take that same approach, right? That's how you change and drive this narrative in a different direction. If you, if you tag it with a cybersecurity thought process, I mean, I mean a safety thought process, I.

[00:18:19] Craig: Right. Yeah, no, you're right. Well, hopefully organizations are, are, are trying to figure that out. You know, look at, What's the regulation within their industry? What's the requirement for their business need? Take that requirement back and say, okay, how can we architect this in? How can we build this in?

[00:18:40] Craig: How can we ensure that it meets our security protocol, our governance policy, our risk matrix? And look at it from a holistic standpoint. It, again, you can't do it in a vacuum. That's part of the problem is organizations are trying to do this in a vacuum and you can't,

[00:18:57] Dino: Well, and it can't be the path of lease resistance either. Right?

[00:19:01] Craig: can't check a box. Nope.

[00:19:03] Dino: Right. and, and so, right. When it comes to regulations, it's usually, it's the minimal amount. Right. They're not raising the bar, per se. The minimum meaning that, so for instance, you know, regulation would say that you have to have an asset inventory.

[00:19:20] Dino: Okay. Should it be continuous or can you just get it once a year or every other year or every other

[00:19:26] Craig: in a napkin and I got it when I built the plant 15 years ago. It's changed a thousand times, but I have, I have an inventory. I'm good.

[00:19:35] Dino: yeah. And in invariably. Most of the plants we go into, what do they lack? Documentation. Right. Back to your point. Right. We're, we're, when we're in there trying to do our due diligence and doing our research and discovery, the first thing we go after is documentation. And we get this deer in the headlight look from most clients like, yeah, we don't have that.

[00:19:56] Dino: Or, or they hand us something and 20 or 30% of it's wrong,

[00:20:01] Craig: Or Bill over there. That's Bill that's run the plant for 35 years. It's in his head and he can tell you anything you want, and he's retiring next week and we can't get him to give it to us.

[00:20:12] Dino: right, right. That, that, that happens a lot, especially when the plants have been bought and sold, merger and acquisitions, you know, and to your point on the lifecycle being so different, things just get lost over time, you know, because they don't change 'em, you know, 3, 4, 5, 7 years, they, it's in measured in decades.

[00:20:29] Dino: Right. And so that documentation disappears. Like the dead, you scrolls. Right.

[00:20:35] Craig: Yeah. Yep. Very accurate.

[00:20:41] Dino: Well, I hope this was informative for you folks out there. You know, so if you have any questions, please look us up. , we have plenty of advice and information to provide on the topic, so,

[00:20:53] Craig: i, I think that as, as you listen and as you are evaluating and you're looking for guidance, you're looking for partners, you're looking for a way to. To take the journey. A couple things to remember. Don't do it alone. Choose a partner that understands the industry, understands your business, that understands the cyber physical system arena and how it applies to various manufacturing and industrial layers.

[00:21:26] Craig: Work with them, collaborate. Don't try to be in a vacuum is, is would be my best advice I could think of.

[00:21:34] Dino: Well, you find an organization has a manufacturing mindset. You know, people that spend all day, every day working within these manufacturing environments. Right. You know, too often we find clients whose it organization for example, have, have never stepped foot on the plant floor. I. I haven't been there in years.

[00:21:55] Dino: You know, a chunk of their staff is even unfamiliar with the, the plant altogether. And so you have to change that dynamic, you know, to the extent of you have to be able to, to speak the same language as the people on the plant floor. And you need to understand what it takes to make that facility operate and run and do what it does and, and until you are at that level, from a technology perspective, from an IT perspective, you're really gonna have to rely on the organization that have designed and built that equipment, right?

[00:22:26] Craig: Totally agree. Alright, Dino, great conversation. Great conversation. I appreciate it. Very insightful. Hopefully everybody took something away from this and can go back to your organization and, and try and move one step forward.

[00:22:41] Dino: Appreciate your listening.