When CISOs Inherit the Plant Floor: What Happens Next?

[00:00:00] Dino: they don't understand it, and I also think that they don't have the influence. Right. I have the responsibility, but I don't have the authority. Right.

[00:00:08] Craig: That's a perfect example of that. We see that all the time, and we talked about that a couple weeks ago, where CISO now has responsibility for an enterprise-wide security program, including the OT space, but they don't have the authority because they're always stopped at the plant floor because They don't understand or production gets in the way, or you know, for whatever reason they can't execute on their task.

[00:01:08] Dino: Hi, I'm Dina Bus.

[00:01:10] Craig: I'm Craig Duckworth. So what do you wanna talk about today? Do you know?

[00:01:14] Dino: I think we wanna chat about what are the, types of cyber events that can cripple the plant floor and the pathway in is typically through IT systems.

[00:01:23] Craig: I like that idea.

[00:01:25] Dino: You have one that recently came up. Maybe you wanna share what that one was?

[00:01:28] Craig: Yeah, so, so Unify, , was just in the news recently. It was very clear that they had an IT related issue. , But again, they were unable. So Unify is a large wholesale grocery distributor. They were unable to fill orders for their customers based on their. Lack of stalled forklifts at their hobbled distribution centers as they reported it.

[00:01:58] Craig: So again, their OT assets, their forklifts, their automated control or automated guided vehicles that were processing food orders were being impacted. And they were not able to ship grocery items to some of their largest or many of their, clients, whole Foods being one of their largest. So they, in essence, had a contract with Whole Foods and Amazon to provide that.

[00:02:25] Craig: I, again, I don't know the details. Something tells me there's a pretty tight SLA on having food on the shelves.

[00:02:33] Dino: This is not unique to Right. We've seen this.

[00:02:36] Dino: , Clorox, we've seen this with, , you know, a very large global bakery that we know of, there's a laundry list of them that have been impacted by events that, come through the IT environment. And there's many, many avenues that that can happen, that drift down onto that plant floor become impactful.

[00:02:54] Dino: The plant floor is using a lot of the same technologies that it is using, and those environments are usually overlooked from a cybersecurity perspective. , They're not patched as often , as the it space is. and it's very costly when those control systems aren't producing goods or moving goods or whatever they're tasked to do, and, their plant becomes idle.

[00:03:18] Craig: Yeah. But again, it, it kind of goes back to what is the organizational's architecture for their industrial control space? Is it one large flat network that's combined with their it so that it, it makes it easier. For an adversary to pivot, get to the OT space or you know, is there microsegmentation?

[00:03:41] Craig: Are there true traditional firewalls with A DMZ in the middle to try and provide that protection? Think defense and depth architecture and layering, and how do you secure and protect those industrial assets, even if they are on one large network in the beginning until you can take steps to further.

[00:04:02] Craig: Protect yourselves.

[00:04:03] Dino: well, you know, we're working with another very large food distribution company and going through a project where they, you know, they've implemented a, an OT IDS platform, but they haven't really gone out and done their due diligence to determine. the breadth and depth of the visibility within their environment.

[00:04:20] Dino: Basically what they did is they sent out collectors to their remote locations. Sometimes they get installed, sometimes they don't. Where they get installed is usually too high up within , the enterprise network at the site, meaning at the plant level. And it's not really picking up the control system.

[00:04:37] Dino: You know, we always talk about North, south, and east West. They're not even picking up a lot of the southbound traffic, let alone the east west traffic. But what I recognize is the IT leadership that's driving the project. Really lack a fundamental relationship with these business units, with these manufacturers to be able to influence, to work with each other in a manner to really improve that cybersecurity posture.

[00:05:01] Dino: , And we see this a lot where, you know, to think globally, act locally. We say that a lot . whether it's the networking team, the InfoSec team, the IT leadership is, they don't necessarily have the influence with, the local site to get an understanding of that environment well enough to determine how do you architect and strategically implement these technologies.

[00:05:23] Dino: Right. And that's a gap. That's a big gap that we see a lot. It's almost like they're adverse to even trying to work with that portion of the business. I think because it's too difficult for.

[00:05:33] Craig: they don't necessarily understand it fully. I don't believe.

[00:05:37] Dino: well, they don't understand it, and I also think that they don't have the influence. Right. I have the responsibility, but I don't have the authority. Right.

[00:05:45] Craig: That's a perfect example of that. We see that all the time, and we talked about that a couple weeks ago, where CISO now has responsibility for an enterprise-wide security program, including the OT space, but they don't have the authority because they're always stopped at the plant floor because They don't understand or production gets in the way, or you know, for whatever reason they can't execute on their task.

[00:06:18] Dino: Right, and I'd say they lack that inherent knowledge of that facility. You know, meaning that sometimes the IT teams, and when they go to the field with us, it's the first time they've ever been to this plant. Or their visits to the plant have been very, , superficial, meaning that they're just there to put in some networking equipment or work on some project, but it's not really getting out on the plant floor and looking at that, , infrastructure and those control system, , architectures, you know, and really understanding how those things are put together, , and where they're connected and, what are they connecting to within the plant and to, to be able to really get a full understanding of it. Right. And you just can't, do that and be successful, right? You just can't have that superficial, knowledge of your manufacturing environment.

[00:07:02] Craig: let's kind of walk through if, we can, some it OT convergent fundamentals, if we call it, you know, the, difference between IT and ot, the plant floor and critical infrastructure. The most common techniques and pathways that potential adversaries can get in.

[00:07:20] Craig: So if we look at from an IT and an OT side, the IT network is fairly well defined.

[00:07:28] Dino: Right, even on the IT asset. You know, not necessarily all that well documented. You know, we see that lot, I mean, 'cause we always request documentation in order to get, you know, site drawings, network drawings, you know, trying to get as much information as we can. And a lot of that stuff is missing. Right.

[00:07:53] Dino: And it doesn't exist or it, it hasn't been touched in years and they haven't updated it. , and so that creates all these gaps, right? 'cause you just don't know what's been implemented from somebody, you know, who shows up the plant with their computer attack. Maintenance person's gonna work on machine X and they just wander into the plant. And nobody's, you know, reviewed their machine or what they're necessarily doing, or what they're touching, you know, when they're in there. So that's one path. And whether you go in and plug right into the PLC or whether they go plug into a switch down on the control system side, or whether they remote in.

[00:08:26] Dino: Right. Then you have OEMs that will put in their own VPN concentrator or their own cradlepoint cellular modem, and then you'll find four to seven different remote access technologies in a single plant because there's several different OEMs and system integrators working in your manufacturing facility.

[00:08:44] Dino: There's not one, there's a month. Which get back to the podcast that we just did in the past week or so, is that the, I believe the CIOs and the CISOs need to start making some new friends. And what I mean by that is you need to start looking at building a relationship with who the primary system integrators and OEMs are better, , designing and implementing and supporting machine centers in your manufacturing facilities.

[00:09:10] Craig: Yeah. and I believe that today more than ever with the, geopolitical landscape, the continuous increase of ransomware and or, . Activist activities that are, you know, affecting organizations, whether they're doing it for chaos or they're doing it for money, or they're just doing it because they just don't care and they have nothing better to do.

[00:09:34] Craig: it's more urgent. It's more pressing for organizations to take stances now to begin securing and protecting and understanding what they need to do to implement. Security protocols on their OT side to protect themselves. You know, these, acts that we see like unify and. A couple years ago and Colonial Pipeline a couple years ago, and you know, the Toyota manufacturing supply chain, of those are gonna continue and they're gonna be coming more and more frequent.

[00:10:10] Craig: I'm afraid that we are almost becoming numb to the media about, it's another one. It's another one. It's another one. But Unified, for example. Their valuation dropped $500 million in 60 hours. Let that set in for a second, 60 hours, $500 million in lost valuation. That wasn't production. That was market cap market valuation.

[00:10:41] Craig: Will that come back? Maybe, maybe not. I don't know. But just like with Clorox, if the retailers can't sell product, they'll find another distributor or another like product to sell from a competitor to fill that space. And that's what will hurt in the long run.

[00:11:02] Dino: Absurdity of it all is, its prevent,

[00:11:06] Dino: What is it that you can do better? You know, why aren't you driving towards, that end, you know, to get to what I call the no and what I gotta do to secure and protect us. And again, it can't be driven a hundred percent by it and be successful from the plant floor. And if you think that you can pull that off, you're fooling yourself.

[00:11:24] Dino: If you think that just you can purely run this from an IT position and be successful in securing and protecting those industrial control system assets.

[00:11:33] Craig: Well, you know, I talked with a client two days ago, and they had experienced an incident two years ago, and I asked, you know, the question, what's their posture? What are they doing? And the response was, we're better this year than we were last year. Okay. They're making progression, but I'm like, so you still don't have any of these things in place, but you are getting better.

[00:12:00] Craig: So I, commend them for that, but I think that the speed at which they are making progress is less than stellar.

[00:12:09] Dino: Well, and those are the organizations that are trying to do this internally,

[00:12:13] Craig: That's correct. There was one person responsible to try and do this.

[00:12:18] Dino: Witnessed that, right? I mean, especially if you've got dozens or hundreds of sites that they're just putting enough resources behind it to accomplish it. And you don't have the resources internally to pull this off.

[00:12:29] Dino: You just don't, , one, you don't have the expertise. Two, you don't have the knowledge and the capabilities with these technologies and how they fit. Even implementing a security framework, you know, have you even started that? And if you have. Where are you in that journey, in that particular case, especially around your industrial assets, you might say you're doing a really good job on the IT side.

[00:12:47] Dino: That's great. Well, let's take a look at the plant floor and see if you're doing the same thing. And that's where we're gonna find the gaps, right? Getting back to. You know, not being able to do this by yourself as an IT organization, you're gonna have to start crossing into that IT OT space. You're gonna have to start spending some time in those manufacturing environments, and you're gonna have to rely on the integrators and the OEMs and the local teams to pull this off.

[00:13:13] Craig: Yeah. So let's, talk a little bit about prevention and risk mitigation. how should an organization or really kind of what defensive strategies are effective today? You know, I mentioned earlier the DMZ, you got a firewall on the north side firewall. On the south side, there's argument of we only need one firewall.

[00:13:34] Craig: Where should it be? You know, what does that look like? How should they kind of approach? That piece of it to try and start doing something. What does that look like?

[00:13:44] Dino: Before you can start building those types of defensible architectures, you have to have a baseline, right? Which means you have to have an asset inventory that's fairly accurate and up to date. And you gotta be, ideally you wanna be continuously monitoring that. And then once you have that, you should be able to map out there.

[00:14:00] Dino: The communication flows between those industrial assets and the , , external devices that they're communicating with, whether it's within. Line or the zone in the area and the plant. , And what is unwarranted? So what is warranted traffic versus unwarranted traffic? Right. , So you can determine how you can start implementing, you know, intrusion prevention, , types of systems and or changing, VAN strategies and or providing, , limited access into certain control system environments, right?

[00:14:27] Dino: As you start building out that defensible architecture, so asset inventory. Then understand the risk and vulnerabilities. Get into those communication flows to determine what is warranted traffic versus unwarranted traffic into these areas. And that includes secure remote access that's being allowed into that environment, right?

[00:14:44] Dino: And then you can start building that defensible architecture that you're talking about, , you know, and start moving in a general direction. Because we have found that those organizations that have gone down that path, even though other areas in their company have been hit. Have been, unaffected.

[00:15:00] Dino: Right. and it's not because there's a firewall in the mix. It's because there's actually been a defensible architecture strategy implemented around those control systems.

[00:15:08] Craig: Well, if we were to go back, you and I both know Unified had multiple, probably many firewalls in place. That was not the concern. Clorox had many firewalls in place. That was not the issue. So we had a firewall is, is not gonna fix things. but once they put it in and they start trying to figure it out, you know, let's say that something does go sideways and there is an incident or a breach or whatever legal terminology the organization wants to use, how does the incident recovery and the respond differ?

[00:15:47] Craig: From an IT and an ot. On our side of the ot, we are always saying safety first. Safety first, safety first. And we say, you know, how do we safely bring the OT assets back up? Bring them operational. what's your thoughts on that,

[00:16:04] Dino: Well, the first responder is not an IT person in a breach in a manufacturing environment. It's the operator, right? Somebody's gotta be able to get that machine and those systems into a safe place, And potentially get product into a safe place. Right? Back to your point, safety is an issue. , Not trying to destroy, you know, maybe millions of dollars in, in product.

[00:16:26] Dino: that's your first response if you have an incident response plan. Your operator is, the first one on the scene in an OT environment when you have a breach of that nature. Right. , And the question is, do you even practice that and plan for that kind of stuff, right?

[00:16:39] Dino: If you have an event. , And then if you do have the event, do you have visibility into the control systems to see if they're being impacted? By the same virus. that's ransacking other portions of the, uh,, enterprise network, right? You wanna make sure that it's not drifted into the plant realm, because if it hasn't, you know, then you can probably continue to operate, and set up, you know, whether you disconnect from, , the enterprise network, you know, to keep whatever's going on, on the enterprise side for making its way into, the plant if you can operate that way. Most plants try to run in, in a, decentralized fashion, meaning they don't rely on anything outside the plant from a technology perspective to operate sometimes we see warehouses that are centralized and if you lose the centralized warehouse system and the warehouse can, , , become, blind and inoperable, but the rest of the plant can still run,

[00:17:29] Craig: Yeah, I get that aspect of it. it's important. , But from a risk standpoint, from a risk assessment. in converged it OT environments that are there, the risk, and I think in a business needs to understand and really convert risk to dollars and cents. if they look at it from a widget standpoint, it sometimes gets lost.

[00:17:53] Craig: , We make 5,000 widgets a day or whatever that number is, or 2000 a minute or whatever, you know, so that number is. It's non monetized, but if, every widget is a dollar 25 and you're making a thousand of 'em, you got $125,000 in a wac, whether that's an hour or in a minute or in a day or whatever it is, a organization understands dollars and cents, and that's something that the risk manager, the CFO can equate to and see what does that downtime.

[00:18:24] Craig: Cost them, or what would their potential loss be? And then begin looking at it from a remediation standpoint, from a cost dollar valuation of does it make sense to do this for the return?

[00:18:39] Dino: Yeah, well, in my view, when it comes to those types of, , , OEE, you know, operational equipment effectiveness, you know, where you're, you're able to determine your efficiencies, which determines your profitability or your revenue generating. I would argue that. It is not really close to that because we provide those types of systems, right?

[00:18:58] Dino: We provide, , MES systems and line efficiencies and OE types of, , activities on the control system side, but we're never working really with it on that. It's usually all around the plan operation and engineering teams that we're generally working with on that particular front. I don't think that it views that a downtime window.

[00:19:19] Dino: Regardless of the event is their problem.

[00:19:22] Craig: I would say you're a hundred percent correct. They are not concerned with because the priorities are different. On the operational side, we know priorities is uptime first, and on the IT side it's data security and they are not the same.

[00:19:37] Dino: Yeah. I mean, again, it may have some oversight role on what's going on, but the people that have to determine what it's gonna take. To get those systems either safely shut down or bring them back up online is purely done by the OT organization. , You know, all's it can really do is make sure that the network is still there.

[00:19:58] Dino: , But if they're not down there, reviewing the OEMs, , machine center to see if there's malware in it, right. If they even have access to it, , would even be granted access to it. , Again, it gets back to my earlier statement of needing new friends where you're gonna have to build a relationship with these suppliers.

[00:20:16] Dino: Who provide these technologies and solutions for you, , so that you can start, you know, driving a cybersecurity narrative with those guys and what it is that you expect and what you would like to see out of them to say that, they're doing this in a cyber safe way. , We're starting to see some of that language in the, , RFIs and RQs that we're getting, where they're asking about cybersecurity practice capabilities, , of the engineering organization.

[00:20:41] Dino: Right. And who's reviewing that remains to be seen because a lot of these organizations, we turn around and go, okay, do you have a cybersecurity tool, platform in place that we should be leveraging as part of the telemetry around the project that we're working on? Right. Which we should be after also, right.

[00:20:57] Dino: From that perspective, what is the telemetry around these machine centers, , to determine. how I can execute my job, make sure that there's no malware down here, make sure that there's no unwarranted traffic in here, or remote access in here. , If I got into life and the support assets that are involved, , so I can understand the risks associated with them.

[00:21:16] Dino: And, so there's a whole approach to do it right. You have to have, whether you're following the five sans OT critical controls. Or the NIST or IEC 6 2, 4, 4, 3, , others that are out there, , the organizations are, what framework are you following, if at all? Right? those are the conversations that need to be had going back and forth between the client and the integrators and the OEMs, it, ot on the client side, you know, getting all the liars in the same room as I call it.

[00:21:47] Craig: Well, another piece of this that is often overlooked that needs to be. Absolutely incorporated are the iot devices. When I'm out on a plant floor and I walk around and I see a hundred cameras, whether they're monitoring a, a production line or they're looking for, they're scanning something or the printers that are printing nonstop for whatever the process is.

[00:22:10] Craig: A lot of these. IOT devices that are in the OT space helping production and facilitate and do these things are often overlooked. That firmware is not normally updated. They're normally put in with default passwords because they don't think of them as an operating system, something that could be an issue, but the manufacturing floors are riddled and laced with iot assets and.

[00:22:37] Craig: I hope that organizations are including those as they're trying to figure this process out.

[00:22:43] Dino: Well, and as you're describing that, the question is, you know, from an IT perspective, is that your responsibility. I would probably argue that it's not right, like the PLC firmware upgrade, you know, or drive upgrade, you know, or HMI or DCS, you know, when it comes to those magnitude of control systems that are out there, AGVs. It is not driving that remediation and mitigation, you know, they, because they don't know, what it takes to do that for more upgrade, you know, what other things have to go along with that? How do you test it? How do you back out? Does it break any of the applications surrounding that firmware as far the systems that is communicating with, you have to be able to somehow engage that and if it's an OEM piece of equipment.

[00:23:27] Dino: Then you gotta go to the OEM and have that conversation. Right? And depending on your contractual relationship with 'em, they may be able to assist or they may tell you, no, don't touch that machine. So then the question is, well, how do you surround that machine isolate it even though you still need connectivity in and out of it?

[00:23:43] Dino: Well, then how do you manage that? Right? How do you put a microscope on those conversations coming in and out of that machine, whether it's the OEM remoting in or that machine talking to another machine? That's part of the process to be able to make sure that nothing changes. I got my baseline and now I'm just looking for anomalies.

[00:24:00] Dino: I'm looking for malware. , Are you getting to that level? And if you are, who's doing that? And I would argue that that's not an IT team doing it at that, level.

[00:24:09] Craig: Right. No, it, definitely is not.

[00:24:11] Dino: And so the OT teams have to really step up their game. And so now you have to ask yourself, the question, well, if that's the case, then it, I'm out. I don't have a role

[00:24:22] Craig: Right. they need to have some role. And because again, in every single client that we've gone in, that's been hugely successful. We've seen it on both sides. We've been successful on the IT side. We've been successful on the OT side exclusively, but where organizations collaborate, work together and to tackle this from a combined effort of IT and ot, and they share skill sets and they share frameworks and they share.

[00:24:49] Craig: Information that is the everybody wins scenario and that's what we like to see the best. as we come to the end, let's look at this. What's one or two pieces of advice that we would give or you would have for organizations that really don't know where to start? Are trying to figure it out?

[00:25:08] Craig: We talked a lot about, what happens, but. You gotta do something. You gotta, every period, quarter, year, month, you gotta be getting better. You have to progress toward a better hygiene. Where do you start? Can't eat the elephant all in one bite.

[00:25:24] Dino: Well, you gotta go interview who it is you wanna work with for starters, right? I mean, some of 'em may already be in your house, you just don't know it. And I'm not talking about on the IT side as much as I'm talking on the OT side, obviously, there's, vendors out there who will sell you hardware, software all day long,

[00:25:40] Dino: and it's up to you to figure it out, right?

[00:25:42] Dino: And then, you know, as far as a good practice is to go find organizations that actually take that stuff and implement it and have done it, you know, in hundreds of plans across dozens of verticals that actually has the experience and the references to demonstrate their capabilities around this versus somebody who either just dabbles in it or who's just selling it The question is, as the client, as the manufacturer, are you gonna build a practice? are you gonna do that? Or is your intent to just pick a tool and just kind of, Around with it yourself, which we've seen that too, and then ends up becoming shelfware slowly because again, you get that gap between IT and ot.

[00:26:26] Dino: They can't get far enough down into the space. They're not sending their people to the field to really do this discovery work, to really get after, getting a clean, clear asset inventory. Right. we see that so often. So you have to find organizations that do this on a regular basis all day, every day.

[00:26:45] Craig: And they can, leverage organizations like ours or someone within their environment today to give them that guidance support to work alongside them until they have this. Skill sets until they are capable on their own. Or if some organizations may say, you know what? I wanna take the IT approach and be a hundred percent hands off and I wanna outsource all of this to another organization that has the subject matter experts to guide us and and lean on for that piece of it.

[00:27:16] Craig: And that's okay too. But don't take the road by yourself. There are all kinds of support systems, all kinds of. Resources available. Reach out, talk to somebody, call us, call somebody different, if that makes sense. But don't try and tackle it on your own. there's all kinds of options.

[00:27:37] Dino: even the vendors that you're talking to from the technology perspective. We've gotta look at, the partners that they have, right? The relationship. And then it's not just so much, you know, talking about cybersecurity and say, oh, these guys are an amis, or Clarity, or a Zomi, , partner. But also they have to be Rockwell and Siemens and in Ellucians and, , Emerson, you know, working with these automation technology. So if you're looking at buying a cybersecurity. Technology and or services from somebody. The next thing you need to look at, it's like, well, who are your partners? are you a Rockwell partner? And if the answer is no, then why would I have you help me do this? There's a whole nother organization that needs to be involved in this. And if they're not working, like I said, with a Rockwell or a Siemens, or an Emerson or a GE or a Honeywell or a Schneider, they don't do anything with those guys.

[00:28:31] Craig: Totally agree. Totally agree. I think that's all the time we got Dino. Great conversation. For those of you that are listening, make sure you're reaching out to your partner community, you're understanding the ecosystem that you're playing in, and leverage the resources available to you.

[00:28:48] Craig: Call us, reach out to somebody else. Do something you gotta continue to improve and, start working on securing your environments.

[00:28:57] Dino: Otherwise you'll lose millions.

[00:28:59] Craig: Hundreds of millions, like 500, that's a big number, 500 million in 60 hours. Let it sink in that's gonna hurt a lot people. So, yep.