AI in Industrial Cybersecurity: Friend, Foe, or Something in Between?

[00:00:00] Dino: anybody who thinks that VPN is the right answer to gain access to control systems, you need to start all over, you have no idea what that SI or OEM or third party is doing once they gain connectivity to that environment and where they go once they land and they're there.

[00:00:44] Craig: Hi, I'm Craig Duckworth.

[00:00:45] Dino: And I'm Dino Boussalaki, and today we're here to talk about our Industrial Cybersecurity Insider podcast. And the topic today, Craig, is going to be what?

[00:00:53] Craig: we should talk about the additional assets that are on the floor that may not be being seen because, again, I don't believe that organizations are truly getting the depth and understanding all the assets that are there. We see lots of gaps. How's that sound?

[00:01:12] Dino: I think that's good, right? Because, we always talk about increasing threat landscape, right? And I think as more and more assets brought into visibility, where you're actually monitoring these, OT assets within your environment, is going to help you determine just how big is that threat landscape?

[00:01:30] Dino: Because I think it's pretty big, meaning that there's a lot of assets that are just not being tracked.

[00:01:35] Craig: and I would say, if we take one little bitty step backwards to that, we hear a lot of times when organizations put an IDS platform in or a tool in that they're looking for the north south traffic. They're looking for that. core data traffic. I think what we're talking about is what you or some of the other industry would recognize as that east west traffic where it's going across below the PLC down in that control layer.

[00:02:03] Craig: That's where organizations are not getting. Is that correct?

[00:02:07] Dino: And I would also say that level on that east west and that environment is probably not a concern of ITs.

[00:02:14] Craig: True statement.

[00:02:15] Dino: I mean, the North South is an easy target, putting in an OTIDS, especially if your IT organization is driving your cybersecurity strategy. Which is good, that's their role, but challenges that they have is, what's the mix?

[00:02:31] Dino: Of actually what's going on inside the control system environment, right? Because there's a lot of different players that are not part of the, OT or the IT ecosystem. And I'm talking about system integrators. I'm talking about OEMs. I'm talking about the local plant teams, that are dealing with all the different assets that they have.

[00:02:49] Dino: and that particular, discussion around that stuff just a murky, dark Spot within that manufacturing environment, right? And we'll be witness that doing some of the work that we do, because, when we go into a manufacturing environment to assist a client with their O. T. intrusion detection system strategy, we actually dig into that particular space, right?

[00:03:10] Dino: And recognize, your North South traffic is picking up 20, 30 percent of the assets in the plant. If that, and the other 70 percent are going to take. Different measures in order to get those assets into that baseline to determine what their current risk vulnerabilities status is.

[00:03:26] Craig: think that a big piece of this is the. Let's call it ingenuity of the OT teams in their inherited ways of problem solving in the last 30 years where they're using PLCs, as example, as network segmentation devices to create those lower level networks and protect themselves from a lack of other methods and ways from IT and other people as they create this space on their own And it makes those lower level networks down in that control space those islands of automation.

[00:04:02] Craig: Let's call it Invisible from that north south traffic flow

[00:04:07] Dino: it's kind of chain of custody is the right word, but it definitely is when it comes to the decision making on the technologies that are being used inside of the industrial control system environment is driven entirely by a different group of people. A different business unit, nowhere else in an enterprise, as we've said before in another podcast, IT's job is to help you manage your business, OT's job is the whole reason why you're in business, and when it comes to making decisions around operational technology, industrial control systems, IT has little to no role And the technology that are being chosen to run that packaging environment or that reactor or those AGVs, or whatever control system machine centers that you're looking to bring in, in your environment is everything from the software that's being chosen to run the HMIs, the switches that are going into the panels, the firmware that's running on the PLCs, even the decision making on the PLCs, you're not going to hear IT say our standard control system technology we use is Rockwell.

[00:05:07] Dino: They're not going to say that. They probably don't have a relationship with Rockwell. The OT teams do, right? , and I don't know if that dynamic is ever going to really change when you talk about this IT OT convergence discussion. Right? Because, how do you change that? when does the CIO or the CISO or the enterprise IT InfoSec teams going to play larger, bigger role in the machine centers that you're going to, purchase, right?

[00:05:32] Dino: And get support from, I've been in many, many conversations. I've been on both sides of the conversation. I've been the team that's selling control system technologies to the manufacturer and, We never engage IT in the decisions making around the SCADA software and the HMI choices and the development of those things to the network.

[00:05:50] Dino: And so when you look at those assets that are out there, I would argue that most of the assets that go in right out of the gate already have vulnerabilities and risk associated with them,

[00:06:00] Craig: it goes back to, and we've said this a hundred times, asset ownership. The OT teams, the controls engineers, own the assets. They are responsible for remediating, protecting, patching, all of the things that come along with the known vulnerabilities. How do you protect those assets?

[00:06:19] Craig: They're responsible for it. They own those assets and it comes back to asset ownership.

[00:06:26] Dino: Well, and it's because they're responsible for production, primary drivers right there change this entire conversation because as soon as you start dwelling into that, you're going to find IT is not going to be engaged in that discussion per se, right?

[00:06:40] Dino: And even organizations that say, well, we've got some IT and OT guys working together to try to solve this problem. You can't do it in a vacuum and you can't do it by yourself, right?

[00:06:49] Dino: bring in that ecosystem of OT partners that you're working with, that are designing these systems, who are supporting these systems, who are maintaining these systems, and in many cases may lock you out of being able to do anything with that machine center in order to keep you from voiding the warranty if you try do something after the fact, right?

[00:07:08] Dino: , you want to have this conversation up front where you're making these machines cyber ready when they show up into your manufacturing environment, right from the get go. So if you have an OT strategy, , one arm of that strategy is going to be what capital projects do we have coming down the pipe?

[00:07:24] Dino: They're requiring us to do a brownfield expansion, maybe a new greenfield, lifecycle replacement, those activities where you would sit there and go, our OT cybersecurity strategy dictates that those new machine centers, that equipment coming in, is going to have this, level of cybersecurity components attached to that, project, right?

[00:07:46] Craig: and it goes back to the philosophy of secure by design as an organization if you are bringing equipment into our plant our facility whether that's a panel a skid a package line, whatever that might be Here are the requirements that we are saying must be there and you have to adhere to that

[00:08:06] Dino: Just like safety. , safety comes in two ways. There's, working in a safe environment and working safely, but the machines also have to be safe, So, there's mitigating controls built around that machine center to make them finger safe, to make it, make them so people don't, open up, if you open up a gate to try to grab something out of that packaging line that's moving, that as soon as you open up that gate, that piece of machine shuts down, right?

[00:08:28] Dino: And we had to learn the hard way in order to make these environments safe. Well, now we're talking about digital safety, right? the undercurrent that right now is invisible to most organizations on from a digital footprint, what is actually going on in that environment?

[00:08:44] Craig: can't draw a straight line conclusion if I'm just going to pick a number out of the air that says if we truly see 40 percent of our assets, let's just pick a number and our Digital safety score, if you want to equate to that, is 60. Just throwing out something there. You can't say that because we have 60% Good for what we see.

[00:09:09] Craig: you're still missing 50 percent of your assets. You have no visibility. Nothing. You're blind. Two of the other half. You can't say well, we assume that they're the same. That's not true. You have no idea what that looks like. So for organizations that are only seeing 20 30 percent, whatever the number might be you have Gaping blind spots in your environment that you don't have any idea Of what's there?

[00:09:40] Dino: right, well, and somebody has a disability. It just may not be the people that are concerned about cyber risk. Right. Like remote access. I mean, on average, we find several different remote access methodologies being used in a single site, right? And you can have one manufacturer that's got 30 plants, 40 plants, and you have several remote access methodologies in place, technologies in place, times 30 plants.

[00:10:08] Craig: Well, there's no consistency. There's no standards. You're not taking the, principles and the frameworks from an IT structure and applying the OT centric tools and technologies to that framework and say, okay, on the corporate side, we've got 30 plants and we've got. 30 enterprises we're using one vp and here it is and everybody uses it but on the ot side It's the wild wild west and you can come in with team view You can come in with tozzy box.

[00:10:35] Craig: You can come in with whatever you want to use and it's okay That's not

[00:10:40] Dino: whatever the OEM chooses to use,

[00:10:42] Craig: well. Yeah for sure.

[00:10:43] Dino: because at the end of the day, they're the ones that are going to get the phone call. And we know that because our organization does the same thing, right? I mean,

[00:10:50] Craig: Yeah true

[00:10:51] Dino: the design group, we

[00:10:52] Craig: Yeah, we are.

[00:10:54] Dino: are a machine builder, right? We build systems, we build these skids, we build motor control centers, we put these panels together. And so we know, right? What's in there, and if we're on the hook to support that, we're going to put if the client doesn't come back and say you're going to use the standardized.

[00:11:09] Dino: Let's let's say it's a clarity's SRA right? Here's how you're going to gain access to these machine centers to do the work that you need to support us. You got to do that up front with them. not let them slide in these multiple technologies we just mentioned into that machine center to gain access, because even then, VPN, anybody who thinks that VPN is the right answer to gain access to control systems, you need to start all over, you have no idea what that SI or OEM or third party is doing once they gain connectivity to that environment and where they go once they land and they're there.

[00:11:44] Dino: Changing other things can they see that they shouldn't see while they're in there? This because of the way that the asset ownership back to our earlier point, anything below level three is wow wow less than your point on how connectivity is put together in a control system, especially around control.

[00:12:02] Dino: They missed that point, right? Everybody thinks of, I got connectivity. I can see the PLCs. But what you're seeing is just data that's being pushed to historians. You're seeing data, that's the engineering programming terminal is gaining access into that environment to do whatever they're doing in there.

[00:12:18] Dino: But what you're not seeing is that PLC machine to machine PLC to H-M-I-P-L-C to drive PLC to robot. You're not seeing that. Interaction.

[00:12:31] Craig: but again, it goes back to and I, may offend some people but It's education people are ignorant in the fact that they are not educated on what's available What's there because this is still fairly new The I. T. tools and technologies do not apply to the O. T. space. They are disruptive. you cannot apply them in a linear, and people are not educated in the fact that these tools are different.

[00:12:58] Craig: From a remote access. If you're not choosing a tool that has audit capability, that has over the shoulder viewing, that has full recording capability so you can see and has granularity to get down to protocol, to a port, to a particular asset's IP address, you've got the wrong technology. But if you were to ask an IT person that's out there Educated on the IT tools and technologies.

[00:13:25] Craig: They're going to say a VPN is fine because we use it every day all day long in the IT Corporate space, but it's different and it back to education

[00:13:35] Dino: To me, it still goes back to asset ownership. At that point, IT doesn't care what happens to that control system per se, because they don't have to fix it.

[00:13:44] Dino: It's up to the operational teams to figure out. How to get it back up and running. Right. And if they can't find a mechanical issue that shut it down, a motor that burned up, for example, right, that they got to replace the motor in order to fill her back up and run.

[00:13:56] Dino: And it's not something easily identifiable, the number 1 priority for that, site is get that packaging line back up and run. Right. Because the revenue has stopped.

[00:14:06] Craig: Organizations are being misled. Rob Lee and everybody knows Rob from Dragos. He made a comment which was a thousand percent spot on. He said, you ask any CEO of any manufacturing organization and they can tell you exactly where the revenue comes within their organization, and that's the manufacturing.

[00:14:27] Craig: when you talk about. Enterprise wide security. It's only the non revenue generating portion, the IT side that's getting all of the funding, all the focus, all of the attention, when all of the revenue is coming from The other piece of it. It's there's a gap and the Executive level responsible party the ceo the board of directors are being misdirected by The cso or the cio because they don't understand the other half of the house

[00:15:02] Dino: in the enterprise space, they try to, blanketly cover firewalls and EDR, which has some positive effect, sometimes negative, but they say for the most part, a positive effect. But you can't take that same mindset and approach and apply it as you get deeper and deeper into that control system, right?

[00:15:21] Dino: You got to think globally. But you got to act locally, that's the disconnect, right? That's where IT cannot, they don't have the resources, they don't have the skill set, they don't have the knowledge of what's exactly going on at that particular level and what it takes for that packaging line to run.

[00:15:39] Craig: Well, here's another curveball into that situation you mentioned earlier If you've got 20 30 40 plants in your fleet, whatever the number is, let's take four of them None of those four are going to be the same from an it Enterprise you can put endpoint protection on every laptop every desktop and it's the same you can control Because it's a uniform Uniform environment in the manufacturing OT, especially control space, it's all different.

[00:16:09] Craig: Those four plants are going to be different. You may have the same team set them up. They may all be greenfield. Maybe they may be close to the same, but they're different. And that's where you run into snags.

[00:16:21] Dino: even in a greenfield environment, you're buying equipment from all over the world built by, thousand different, OEMs

[00:16:27] Dino: that are that are involved in these machine centers. And so then the question is, how are you going to herd those cats?

[00:16:34] Craig: Well, especially when you have an acquisition. We just bought three plants or we picked up a dozen plants or we added to our facility from this manufacturer because they had something that we liked better. Again, you have no idea what that other organization's cyber hygiene is and what you're bringing into your environment.

[00:16:53] Craig: We have no clue.

[00:16:54] Dino: and even then when it comes to M& A, those organizations that help you with that aren't even schooled well enough to be able to help you find, well, you already have malware in that plant. you got a dozen Windows 7 machines in that plant, and to replace those Windows 7 machines is going to cost you another 25 million, right, to do recontrol or rip and replace.

[00:17:12] Dino: What would CISO do if they found a Windows 7 machine in their enterprise? rip it out, root and stem, right? But you can't do that on the plant floor. So now you have to pivot and determine, how am I going to protect and secure the best I can that Windows 7 machine? And if you think a firewall is the answer, again, you're missing the point.

[00:17:32] Craig: Yeah, for sure.

[00:17:33] Dino: somebody has remote access into that machine, number one, or number two, some guy is going to show up and plug into that machine and you're never going to see him or their VPN, their way in all the way down to that machine. And you don't even know what they're doing once they're there.

[00:17:45] Dino: And there are technologies where you can say, well, I can't spend 5 million to replace that Windows 7 machine running that particular process area. But there are technologies out there where I could do something like the virtual patching, or I can segment the network, right? But it's on the control side.

[00:18:01] Dino: And I does not design and build networks on the control side. Purdue model level 3 and up. They got a lot of latitude. You get down below level 2 and 1. I. T. has no latitude.

[00:18:14] Craig: Well, they have no, they have no experience.

[00:18:17] Dino: And so trying to throw a blanket on top of it with your firewall. And thinking that you're going to call it a day, you're missing the point, or even if you do get an OTIDS platform, then you're back to, well, now you're only seeing 30 percent of the assets, you're missing 70 percent of them.

[00:18:31] Craig: and if we bring this back to the conversation beginning, it's how do you begin securing and protecting all of the additional things that are there that you have no visibility into? you gotta bring it back. It's gotta be there. You gotta start with.

[00:18:46] Dino: assessments aren't going to, aren't going to cut it, because those are invalid the day after the assessment's done.

[00:18:52] Craig: Yeah, for sure.

[00:18:53] Dino: these, these environments are not static. They're very dynamic and assets come and go all the time. Applications come and go. Users come and go. New protocols are being inserted in the environment.

[00:19:03] Dino: Unsecure protocols are already in the environment. So the assessment is a snapshot assessments. aren't valid anymore. And why would you do that? Because we have tools now that can continuously monitor this environment, passively, not destructive, like an EDR package running on my HMIs or a firewall that doesn't understand an industrial protocol and decides to block it, which is, preventing my, utilities from sending steam down to clean these vessels out that I need so I can make my next, batch of whatever, Firewall is going, I don't know what SIP is, and it blocks it, or S7 from Siemens, and ignores it, and so there has to be some continuity when you start inserting these tools in there, again, back to that acting locally, because what you do at one plant to solve a problem isn't going to be the same rules that you're going to apply at the next plant, right, because

[00:19:50] Craig: to the all the plants are different.

[00:19:52] Dino: that's a Honeywell plant, or it's an Emerson plant, or, it's It's a hodgepodge of stuff running Modbus TCP and a bunch of other different things, and so again, you got to get back to working locally in order to really dig into this.

[00:20:04] Dino: You can't just like safety, right? Safety is on the site.

[00:20:08] Craig: and again If you look at digital safety, how are organizations applying safe methods digitally to their environment? There should be no distinction between Processes and applications of the mentality, and the, corporate understanding, this, we have to do this no different than a physical safety portion, we need to do that from a digital safety as well.

[00:20:37] Craig: it needs to be the same. The culture has to be there.

[00:20:40] Dino: Well, and when you're looking for the experience, right, the only way you're going to have to draw from the OT side to help solve this problem. Right. There's already a shortage of cybersecurity resources. As just generally speaking when you're talking about OT cybersecurity, well, it's even a smaller group of people that even understand that.

[00:21:00] Dino: And so in order to be successful at it, you're going to have to draw from the OT side, the knowledge of how these machines are designed and built and run in order to assist with that.

[00:21:10] Craig: It goes back to that partnership. We say it, and we've seen it successfully every single time. Where organizations win and are successful is when they partner together, and it is a joint collaboration between IT and OT. And together, they try to tackle , the situation. And they say, okay, let's use some of this and some of that.

[00:21:36] Craig: And the policies and practices from the OT side, and maybe the framework from the IT And Hey, I've never been on the plant floor. Let me walk down there and look and see with my own eyes. What are you talking about? Because I don't fully understand that and really immerse themselves as a collaboration.

[00:21:54] Craig: That's where we win.

[00:21:55] Dino: that show up on the plant floor for a day is not good enough, right? You actually have to work in that environment on a very regular, consistent basis just track how that plant operates, how it does line changeovers, what partners and vendors do they leverage to help them support and maintain that.

[00:22:11] Dino: They allow remote. Access into their, what is their, plans for life cycle replacement? when are they going to start doing an expansion? When are they going to start doing life cycle replacement? When are they going to start remediation and mitigation stuff? Right. If you got known exploits in the environment, And you have to work with partners that are agnostic. that have a wide breadth of not only, cybersecurity technologies they might work with, but also the wide breadth of automation technologies that are spread out through these plans because , it's very rare. And I can say, I can probably count. and I've been in several hundred manufacturing facilities, if not a thousand in my career, I can probably count on one hand, how many are 100 percent purely one automation technology vendor, for example, Rockwell, but even then, when it comes to the, peripheral. IOT assets like building automation and building management, well that's not going to be Rockwell.

[00:23:03] Craig: no, not at all.

[00:23:04] Dino: in utilities isn't going to be Rockwell. Maybe all of your packaging is Rockwell. Maybe part of your, batch process might be Rockwell. But there's going to be other ancillary systems in there that are going to be of different vendor types. So you have to really understand that you're not going to find a singular automation technology vendor in any single plant.

[00:23:22] Craig: Yeah, well the one thing I want to touch on that you just mentioned in that scenario in my opinion the key critical piece is find a Partner that can help you, you look at what we do where? You mentioned we are agnostic across the technology. We are comfortable and familiar and have experience with all of the automation vendors technologies.

[00:23:45] Craig: Pick a partner that you trust that you can rely on to guide you, support you, walk with you on this journey until you achieve whatever it is your organization is trying to do. Are you trying to build a team? Well, we can help you. Do that we can be there to support you while you do that. Are you trying to hire somebody?

[00:24:06] Craig: what does that look like? What does it take to get them on board and get them up to speed and down the road? Get someone today That can help you along the journey to hold you as you go down this path Until you as an organization are competent capable And ready to take it on your own. And some organizations don't want that responsibility.

[00:24:29] Craig: They only want someone else to do it for them. As this happens on the IT side with all the outsourcing. fine. But you can't take it alone.

[00:24:38] Dino: No, you can't. , and again, it's a marathon. It's not a sprint. It's a marathon. And it's going to be an ongoing relationship, just like you have today. all manufacturers have ongoing relationships with their partner, That ecosystem that I call it helped them along the way with the machines that they're using to produce the goods that they make.

[00:24:56] Dino: and that were those relationships last decades. Because , those assets are out there for decades. And so there's always some relationship , that they have with their partners. And it's the same with this. , from a consulting perspective, from a support maintenance perspective, incident response, all of these other things that help them through this journey.

[00:25:15] Dino: You just can't say, I'm going to start, and then I'm going to stop, and I'm going to try to figure it out on my own. you will fail, organizations have to rely on partners as you do, even in your IT space, you rely on your partners, People that you buy technology from, whether you're buying directly from the vendor, or there's a, middleman that piece, you have to do the same thing on the OT side and find competent groups and working across multiple verticals, there's some that are really good and critical infrastructure and you got food and consumer packaging and life sciences and, automotive and tires, got to find groups.

[00:25:48] Dino: In the ecosystem that work in those different

[00:25:52] Craig: Yeah. Agreed.

[00:25:53] Dino: they get best practices, because if you're a tire manufacturer, chances are pretty good organizations. Like, yeah, we've been working in your competitors plant also. even if you're in a similar business, soft drinks versus, alcohol, right?

[00:26:06] Dino: Not so you're not competitive, your colleagues, but the processes are the same, And so you need to work with vendors who work in those different verticals to help you on the OT side.

[00:26:15] Craig: Yeah, I would say we can have this conversation for the next four days and not stop but Unfortunately, we are running out of time So one thing that I would like to make sure that organizations know if you're listening and you're not sure Where to start pick a vendor pick a partner choose someone that's a practitioner that understands what you do and your field of Expertise whether that's like dino said manufacturing in food and bev life sciences Whatever that might be choose someone with a reputation that is an industry leader and help partner with them To get you where you need to be.

[00:26:56] Craig: You can't do not try and take this journey this marathon On by yourself. This is not a one and done. This is a continuous process

[00:27:05] Dino: Yeah, what I would say is if you're an IT executive, you need to sit down and have a serious conversation with your leadership on who is that credible partner that you work with, , who is your go to, when you're having plant issues or you do want to do some expansions or life cycle replacement or anything to do with OT technology, talk to colleagues on the OT side and say, who are your partners?

[00:27:27] Dino: Yeah. Who are you working with? And why are you working with them? and you're going to find out, it's like, yeah, we've been working with a group like the design group for the last 10 years, and they've built 20 packaging systems for us and maintain those over the past 10 years, Well, that might, , that might be a good start, right?

[00:27:44] Craig: for sure.

[00:27:45] Dino: you're an IT executive and you want to get into this game,

[00:27:47] Dino: Not that you need to convince your OT organization to start building their own OT cyber security practice, and help the executive leadership understand this is a different animal, a different beast.

[00:27:59] Dino: I really can't help them well as I could. I don't understand it. I don't have the people that understand that world. We need to start helping them find partners, a budget together. For them, resource allocations because they're going to need this moving forward, I would help facilitate that.

[00:28:14] Dino: So you can either lead it or you can help facilitate it.

[00:28:17] Craig: Yep. Agreed. Well, great conversation, Dino. Hopefully , the listeners got something outta this. , you have a, point to dip your toes in and, go down the road and, begin taking the steps , to , get safe sooner. You gotta start somewhere.

[00:28:29] Dino: Call us

[00:28:30] Craig: Yep. We're here to help. Just start somewhere.

[00:28:33] Dino: Happy new year.

[00:28:34] Craig: Happy New Year.