Bridging the IT-OT Divide with AI-Powered Insight

[00:00:00] Dino: When those machines aren't running, you're losing revenue, right? That's costing you real hard money. and the OT guys, all they wanna do is get back up and running, right? Because they know the revenue's being is being lost.

[00:00:38] Dino: Hi, this bus.

[00:00:41] Craig: I'm Craig Duckworth, so today is the end of the week. Dino, what you got on your mind today?

[00:00:47] Dino: Well, you know, this term shadow OT or some might wanna call it shadow it. Shadow, it has been a term that's been around for a long time, which are shadow groups within an organization that weren't necessarily reporting up through the IT leadership or through the enterprise organization.

[00:01:06] Craig: Yeah, I'm familiar with that.

[00:01:07] Dino: I don't know if I'd call it shadow IT for the OT organizations that are, dealing with, , technology.

[00:01:14] Dino: Right.

[00:01:15] Craig: I wouldn't either. I would call it ingenuity because the plant team is being ingenuity to get things done and they're gonna bring in whatever they need and go around the policies, procedures that are in place cause we hear it all the time. Security gets in the way of getting stuff done,

[00:01:32] Dino: Yeah, but I also think that, and we've said this before in other podcast, that it isn't necessarily invited to the table if there's some type of lifecycle. Placement activity going on, or an expansion of some sort. When it comes to the industrial control systems, those are run and gun by the OT organization, by the engineering teams, their OT ecosystem.

[00:01:52] Dino: You know, whether it's driving product innovation, , , . Plant floor, , efficiencies, right? Improving operations, improving safety, whatever the case may be. And that gets you back to that think globally, but act locally. And this is entirely an act locally activity for predominantly when it comes to the specific environment within that plant floor.

[00:02:12] Dino: I don't know how you can consider that to be shadow it when the networks involved with those control systems, the applications that are being chosen to run those machine centers, the PLC choices that they're making, the SCADA application choices that are being made, I. It is not making that decision, right?

[00:02:29] Dino: and for this whole cybersecurity conversation, in order to really get to the meat of trying to get the visibility that you want within that plant, you have to breach that somehow, right? It means you have to be able to go into that plant, open up those panels, which in a lot of cases. You have to work within certain windows to do that.

[00:02:46] Dino: There are certain safety activities that have to be in place in order to do that. It is not working in that site every day. You got a lot of plants in your fleet. It's a very hard logistical problem to solve, and so. At some point, you have to recognize that there's an OT ecosystem out there that has a lot to do with the maintaining and managing and support of that stuff, and also along the lines of designing it and implementing it and startup and commissioning it.

[00:03:12] Dino: That has little to no it participation at all.

[00:03:16] Craig: Yeah, I'd say you're right.

[00:03:18] Dino: So on from a cybersecurity standpoint, how are you gonna solve that problem in order to determine the risk associated with those assets, to gain the visibility that you need to get in there, to put the technology in there, to do that, and then manage through when 85% of the data thrown by an OT and intrusion detection system.

[00:03:35] Dino: It's for OT to solve those problems, whether it's a PLC update, firmware update, or somebody made a PLC change, right? And maybe the, at some point the vernacular needs to change from a cybersecurity conversation for another topic of a podcast is, should it be more along control system integrity, OT system integrity, and not necessarily be a cybersecurity.

[00:04:00] Dino: Conversation because that immediately puts it into the hands of it and everybody's thought process, right? And they're just not getting it done. They're hitting the wall because they, don't know how to get past the carpet under the concrete.

[00:04:13] Craig: Yeah. I would agree with you a hundred percent. That's probably another conversation, but you're right, the vernacular needs to change because the minute that the word cybersecurity comes out of anybody's mouth, everyone in the. Manufacturing arena. Immediately deflects sends it to somebody else and says, we've been told that's this person's issue, and they just don't have it.

[00:04:35] Craig: They're just not capable to do it.

[00:04:37] Dino: Right. And so here's a real world example, something that's recent we've been working on, right? So we have a client that every morning at 8:10 AM they lose portions of their production. Their plant floor shuts down, and this has been going on for several months to almost a year's timeframe.

[00:04:56] Dino: And they have an O-T-I-D-S, right? They have one of the, the top tier O-T-I-D-S platforms in their environment. And they're not leveraging, it's something that it put in, right? Something that the OT people aren't using, but said here they have this issue. It's not so much aware of it because in their view, when you tell them that you're having this problem, if they can't pinpoint it for something that's impacting control systems, they don't think it's their problem.

[00:05:20] Dino: And their, go-to is always to look at the no. That's the first thing they're gonna look at on from an IT perspective, is they're gonna look at the network and go, Hey, the network's working fine, right? That they're gonna go all the other applications out there working in that facility. I don't know what's going on with your control systems. So our team is granted access to the O-T-I-D-S platform, you know, which is a machine learning tool, right? It's got AI capabilities and within the first half hour. Sitting there watching this event unfold, they found three servers in the environment. They would fire up at this eight, 10 timeframe and start blasting the network and the hmis and the control systems with all this unnecessary traffic and services. So then that was brought up and say, okay, these are the, three devices, these three servers seem to be at eight 10, which is the timeframe you're telling us that you're having plant production issues, are doing all, is doing all these activities. So then that's passed back to the client. The client's response was, well, those three servers were, decommissioned months ago.

[00:06:21] Dino: well, they're still here, weren't decommissioned, they're still here. that was issue number one. Issue number two is, okay, we'll get those things off the network. So then they went out there and disconnected those devices 'cause they weren't supposed to be there, only to find out that one of those servers was still servicing three hmis.

[00:06:39] Dino: Again, they didn't know it. So my point is, is that you back to the visibility, right? And back to process integrity or control system integrity or production, plant floor integrity, not a cybersecurity issue.

[00:06:54] Craig: we had an engagement with a brewery down in, Texas. We were called in specifically because they were having network communications issues. They didn't understand why they had a third party. Outsourced IT provider in there helping to support them, and we were able to go in with a, top tier IDS platform, diagnose what's happening, what they're seeing, and resolve that for them because of the experience that we have.

[00:07:27] Craig: In that OT network and analyzing and providing insights and data into those IDS platforms, so the business can make a decision. But organizations, they don't understand that. It doesn't get that.

[00:07:44] Dino: Well, I mean, and in that particular case it was, unrefuted, , data, it was pinpointed exactly where the problem was at. Right? In that particular case, you had Siemens in the brewhouse, you had Rockwell and Packaging. You had a ProSoft module in the middle. It was handling the communications between those two different automation technology, SCADA systems. And the Siemens side was throwing so much data over there towards that ProSoft module that was supposed to be communicating with A PLC. The PLC couldn't consume it. It was just throwing data on the floor, right? And so once we're able to determine that one, they had to replace the ProSoft module 'cause it wasn't the right module to be in there outside of being misconfigured.

[00:08:25] Dino: And they solved that problem by having to replace a piece of equipment and configure it correctly in order to solve that problem. Again, had nothing to do with cybersecurity.

[00:08:33] Craig: But they would've never have known that had we not gone in and used an IDS to, well think of it, , from A forensic accountant standpoint, try chasing the money. Someone is white collar crime. Follow the money. All the shell companies, you're going into the network. You're doing forensic investigative kind alike on a network, and you're analyzing that data that's there.

[00:08:57] Craig: Back to those shadow networks that we started on. You would not know those existed if you don't have a Tool in place that's gonna give you that visibility and those data flows to tell you what's happening.

[00:09:12] Dino: and the irony is, is a lot of these companies have bought these tools, right? It's like buying a an MRI or a CT scan or even an x-ray machine, and you have this, equipment that gives you visibility into things that the naked eye can't see. I. Yeah, you're trying to do diagnosis without using the tool, or you don't know how to use the tool in order to find out what's going on in order to determine what kind of, procedure you might wanna do in order to solve the problem.

[00:09:36] Dino: Right. So you bought the CT scanner or the MRI machine and you're not using it

[00:09:41] Craig: patient comes in and says, my arm hurts. What does that mean? So you gotta, diagnose it with the correct technology to understand what's wrong.

[00:09:49] Dino: Right. and we see this, right? And we're starting to see this a lot now because the technology's been around as far as these, you know, operational technology, intrusion detection systems these companies have bought over the last handful of years and they're recognizing they're not getting the value and there's pressure being applied to it.

[00:10:07] Dino: Who bought this tool?

[00:10:08] Craig: Oh yes.

[00:10:09] Dino: and they're getting pressure to go and you're not getting it done. I still don't know the risk associated with my assets in my manufacturing environment. What the heck's going on?

[00:10:18] Craig: Well, yeah, we see that more often than not, I'm working with a client now that that bought. a hundred assets or a hundred licenses of an IDS platform two years ago, and they've got like a dozen installed, The organization doesn't have the ability to act and think locally to deploy that within the environment where they bought it from a global standpoint, and they just can't roll it out.

[00:10:44] Craig: It can't get it there.

[00:10:45] Dino: Well, they're not getting OTs buy-in, right? Because OT OTs sitting there here and going, oh, this is cybersecurity related. That's not my problem. Right. that's their mindset, you know? And, and that's the miss in this conversation is a real miss. And does not how to effectively carry this forward.

[00:11:02] Dino: Because if you're not engaging in the OT ecosystem, you know, and you have to really, when you look at the OT ecosystem in your manufacturing environments, you have to look at who all the players are that are involved. You know, you know, I. I used to say, you get all liars in the same room, right? And there's third parties, right?

[00:11:17] Dino: you got the plant, internal plant people, you have third parties. You got system integrators and OEMs that are in the mix. and OT hasn't had the ability to reign these cats in, you know, to be able to know what the role is, what the technology is, what the responsibilities are, you know who they are, right?

[00:11:35] Dino: And so their, fallback position is as they look at the network and going, as long as the network looks like it's operating well,

[00:11:41] Dino: good. Because I don't have any responsibility for the applications on that side of the fence.

[00:11:45] Dino: they just don't.

[00:11:47] Craig: they're an infrastructure only. At that point, they may provide the compute platform or the vert platform or you know, the infrastructure. But as far as applications that go on it, you talk about the SCADA systems and you talk about the MES, those applications, they're not part of the decision making platform.

[00:12:03] Dino: I, and I have the background 'cause you know my, I started on the networking side of things, right? And when somebody was calling up and asking, saying, Hey, there's something to matter with the network, the first thing I went and looked at was to see what other applications weren't working. And if I found that there was just one specific app that wasn't working, my immediate response was, you need to fix your app.

[00:12:22] Dino: because everything else is working fine. Right. why would I spend time trying to troubleshoot a networking issue when I don't have a networking issue? Whatever problem you're having is with your application or with the server, you know, that's running that app and the machines that are communicating with it, it's like you gotta figure that piece out 'cause it's not the network, right?

[00:12:40] Dino: But today. Now we have better visibility tools that can get in there and really help shine a light on what that interaction is. So you can track the problem from just beyond the networking portion of it, right? So when you think of the OSI model and you think of layers one, two, and three, you know, the network kind of stops at three.

[00:12:56] Dino: And then from there on up you have, you know, the operating system and the drivers all the way up into the application level. It moves to a different group of people to go figure that out. Right? and these O-T-I-D-S tools, they can map out their whole stack right? From the network all the way up to the application.

[00:13:13] Dino: and that's very beneficial. But you have to know the app. And these are not applications that it is responsible for. It's not trying to throw them under the bus. they're not responsible for this SCADA application. They're not responsible for the PLCs. Right? that's not their role and goal.

[00:13:28] Craig: But you know, I think the takeaway is, you know, as organizations are trying to figure out do they have shadow networks on their environment, do they have rogue assets in their environment? Do they have.

[00:13:39] Dino: Yes, they do.

[00:13:40] Craig: Well, we know that, multiple ways for remote access into their environment.

[00:13:45] Craig: how are they validating that? Who's the partner they're choosing to go down that road with them? Are they making the right decisions upfront to try and address these? and if you're not, reach out to us. We can help you. you gotta start doing something.

[00:14:01] Dino: Yeah, the IT folks, what we're basically trying to say is, you need some new friends. and those friends are the people that have a manufacturing mindset, right? Or manufacturing. That's what they do all day, every day. And that's, their scope. You gotta bring them into your tent somehow.

[00:14:17] Dino: Some way you have to start bridging that gap to find out who are our key critical suppliers that are helping us with our manufacturing facilities and get 'em in there and start having a conversation with 'em,

[00:14:30] Craig: Right. No different than, I've heard you say a hundred times. You know, the automation vendors and your key partners need to be on speed dial. If you're spending X amount of money with 'em. It, it's critical from a manufacturing side.

[00:14:43] Dino: because it knows this, but OT really knows this. When those machines aren't running, you're losing revenue, right? That's costing you real hard money. and the OT guys, all they wanna do is get back up and running, right? Because they know the revenue's being is being lost. And, I don't think it necessarily feels that pain point like they do. and I don't know if it's important and relevant , to get 'em that close to the fire, but at the same time, I think it, who's responsible for a cybersecurity strategy and the budget needs to start, expanding their, scope of influence with those folks that are responsible for what's going on inside that manufacturing

[00:15:19] Craig: Yeah, without doubt they do. For sure. Great conversation that I'm certain there are thousands of organizations that need to hear this message.

[00:15:29] Dino: Yeah. And we'll have to hone in on what's, a real message for OT to start being concerned about them, right? And start driving from the other direction. You know, again, whether it's control, system integrity, ot, network integrity, something that demonstrates. It's not necessarily a cybersecurity.

[00:15:47] Craig: .

[00:15:47] Craig: Totally agree.

[00:15:48] Dino: The tools have that, , AI slash machine learning cybersecurity slant to them because they can track malware. They can tell you the risk and vulnerabilities associated with those assets, which is all good stuff. But at the same time, somebody needs to be able to interpret, you know, how's that machine environment actually really running?

[00:16:07] Craig: Yeah. You know, for anybody out there trying to get started, give us a call, you know, do some research, do some homework. We're here to help, we're here to support. , You know, don't be bashful.

[00:16:16] Dino: You can find us on LinkedIn if you can't find our emails.

[00:16:19] Craig: Yeah. For sure. Appreciate it. Great afternoon. Thanks Dino.