Four Distinct Companies & One Critical Gap—The Ownership Crisis in OT Security

LuRae - Intro Welcome to the Industrial Cybersecurity Insider. In each episode, we dive into the world of industrial cybersecurity. Join us as we cover the latest trends, innovations, and practical insights as we talk with leaders and practitioners across the industry. Gear up and let's get into this week's episode.

Dan Ownership that's so core to the problem space as it exists in all of it. Really, it all comes down to ownership in both understanding who is the owner, but also making sure that they're at the table when decisions are being made, so you can't have a different group. Making a whole bunch of different decisions.

Dan So let's say, let's throw an example out there. They have the IT security team. They suddenly have a mandate to put cybersecurity in the OT environment in this new production environment. So they [00:01:00] make all the decisions themselves that don't, uh, that don't. Consult, get any input from, they blow into the, uh, to the manufacturing floor and say, here we go.

Dan We've gotta go and deploy all of this stuff just to be rebuffed and told, take a hike because you didn't consider anything that we need to manage here. The priorities were different, the objectives were different. Seen it happen a lot of times,

Dino right. Process and production and safety. Safety's job one right on the plant floor is so I think sometimes it kind of miss that and, and their practices and the technology, quite frankly they use, are very disruptive in, in a industrial environment.

Dan Yeah, that's absolutely the case. And a while back as I was thinking about this problem. A good analogy struck me on how to describe in a lot of cases and not being done in a malicious way. It's kind of just how we go about approaching problems. So, so if we consider IT security, so it has its approach to doing ic, IT security things, right?

Dan So inside the data center, inside the [00:02:00] on-premise corporate environment. Kind of know how to do that. Yeah. There's changes in developments for the most part. There's a pretty well-defined approach to take. Then along came cloud computing and. Now the, the IT security teams were responsible for securing these cloud environments.

Dan So they were the wild west initially, and then bad things started happening, and now requirements that existed that they had to be secured. So the IT security teams. Were able, for the most part, to take their knowledge and their approach, their experience and apply it to the new domain of cloud security and how do we approach this to provide the security for it.

Dan Of course, there were changes that have been required and new technologies that come out, all of that kind of jazz. But for the most part, that skillset, that experience could transfer. So now enter the space is OT security and the same thing's trying to be done where it's, okay, well this works for it. It also works for cloud.

Dan As I now approach this [00:03:00] new security problem, I'm going to take the same mindset, the things that I know, the approaches that I know, and apply them into this new domain. The issue is that without understanding the restrictions and limitations, you don't realize until it's too late or hopefully someone stops you.

Dan That as you mentioned, what you do is gonna be very disruptive. You can't take a stop things and then fix it later if it goes wrong. You have to. Adapt any approach that you take to identify the priorities and respect the priorities of that operation. It's a mindset change that's needed for a lot of teams to say, okay, I have this wide domain knowledge, but it may not be directly applicable and I should wait.

Dan I should work closely with these other teams. Collaboration is so important.

Dino Yeah.

Dan To understand why and then go forward.

Dino Yeah. I like the cloud [00:04:00] analogy because, and even the client that was wanting to leverage the cloud still had to rely on multiple external resources to help them. Right. The cloud vendor.

Dino The security that that vendor's using and or third parties and the technologies that you're deploying and buying, whether it be Palo Alto or Cisco or Juniper, or CrowdStrike, what have you. It was a culmination. Organizations to help with that. Right? It wasn't just the client pick company A who wants to leverage the cloud and say, I got this all figured out.

Dino I'm gonna start leveraging cloud. They had to rely on those third parties to help them. Right? Right. And I see that as a little bit of a gap on the OT side, right? Because I hate the term here of whose truth is actually real. Right in this discussion. Right. And how do you make that happen with OEMs, sis plan, operations teams, engineering teams, IT InfoSec, network, right?

Dino Plan [00:05:00] management risk. All these different groups that you gotta bring to the table to tackle this area that we're concerned about, which is that industrial control system environment. So the cloud that, that's a good one, right? So whatever playbook you had to use then to bring all these people to the table, you have to do the same thing.

Dino Again, if you really wanna get after your plant floor, cybersecurity's posture and strategy.

Dan Ab, absolutely. This is definitely even more broadly than this for OT security in general. I think by identifying that, hold on, maybe we have a playbook that we can reference here. How we went to cloud, how can we use that?

Dan But even just OT security versus it. I think we can learn as an industry, we can learn from the mistakes, boy, where there's some missteps in it, security, some things that didn't go well, some approaches that ti the security organizations took that just really rubbed everyone the wrong way. Essentially the, and I hate when I [00:06:00] read, when I see it, that it's always the user's fault or the user's always wrong, or the user knows nothing or it's a user error.

Dan Well, it's because you never told them what to do in the first place and then you yelled at 'em when they didn't do it.

Dino Yeah, we see that a lot and we, it's the finger. Quitting game right between IT and ot and OTs has got a little bad taste in their mouth because they've had disruption, they've had shutdowns, not really sure what happened.

Dino It is claiming they didn't do anything. It wasn't on their end. Nobody knows what really happened, or they do find out what happened. And it was, it scanning the network at the wrong time trying to do their due diligence, but not realizing that they were gonna be very disruptive in their activities. So, because I will tell you as, as a, um.

Dino Long term manufacturing guy and working for a group. The size of design group, the Barry Weill teams. We got hundreds of clients do tens of millions, hundreds of millions of dollars worth of business every year. And I will tell you right now, nobody calls us uh and says, Hey, we need to [00:07:00] tell this project that you're working with us on.

Dino We need to sit down and have an OT cybersecurity strategy planning session.

Kevin Mm-hmm.

Dino Think about that for a second, right? Never once is a CSO or A CIO, or the engineering team came to us on a $25 million project, which we're doing for them, or even a new plant. Build out a $200 million plant build out. And nobody's sitting down to us and going, Hey, we need to all these machines, all these systems, all these networks that you're bringing into this manufacturing environment has sat down and said, we need to talk about our cybersecurity plan.

Dino A lot of times we're not getting phone calls from the CIO or the CISO or the IT leadership having us come in to talk about the project. That we're getting ready to do, whether it's Brownfield, Greenfield lifecycle replacement, recon control integrations, where they're really digging into what it is that we're bringing to the table for them.

Dino Do you, are you seeing that from an artist perspective? I know you guys kind of, you work with it quite a bit, but what does that move towards OT [00:08:00] look like?

Carlos It is very diff difficult, right? So, and then that's why I come in the conversation because having that operational experience from an OT perspective, I can actually understand the positions from an OT perspective and also having the opportunity to be a system administrator, network engineer.

Carlos Then I can also see the perspective from the IT and I try to kind of join forces and see the positive between the two and the conversions between the two. But the one thing that we always need to understand. Is it networks? Andt networks are completely different and they're designed for different reasons.

Carlos Like the IT networks are typically designed to improve productivity and provide more tools to the users to be more productive and to, you know, the more tools, the better in a way. And of course. Great power responsibility, and then you have to create or put more security in place and add agents and to monitor the traffic and to monitor the, [00:09:00] the information that's going in and other devices.

Carlos So it becomes very process intensive when it comes. To understand the itot networks, the OT networks on the other hand is completely the opposite. The all the OT networks are defined for a purpose, and there's the purpose of communicating ICS devices and perhaps some other iot devices like cameras and TVs and handheld, but dedicated and purely dedicated to the process, the OT process.

Carlos When you talk to a plant manager. You know, there's only two things in, in his head. Keeping production improving. Pro i, improving production, but also safety of the people. So then when, when you actually think about a control systems, especially when it's actually critical systems, if you ever lose visibility of a naughty process like operators, the display of the operators go frozen, or there is no communication to the controllers.[00:10:00]

Carlos That's actually reason enough. To do a shutdown, and if you actually gonna install agents from an active perspective to protect those devices, then there's a risk and there's a risk that increases very rapidly. On top of that, you also have to co to consider that this. Devices are maintained with very strict SLA when it comes from the vendor perspective.

Carlos So what that means is that if you have a system that is driven by Yoko g Honeywell, they will actually have a very strict requirements of what's, how do you configure the active directory, what EDR you can actually install in the machine while in points and so forth. So there is a lot of. Really requirements that it needs to be met.

Carlos And if you let it to start in installing, then you could actually potentially break that SLA and then lose the support that you need to maintain operations. So it seems like an impossible task. To, to join them together. [00:11:00] And in fact, when going back to my role in, in the energy company, so we were able to kind of work together in, in one effort.

Carlos And that was actually to build a new, uh, control operator controls area, right? So data center with the, the control room. And the way we did it is like we have the SMEs from the control system, then we have the SME from the IT system, and then we actually create common design. But the common design, it wasn't merging the two networks together.

Carlos It was still, we have our subnet, if you wanna see it that way, of the OT and which are actually managed by us. And then the IT network. And then we have of course the DMZ and then two firewalls where one firewall was controlled by the OT net OT team and the other firewall, the OT team. But the good thing is that we created.

Carlos A design where all the drawings were the same. Ht NOT have the same drawings, all the procurement was the same. So we [00:12:00] actually used exactly the same hardware across the board so that everything was actually a lot cheaper and everything that we designed was in, in. Taking that into account, Hey, we know what devices we need to connect across from the OT to the it, and then we designed the policies and the firewalls with a what listing procedures to make sure that all the communications that we needed from pi, for instance, to, to this CADA service and so forth were in place in both files and so forth.

Carlos So that worked really well because it was very efficient. 'cause we didn't have to actually ask for permission because the design was from the ground up together. So there is ways that we can actually work together for sure.

Dino Yeah, that's a good point. I call 'em the aftermarket parts when you have it. For example, as an si, we don't implement EDR.

Dino And design EDR into the control systems that we're providing to the client. Right? Because to your point, they can be disruptive, right? Yes. Yet we'll [00:13:00] find in a lot of cases after we've done that work, that it will come in a later date and install a CrowdStrike or Sentinel One in the environment, and all of a sudden you start having odd problems within your manufacturing environment you can't quite put your finger on, which is really good when you have tools like Arm in place because it's gonna show you that.

Dino That disruption. Right,

Dan exactly. Yes.

Dino That's a good point. Right, and, and for the IT people out there listening to this, think about that before you start just throwing a CrowdStrike or a Sentinel one or some EDR package into that OT environment around those hmis, because it can't be disruptive if it's yes wording communication, scanning at the wrong time, doing its av, and all of a sudden that HMI freezes for 30 seconds and all of a sudden my process is at risk.

Carlos think the first question they need to ask themselves, and it's actually something that I will actually communicate to the IT people, is it qualify by the vendor, the, the manufacturer of the system that you're, you're install wants to install [00:14:00] it and if you can't answer the question, you need to investigate which vendor is, is being used and then whether or not that the vendor supports the EDR.

Carlos That's actually very important part that it people. Tend to to ignore.

Dino Right, right. Well, let's talk, and I wanna focus a little bit on a term that I heard at us for this past month in February, is cyber informed engineering. Right. Which to me goes along with if you have a client who's it's chosen ARM is their OT cybersecurity platform, and you've got projects going on in your manufacturing environments to be looking ahead of saying, well, we're an arm.

Dino Shop, and I've got all these new machines and skids coming in over the next year or two, three, whatever, what should they be doing in order to ensure that they're improving and increasing their cyber posture within the design build of those new machines?

Carlos Well, there is a lot of [00:15:00] things that you should be doing.

Carlos The first, of course, and the first and foremost is that you have to make sure that you have that visibility in place. Okay? So it it, the problem that you have is that you actually adding things into the network packages and new equipment and new lines, and you lose focus of, you know, what devices you have and you're maintaining and, you know.

Carlos Not seeing the devices that you have. What is gonna happen is that you're gonna have risks and vulnerabilities that you're not aware of, and they could actually be introduced into the systems without noticing. Okay? So if you have a continuous monitoring of what devices are actually into the network, and then you start looking at brand model framework versions that they're using, and then match that with the vulnerabilities and the risks, uh, on top of that, then.

Carlos What these devices that tend to communicate are they configured properly. So that's actually very important. Okay. And one thing that [00:16:00] I find as well is that sometimes when you actually bring equipment, depending on what they have in the stores, they might be already old. And meaning that, you know, you have a device, A PLC, that you know, even though you're buying it brand new, it might have a a, an out of date framework, for instance.

Carlos And before you're actually. Put it into production. You need to actually check whether or not you can patch it and you can actually get it up to date. Because one of the things that you know typically happens when you're in production is that you can't shut down whenever you need to patch or you find a vulnerabilities.

Carlos That's why you have shut down windows, right? So that you can actually start looking at, you know, what you need to fix. But what people don't understand that a shut down windows are project altogether. Meaning that you have a project manager that is in charge of all the activities across that shutdown window.

Carlos So sometime you can't even actually patch during that shutdown window because they [00:17:00] might be using a piece of equipment for testing as some changes that they need to make in the control, in the control system and the program. And then you need to test it. And then if you wanna bring that down, just because even if it's in the shutdown window and they won't be able to test, then the pro project manager is gonna tell you fortunately.

Carlos And use it and you can't, you know, can't patch it. So that's one of the things like make sure that when you're bringing things in, they're up to date with the latest. The patches also understand, you know, what the behaviors from a device perspective, what are they, and to communicate. If there is a mis configuration, for instance, they try to connect to the internet.

Carlos I would try to connect. To the men, the vendor website, just to, to grab updates or things like that, like cameras for instance. That typically what, what happens. Just make sure that all that is actually taken care of before, before you start introducing your equipment into your plant.

Dino Yeah, makes

Carlos sense.

Dino Yeah.

Dino Yeah. And again, for the IT leadership out there, kind of [00:18:00] sum this up. One, just make sure you're aware of your company's capital plan. Just be aware of it, right. Timing of it and what they're. Focusing on as far as their capital spin. The other is becoming in tune with their maintenance windows, right?

Dino Because that's your opportunity to go in and they're gonna have a whole bunch of other activities going on, which is why they take those maintenance windows. They usually don't take 'em down just for a patch update. They've probably got several activities geared up for that downtime window. Just be aware of those because that's your opportunity to get in there and do some of this work.

Kevin When they get to the plant door where the products have to be implemented, they try to go through the IT door. And that's like going through the side door into a house. It's like, it's not the entrance that you're welcomed in. It's the one where I'm here now, what do I do? And they don't have that path to meeting the plant managers.

Kevin They don't have that path to knowing the problems because many of them show up and say, I'm here because it purchased this. I'm on a [00:19:00] contract to support this and we're now gonna put it into ot, and you see that all the time where it's, we just showed up, somebody paid us money and we're here to do something.

Kevin But that something is nebulous.

Dino Right. Well, and they're not demonstrating the value or the benefit right. To, to them. I've heard OT professionals, the plant floor peoples say cybersecurity just gets in the way of getting things done. Right? Because they've witnessed it, they've lived it. Right. It's been pain points for them, and I run into IT organizations all the time that has spent little to no time inside their manufacturing facilities, nor are they building relationships with the OEMs or sis that, that their manufacturing teams are hiring.

Dino And or buying their machines or it is really engaged or involved in that discussion. So is that something that's warranted? Do, do you think that will change this dynamic? Do you think that those folks are ever going to take that leap of faith and go, I'm gonna have to go spend an ex an an exorbitant amount of time [00:20:00] in my plans to figure out how those things run and tick.

Kevin They will, and I equate it in a microcosm way to restaurants now that are farm to table and they say, I go out and I meet the farmers. I know where things come from that are going to my end products that I serve to the public manufacturing, agriculture, transportation, e energy, you name any of the cities of critical infrastructures out there.

Kevin That has to be the approach. I need to know where these things come from. And they do it on the supply chain side with sourcing and other things where they drive down in detail. But when it comes to security, they're looking at safety, they're looking at uptime, they're looking at, nothing has happened, be before.

Kevin Why should it happen now? And they're also looking and saying, I need my. Vendors to be here when something happens and to not get in the way and try to help me run my business [00:21:00] better because I know my business, they're just here to sell me technology. And that's really the approach that technology, we've got it in the door.

Dino And do you think there's anything missing from the technology or we have the tech, it's just not being implemented in a manner that's gaining benefit for the organization.

Kevin Let's talk about that one word implemented. You go out every day, and of the products that a company has bought for their plant security focused help desk improvement for them, monitoring anything and everything over there, not just security.

Kevin How many of those have truly been implemented properly? How many of them are shelfware?

Dino I, I find a lot of shelfware. When we go out there, right? Or tools that are, that the client bought, somebody sold them, and nobody really taught 'em how to use or how to apply it to their specific use cases, right? Even across their plants, right?

Dino It's taking it into one plan. You're not necessarily gonna go through the same [00:22:00] scenario that you might at another plan. Or because there may not be, you might have one plant that's all Rockwell in a mix of other things, and you go to another plant that's primarily Siemens and a mix of other things and they, those plants might make the same stuff, but they do it differently.

Dino Right.

Kevin And that's exactly it. And I've been there and this is the have not have and have not world where when I was over a major manufacturing company for agricultural equipment, off road and on road or off highway, on highway. The big plant that produced the end product that would sell for millions of dollars.

Kevin They were flush with money with what they wanted to do. The little plant that was spinning out the widgets that went into that big product, it was patch the parking lot, fix the hole in the roof and deal with osha. If I don't do that, or it'd spend the money on cybersecurity, which who's gonna come up my door and say, you're gonna be fined and we're gonna be the boogeyman.

Kevin Nobody really at this juncture. So [00:23:00] you have to treat the entire thing properly. And in it you have a buying group that buys for an organization. One department doesn't say, well, I'm gonna go with this AV software 'cause I like it. It comes in and says, here is the standard and an ot. As you said, one plant's Rockwell, one plant's Yokogawa.

Kevin One plant is basically Siemens or any other vendor out there. And granted they are for different things on the production line, but when it comes down to certain areas, you have to have commonality.

Dino Yeah. To me, I look at cybersecurity tools because they're labeled as such, right? And cybersecurity is kind of a boil the ocean kind of term.

Dino But then I also look at that process integrity, because I come from a manufacturing background and we all know safety is job number one, but number two is reducing unplanned, unscheduled downtime, right? Because it's very costly. And so. If these tools could help improve your efficiencies within your manufacturing environment, [00:24:00] even if it can move it a few percentage points or several percentage points, that's big money.

Dino And so, and when you're talking about cost, because you know the budget's cut constraints. If you're saving, if you're losing millions of dollars on your plant floor annually due to unplanned, unscheduled downtime, where a tool might, this tool might save you a million or two plus million dollars a year.

Dino Is that worth pursuing versus fighting through buying a tool? Either one. 'cause we see it do this a lot just to check a box, but then they never share it with the OT teams who really are the recipients of that data that comes from that platform. Right. And at the same time, can you get the OT guys to go ID this?

Dino And these are the reasons why, right? I mean, that's one of the challenges that I see in my business is teaching the client facing engineering teams. On leveraging these tools to help with just project execution and startup and commissioning, right? Because they can't see, again, back to the [00:25:00] underwater camera piece.

Dino They don't know what's going on underneath the surface. There

Kevin is a term that is out in the IT world that I've used called left of Boom. Left of boom means if you're looking through a Mitre attack framework or you're going through an X-D-R-E-D-R type tool, you're building analytical data to determine if something is bad or good and you're trying to stop it before it happens.

Kevin If we take the security tools and say, we're doing more than security here, we're giving you visibility into your environment, and we're now pulling back data that is cyber physical. So that means giant industrial turbines or things like that. Natural gas equipment, for example, I've dealt with an entity that said if we have to have a flare burn off of methane natural type gas products out there to get out of the landfill.

Kevin If something's going on, [00:26:00] I need to know about it. Well, you can't know about it. If it's in the middle of nowhere with no connectivity, you now give it connectivity. You can have a left of boom moment where you can say, I know pressure's building up. I know heat's building up. I know somebody is there from a door sensor.

Kevin Now I can see what they're doing and now I can prevent it. That's left of boom thinking. It doesn't matter if the security component is there because most of the people in the plants, as you said, it's safety first. It's get Me in with any product. The vendor has basically dropped in as part of their package and it's keeping uptime.

Kevin So that is a left of boom moment is can you get analytical data, situational awareness outside of the facility to give you that advantage? Yeah.

Dino So, so who do you see as being the true leaders in this discussion, per se? I mean, as you mentioned, there's consultants, right? You got the technology vendors that are out there.

Dino I don't think the government [00:27:00] is driving this narrative to that extent. For the majority of manufacturers out there, our critical infrastructure organizations, who's really got their finger on the pulse. I mean, if you're a CEO or a CFO plant manager, who would you recommend that they need to go talk to to get better at this?

Dino To get smarter with this?

Kevin Yep. So the plant manager or the operational lead at a facility is the person that runs things. And you've met a bunch of these people, we know a bunch of 'em in common, and when you go and see 'em at a show, you can tell they're the guy that you know has their pulse on how the facility operates.

Kevin The problem is that their pulse is generally on the uptime production type things in the plant. It's not on the, how do I make it more secure because I'm fearful of something going wrong. And then you trickle down to the vendor that comes in, puts in an entire production line, an entire packaging or bottling [00:28:00] process, and the plant manager and the company go, you own that.

Kevin You're responsible for it. And it's really at that point, contractually driven, that if you have downtime, there's huge penalties. Look at the automotive industry. Tens of thousands, if not hundreds of thousands, dollars a minute or hour for when you're down.

Dino It's like $22,000 a minute. At least several years ago, that was the metric.

Dino And automotive, right?

Dino minute,

Kevin and now I hear people out there where it's 50, 60,000 for critical things, but a lot of people focus on manufacturing side. If you look at other areas that's in there, like the energy, like maritime, like even financial is part of the CISA critical sectors. If you look at the New York Stock Exchange now, those were built, those computers and everything are right there on the floor.

Kevin They're dedicated links because those transactions have to be in a timed capacity. And so all of these industries know what good looks like from a process [00:29:00] communication process flow, and how to make it work at maximum efficiency. What they fail at then is, okay, now how do we wrap security around this?

Kevin Because they view security as it's going to slow things down. That's true. It's going to cause delays. It's going to cause outages. And we've seen those sadly. And so really now it's not selling. You're going to be more secure. You're really selling to the plant manager. We promise we're not gonna break anything.

Dino To see. To me, that gets to the point of where you have to have, just like your operations team, your engineering team, your production teams who know how to run that equipment pretty optimally, right? Generally speaking, they do. Yep. But they're, but again, they, but they don't know what's going on from the data streams underneath, within that machine center.

LuRae - Intro Thanks for tuning in to the Industrial Cybersecurity Insider To stay up to date with our latest episodes, be sure to click the follow or subscribe button now, [00:30:00] and if you found this podcast helpful or have a topic you'd like us to discuss. Please leave us a review or let us know. Thanks again for listening.

LuRae - Intro See you next time.