Zero Trust in OT: Adapting IT's Playbook for Enhanced Security

[00:00:33] Dino: Hi, my name is Dino Boussalaki and I'm the Chief Technology Officer for VELTA Technology. My primary background is in operational technology and

[00:00:42] Craig: manufacturing. Hi, my name is Craig Duckworth. I am the President of VELTA Technology and welcome to today's episode. Today, we are going to talk about some of the higher visibility cyber attacks and that we've seen the last year or so and what ties them together and maybe about what separates them.

[00:01:00] Craig: And then at the end, we're going to look at a very recent one in the casino industry for gaming with MGM that was just in September of 2023. So let's start with the first one. It really goes back to the water treatment facility in Oldsmar, Florida, where someone got into the remote control access and began modifying some chemicals in the fresh water.

[00:01:23] Craig: As everyone knows, freshwater is critical to all of us. We take it advantage and a lot of times I think we take advantage of it. We don't recognize the importance of it. There's a limited amount of it and it's hard to say how truly important it is, but let's take a look at that one.

[00:01:40] Craig: Dino what's your thoughts on, what happened in Oldsmar and how we could help clients that are in a situation like that.

[00:01:46] Dino: Osborne was remote access, right? That was provided by a bad actor, right? Whether it was an inside job or an outside job. I don't even know if they really ever figured that out.

[00:01:55] Dino: But at the end of the day for them, that's a facility for the most part that can run independently of being connected to the outside world. It should right. And I believe that the safety systems that they had in place would not have allowed for. The water to be contaminated with large amounts of caustic like materials.

[00:02:12] Dino: for the world looking in from a media exposure perspective, I think it got blown out of proportion. One, yeah, you got remote access to the environment, but the reality is that the safety systems that these, those types of facilities have. Would not have allowed for that to happen.

[00:02:26] Dino: It would have been a manual process do that for them. It was more of a media type and how you handle incident response to get out in front of what's happening to you to say, hey, look, yeah, we were breached. But there's no way this would have happened. And here's why, right? In that particular case, in my view.

[00:02:43] Dino: And so that's part of, when you think about the five ICS critical controls that the SANS organization has published around incident response, and having that media and communications plan in place when these types of events happen. Moving it along, in the manufacturing world, can you operate without being connected to the enterprise or the internet?

[00:03:02] Dino: The answer is yes, maybe, who knows, right? Some organizations from a control system perspective might say yes, but I've worked in organizations where yeah, if you can operate your plant without being connected to the corporate network or to the internet, where is your central warehousing applications running, for example, right?

[00:03:21] Dino: And so maybe you're required to be connected to the corporate network in order to run your warehousing applications, which means I can't move product or inventory product. If I don't have to the corporate network, that can cause downtime, right? Meaning that got to shut down the connection because I can't afford to have any.

[00:03:40] Dino: seep into my plan, maybe that's a defensive measure that they take when they get hacked. I've seen that organizations get hacked, cut the communications off to the plan. So the infection can't go far. And why? At the same time, we see this a lot to manufacturers. Have connectivity, not just through their corporate network, but actually we've seen plant to plant connectivity,

[00:04:00] Dino: on plant to plant connectivity, where sometimes it can't see those types of connections between control systems goes back to the visibility, a manufacturer, you have the asset inventory and the visibility into your environment.

[00:04:15] Craig: What I think what it holds more of a part of it was, and you touched on a little bit the remote access.

[00:04:21] Craig: We heard so many clients say that couldn't happen to us because we don't use team viewer, which was the application for the remote access. That was, being utilized when the remote control took place. I think it's important for organizations to understand. It's not the application where they were using team viewer.

[00:04:41] Craig: They were using webex. They were using any number of remote control Vehicles. Yeah, it wasn't that particular one. It could have been any of them It was the procedures and the policies and the frameworks around that I believe Which allowed it to take place

[00:04:57] Dino: it goes to show that if you don't have remote access, secure access to your control system environment and using the technology that's built specifically for control systems, you're probably using the wrong thing, whether it be team viewer or some type of VPN or RDP pick your bomb guard, take your pick.

[00:05:16] Dino: of remote access tools that are used to get into that space. If you don't have tools that have certain attributes specifically for remote access in a control system environment, I'm going to tell you right now you got the wrong thing. And chances are those decisions were probably made by groups of people that aren't really doing the homework that they need to do to determine what are really good OT, cybersecurity, industrial control system, remote access tools,

[00:05:41] Craig: and we hear this a lot from a VPN, you come into the network and you have access to every single thing on the network versus something that allows you to get very granular and very specific down to a port or a protocol or something like that. this is reporting.

[00:05:57] Dino: Yeah, according to session auditing it.

[00:05:59] Dino: So you have a record of it, being able to control specifically which control systems you have access to, normally, if it's remote access, it's set up by it, you their understanding and tribal knowledge of operationally what's going on inside that plant, when vendor X gains access, it only has access to fillers versus vendor Y that might have access to.

[00:06:19] Dino: Mixing and blending vessels. IT typically doesn't know what it is that specific vendor is after. So how are they going to be able to determine, somebody that gives access to the plant is only working on the fillers, but yet I have access to utilities or I have access to warehousing or packaging,

[00:06:35] Craig: And that makes total sense.

[00:06:37] Craig: Let's shift gears a little bit and let's talk a little bit. About colonial pipeline. That one was something that really affected the us economy on the east coast. People equated it to the gas shortage back in the sixties and the seventies. People were afraid they couldn't get gasoline. We saw a lot of really poor decisions made where people were putting gasoline in the trunks of their car in plastic trash bags.

[00:07:02] Craig: And it was eating through it because they did. So the videos. Yeah, they didn't know they were going to get gasoline again. this is something that, as an economy we rely on fuel for our vehicles in our everyday lives. how would you say this is different from Oldsmar? lAck of

[00:07:20] Dino: due diligence on their part on Colonial Pipeline's part, because they didn't have visibility into their control system environment to determine how far and why the malware It persisted through the environment, right?

[00:07:29] Dino: They didn't know.

[00:07:31] Craig: Because we hear all the time, Hey, that was an IT attack. That was social engineering. That was not even on the OT side. So why are you even talking about it? Help us understand that.

[00:07:42] Dino: didn't know whether it was down there or not. They're not watching that.

[00:07:45] Dino: They're not continuously monitoring their control systems, looking for malware signatures. They don't have vulnerability management practices in place to determine the weaknesses of the ICS environments that they have out there. They don't necessarily have the proper remote access controls. Tools being used into that environment.

[00:08:01] Dino: So they weren't able to definitively sit there and go, yeah, we don't have any malware down here. And our control systems running these pipelines using this fuel, from the Gulf to main moving 5, 500 miles of pipe. Moving fuel. They didn't know they didn't have any visibility into whether they were safe or not.

[00:08:17] Dino: So they had to shut down in the precaution of being safe.

[00:08:21] Craig: back to the number one tenant in the OT space, which is safety. Everybody, that's the number one thing on every manufacturer, whether they're pumping fuel in this situation is safety, and I think I would agree with you.

[00:08:34] Craig: The engineers on the operational side could not definitively say. They can safely move this fuel being invisible to what, is happening.

[00:08:45] Dino: They didn't have the right tools and they probably wrongfully believe that it had visibility and the right tools in place for the control system environments, which they don't, don't have them. out of precaution, had to stop production operations because they couldn't determine whether the malware was down there in their control system environment, because even though it was a Windows specific attack, you've also got Windows running your control systems, I hear too many defensive tactics coming from mainly OT people who sit there and go they're not attacking my PLCs.

[00:09:17] Dino: My answer to that is yet. Do you have vulnerabilities on your PLCs? Yes, you do. Do you know what they are? No, you don't. What's running those PLCs? Are your Windows operating systems pushing the data to those things? Yes, they are. What are the vulnerabilities associated with those? I don't know, and here we go and it's not down there trying to figure that stuff out because it's not their world.

[00:09:37] Dino: They don't own those assets at the end of the day, right? They don't understand the process activity that's going on inside those environments, or even some of the technologies. These guys pick to run their pipeline to run their plan, How are you going to move this from a partisan.

[00:09:52] Dino: Alignment and arrangement to a bipartisan relationship between it and.

[00:09:56] Craig: Yeah. And again that's been the struggle for the last, let's call it six or eight, 10 years as I. T. and O. T. are beginning this transformation or this collision, as we like to call it, and trying to understand each other's technologies and the abilities and try and work together as one cohesive unit to better understand that environment.

[00:10:15] Dino: if you're the smartest guy in the room and the OT guys probably think they are, and the IT guys think that they are, and if you truly are the smartest guy in the room, you probably need to go to a different room. Make some new friends. And that's what these two need to do. And that's the challenge that's going on in the relationship between IT and OT, because they both think they're smarter than the other guy.

[00:10:35] Dino: And what they need to do is they need to work together to become aligned, to become collaborative, and learn how it governance and policies and procedures and technologies and solutions can help the OT environment. At the same time, it needs to learn how those facilities operate.

[00:10:52] Dino: And the control systems they have, the technology that they use, who their ecosystem of partners are, because they also have different ecosystems as far as the suppliers of the technologies that they use. OT come they're going to go down a different path to go search for this stuff, just like IT goes down a different path to search for the stuff that they use in their business.

[00:11:11] Craig: And the most successful projects that we've done, we've been involved with. We're in a collaborative effort between I. T. And O. T. Where the asset owners, the O. T. Teams were driving the narrative and really being the project quarterback. If you want to look at it that way, and I. T. Was there to help support them where they might not have been as familiar with the technologies.

[00:11:33] Craig: They didn't understand some of the policies and the procedures and the different frameworks because that's not what they're tasked with. But collaboratively together. They were both doing the same project I think that's the message on this one Let's move over and talk about a little bit about jbs this was another situation where It really affected the us from a meat packing and a production You know you look at again you go back to Food shortages on the shelves of every major grocery store and retailer across the u.

[00:12:07] Craig: s And it put the public in a panic. They felt that they were not going to get food. I'm sure there was a huge cost to the manufacturer of spoilage of loss. But, when the public people feel threatened or feel that there's a concern that they're not going to be any food and they're hoarding food, and then it really exaggerates the effects.

[00:12:31] Craig: what's your thought on this one?

[00:12:33] Dino: not knowing the specifics of JVS per se, what I would probably say is that they are probably not different than a lot of manufacturers who've co mingled their industrial environment with their IT environment. To the extent where you can't differentiate one from the other.

[00:12:47] Dino: if you have an attack that, that's, running through your IT systems pretty rapidly Your OT systems are going to get caught up in it, right? Because you don't have separation. You don't have a really good logical separation, maybe even a physical separation, but more logical than physical, because everybody's connected today.

[00:13:07] Dino: And determining, again, how do I know that my control systems are safe and can run without being impacted by everything that might be happening to the IT stuff? And we see that a lot. We see these very large, flat layer two networks that a lot of these organizations have. And we go out and do some of our assessments and the works that we do, we'll literally find several 100 devices.

[00:13:29] Dino: Running around on the same network, but the control systems on without much of a separation of even a firewall between them. Again, not knowing what JBS did, my guess is that the impact of one immediately impacted

[00:13:42] Craig: the other. Yeah, and I would say that, while we always hear that the air gap, we're air gap, nothing's going to get in our environment, I think the vision of the old fashioned.

[00:13:53] Craig: Meat butcher shop where someone's back there with a bandsaw and a knife and they're cutting the stuff up and they're wrapping the meat together and they're putting it on it. That's not the situation that Today's environment is, I can, I've not been in a meat producing facility, but I can imagine there are large saws and all kinds of connected devices manipulating these carcasses into different positions to cut them precisely based on what they need to do.

[00:14:21] Craig: it's all about the productivity and the efficiency of those devices. I would say, confidently, there is no way that their environments are air gapped in what we traditionally would hear about, Ah we're good, nothing comes in and outside of these four walls.

[00:14:39] Dino: Yeah, that's a myth, there's various ways that these things are connected. Again, if you're not continuously monitoring it, how would you know? Assets coming and going, connectivity, coming and going, new pipes being added. Somebody boots a firewall, does an update on a firewall, all of a sudden opens up a new port on the firewall and now you have a new access point or a protocol that's become available, an unsecured protocol.

[00:15:02] Dino: Any junk boxes, dual nick devices. There's a whole bevy of what wireless cellular, USB ports. There's a whole bunch of things out there that demonstrate that these plants. They leak like a sib, it sounds like

[00:15:14] Craig: we're talking about Swiss cheese and not JBS.

[00:15:17] Dino: it depends on who you're talking to, right?

[00:15:18] Dino: Because if you're talking to the local plant folks, they're gonna tell you we're air gap. Then you send somebody out there with the right tools and experience the knowledge, turn around and go no, you're not. You've heard me say this a bunch of times. If I walk into a plant. 8 and 10 will let me take my laptop that we're doing this podcast on and connect it to their network to do whatever work is required, right?

[00:15:36] Dino: They don't scan it. They don't look at it. They don't know if I got cellular or Wi Fi on it connected to it. They have no idea. I've, I have found wireless dongles hanging off of PLCs and there are people out there. They'll say that's not our plan. And again, until you send the right people out there, going out there and looking for this stuff, and even IT organizations struggle with that, even though they have the expertise, they don't necessarily know what to look at when those panels are opened up. They got all that stuff in there, it's got drives in it, PLCs in it, maybe a panel view or HMIs, you got all switches in there.

[00:16:11] Dino: They got all this stuff in this panel and they've got hundreds of these things on their plan floor and IT doesn't even know what's in there. They don't know. And so how do you get to change that dynamic, right? Which is really what needs to change in order

[00:16:23] Craig: to help solve this problem. Part of that dynamic is changing though.

[00:16:26] Craig: Do you know we're seeing the SEC step in and forcing publicly traded organizations to disclose any material effect, any cyber attack, anything. So we're seeing those come into play today. Real recently in September and August, September Was clorox disclosed that they had a half a billion over 550 million dollars worth of loss or loss that they are experiencing.

[00:16:52] Craig: They're still struggling. I believe today to get all of their systems back up and running at full capacity people can't get certain things on the shelf I'm serious.

[00:17:03] Dino: My wife, right? She was trying to get her kitty litter from Costco and couldn't get it. And couldn't figure out why they didn't have it.

[00:17:12] Dino: And you're talking about an avid shopper.

[00:17:14] Craig: Right.

[00:17:15] Dino: And she learned not listening to me or watching the news because she doesn't pay attention to this stuff. She heard it from Costco that the reason why she can't get the kitty litter that she wants was because of the problems that Clorox was having to be able to produce goods.

[00:17:27] Dino: and that came straight from costco to the

[00:17:29] Craig: consumer and it won't take long for costco to realize there's another producer that can make kitty litter that has a Better hygiene cyber hygiene and are going to continue to do that Versus someone that is not taking this stuff seriously And are allowing it to affect their brand because costco is the one that's getting the bad rap not cool.

[00:17:53] Dino: There's a tire manufacturer that did the same thing. Quit selling that tire manufacturer's tires in their big box stores. Just like what's happened with the kid dealer. This all rolls back to the senior leadership in these organizations not paying attention. And whoever it is that they're talking to, like I said earlier, if they think they're the smartest people in the room, they need to make some new friends.

[00:18:12] Dino: Because whoever their little team of people are that they're working with aren't cutting it. And the sooner they realize that and recognize that, and we've seen that with some of these board of directors because they work on these other work. They've witnessed these problems and are basically saying, we got to get through this and pass this and the pressure coming from the B.

[00:18:29] Dino: O. D. telling that senior leadership, you have to start looking for O. T. Cyber security solutions. And if you're resting on it to figure that out for you, you're not going to get there and they can't do it by themselves. It's impossible for them

[00:18:42] Craig: to do that. They're going to start forcing like you just said, that cyber experience on the board, they're going to start forcing CEOs and other leadership members to be personally liable where someone it's unfortunate, someone's going to go to jail and they're going to make an example out of several people.

[00:19:02] Craig: think the fines, I look at Clorox, a half a billion dollars. They're absorbing it. It is what it is, organizations are still paying ransomware and they're relying on insurance on the back side to cover that cost as a scop gap When someone goes to jail a fine's that's not gonna fix it if you have a board of directors And let's say that there's 10, whatever that number is, and five or six of them go to jail for terms and the CEO goes to jail, people are going to wake up because they don't want to be the next one to do it.

[00:19:33] Craig: Yeah

[00:19:34] Dino: to me, it's like safety, right? Are you going to treat it like safety, to that extent, or EPA reporting, or you would go to jail if you weren't reporting to the EPA in a timely manner that whatever hazardous materials you were burning off into the atmosphere. Beer storing or moving or whatever.

[00:19:50] Dino: if you weren't following specific guidelines or policies, you're right. You would go to jail. The question is, are you going to put the cyber side of this equation, which has the potential to cause dismemberment, death, hazardous, environmental activities, spills but at the

[00:20:05] Craig: same level, absolutely has to be at the same level.

[00:20:09] Craig: Let's switch gears and we have a little bit of time left. I want to talk about MGM that's your world man the casino industry and let's gamble on something real quick. MGM reported material event of a hundred plus million dollars Where their systems were down they struggled for about 10 days to get the systems back up and running And it cost them over a hundred million dollars The second largest or one of the equally as large organizations caesars Came out and stated that they paid the ransomware and they never went down.

[00:20:44] Craig: So mgm said we're not going to pay it They had issues that ranged from The electronic locks on all of their hotel rooms across their properties to checking in and out and into the rooms to machines not working to all kinds of various Database let's call it data theft where they took personal identifiable information from the properties from the player database and they disclose that and they sold that or whatever they did with it.

[00:21:15] Craig: I know that's not industrial But in kind of way it is, you look at a property that's making money whether that's clorox. That's mgm That's jbs And the bad actors are figuring out how to hurt them in the production where it hurts, what's your thought?

[00:21:33] Dino: I think it gets down to risk, I think if you make money hand over fist, like maybe a casino does, maybe like Clorox was doing is that for them, if I'm making money, especially coming out of COVID, remember the supply chain was so fragile, right? and these plants are running 24, 7, 365, again, printing money. They'll probably accept the risk of losing some downtime, and gambling, they'll throw the, they'll roll the dice.

[00:21:58] Dino: And say, after we get hit, we lose 20 million or 100 million. It depends on the number, right? What is your threshold for pain, right? If I lose 20 or 25 million dollars every few years without having to go and spend several millions of dollars over those few years to do a better job of securing and protecting my environment and still get hit.

[00:22:14] Dino: Got to remember all of these organizations got cyber security professionals on staff. They got cyber security technologies in place. I've been doing it for years. I'm pretty sure everybody that we've been talking about that's been hit has got firewalls. Right when you look at it from that perspective you're not doing enough.

[00:22:31] Dino: Are you going to take the risk and assume that if I get hit periodically, and I have to pay, I'll do that versus doing my due diligence to get better to harden my security posture.

[00:22:41] Craig: in GM,

[00:22:41] Dino: aren't they getting in cybersecurity

[00:22:43] Craig: insurance, but part of their bill, so cybersecurity insurance, they had a 200 million policy that's supposed to be paying that.

[00:22:50] Craig: whether they do or they don't is yet to be seen, but I don't see the insurance company just offering up a check. Or a hundred plus million dollars to cover it. may say, you know what, here you go, have this money. And it may take years and more millions in lawsuits to get that money.

[00:23:08] Craig: And they may only end up with a very small portion of it. The policy is

[00:23:11] Dino: going to go up way

[00:23:12] Craig: up and there's the fact that there are now. Either eight or nine federally recognized lawsuits, class action suits between MGM and Caesars for failing to properly protect the data in those organizations.

[00:23:28] Craig: yoU mentioned a while ago about the fine, and it brought something to my mind in the U. S. I would say you're probably correct. The fines are not appropriate to the penalties. In the europe they have gdpr The fines for gdpr can be one times annual revenue from a prior year And there have been situations where organizations been fined billions with a b billions of dollars That is a significant difference than 20 million or 50 million if you've got a billion dollar fine People lose their jobs.

[00:24:07] Craig: Organizations go bankrupt. That is material. actually

[00:24:11] Dino: in Europe, cybersecurity practices are further advanced than they are over here in the U. S. and North America, for sure. No doubt about that. And you and I both know there are manufacturers out there that can't even get cybersecurity insurance.

[00:24:22] Dino: True. Can't even get it. is there a lot to do? Yes, there is. Are a lot of them just rolling the dice? Yeah, they are. And then there's a lot of them that are getting hit that, have to hear about private industry that doesn't have to report anything, right?

[00:24:33] Craig: And that's part of the problem is everything is so cloak and dagger.

[00:24:36] Craig: If it was. More openly communicated. I think that we as a industry could better protect everyone because you don't have to share trade secrets. But if it's anonymized, it's put in a database of some sort. And we're saying here's the method this attack took. here's what happened and let organizations say, maybe I better start focusing on that.

[00:25:00] Craig: Or I hadn't even thought of that. And then that would help. I believe

[00:25:05] Dino: they have to invest in the resources and the talent too, right? Because is a shortage of the talent for this, right? In my view, especially when you start crossing that Rubicon into that OT environment, when you start talking about an industrial space.

[00:25:19] Dino: And the level of sophistication that's required of knowledge of that environment, the tools and the solutions that apply that space, we don't have a lot of resources out there. heard that from Rockwell, their HR executive came out describing the situation late summer and we witnessed it ourselves because, we know we have to grow these resources, you're not, the schools aren't pushing this out. They don't teach cybersecurity, but they're not teaching it from an industrial perspective,

[00:25:46] Craig: They're engineering

[00:25:47] Dino: schools and the schools that are providing technology degrees need to start adding industrial OT to the environment.

[00:25:56] Craig: Perfect. We're at about the half hour. So is there any parting wisdom that, that you can think as, you look back over the last two years and you think colonial, you think Oldsmar, you think JVS, you think tire manufacturers, you think Clorox, even the casino industry. This is not one vertical or another.

[00:26:16] Craig: This is. Industry wide and clearly it's not just manufacturing It's where the money's being made on a slot floor in the plant, it's everywhere. How can organizations take steps to begin protecting themselves? Or what can they start doing?

[00:26:35] Dino: You got to start putting a plan together, right?

[00:26:37] Dino: I subscribe to that five ICS critical control, Incident response, continuous monitoring. Vulnerability management, remote access, defensible architectures, Defensible architectures for your control systems, not just for I. T. who's worried about their stuff, You got to move it into that control system arena, determine how am I going to build a defensible architecture around that?

[00:27:00] Dino: And still not be able to impede that control system's ability to make goods, in a timely, quality available manner, Safe manner. too many times we've seen IT very disruptive in that OT environment based on some of the things that they do. And trying to administer and manage their tool sets that bleed over to that control system environment and knock it down, knock it offline, shut

[00:27:23] Craig: it down.

[00:27:24] Craig: We've heard often that. Sometimes when I. T. gets involved, the project gets drug out and gets more expensive and they just, O. T. doesn't want to deal with them because of past blood in the, bad blood in the past. They do.

[00:27:36] Dino: they drive up the project costs and they delay the

[00:27:38] Craig: startup window.

[00:27:40] Craig: I would say that they don't know what they don't know. some good old fashioned information sharing like we talked about in the beginning where successful projects or a collaboration between both where it truly wants to understand and wants to help drive this and help this process be successful.

[00:27:59] Craig: O. T. Wants to do the same thing, and together they try and learn the best they can to advance the initiative for the organization.

[00:28:08] Craig: Perfect. I appreciate it. been a great conversation. I would say that organizations have to make sure they're practicing due diligence. And ask themselves, are they providing the same amount of due diligence and care on the operational and industrial side as they are the enterprise side?

[00:28:26] Craig: And if they're not, these tools are available. They are out there. Why aren't you? Because you're not doing the, you're not doing yourself a service if you're not.

[00:28:37] Dino: Yeah well, IT needs to go out there and get their hands dirty. And OT needs to clean up a little bit.

[00:28:43] Craig: Yeah, both agree. Yes, totally agree.

[00:28:45] Craig: See you on the flip side

[00:28:46] Craig: Perfect. Thanks, buddy