Proactive vs. Reactive Cybersecurity: A Strategic Communications Perspective

[00:00:00] Dino: And so the feds there hunting them down, trying to remove what they're doing. And then after they do what they do, then they'll come back and talk to the client again, business owner and say, Oh, by the way, we broke into your building and, sorry, we destroyed a bunch of shop. We shut down your plant floor.

[00:00:12] Dino: Didn't mean to, we had to go get these bad guys.

[00:00:14] Jim: Yeah. I'm with you. That's just generally, my impression is because, the government has changed tactics and are those tactics that they're using. Add more risk to the OT environment if you're not managing it appropriately?

[00:00:25] Dino: I'd say they don't even realize that they're impact. They don't company's business to produce the goods that

[00:00:28] Jim: they make. They're not sure.

[00:00:29] Dino: They don't care.

[00:00:30] Jim: that very first lesson from the field, which was, you know, the scanning disruption to prevent disruption.

[00:00:36] Jim: So How does it go? The cure is just as bad as the disease.

[00:01:13] Dino: Hi, I'm Dino Busalacchi. I'm the Chief Technology Officer for VELTA Technology. Hi, I'm Jim Cook,

[00:01:18] Jim: the COO of VELTA Technology, and what are we talking about on this

[00:01:24] Dino: one, Dino? We're going to talk about lessons learned in the field and installing OT cybersecurity technologies and platforms.

[00:01:32] Jim: this is fun one, jotted some things down that, you get your head scratching the first time I saw and now it's just common, right? And if you're not looking for it. you don't realize these things are out there. I'm going to start with 1 and then we'll see where we go with this.

[00:01:45] Jim: I call it scanning, to stop disruption. Right? And I always like to come from the IT side. It's very common on the IT side to do the scanning, but if you're not controlling your scanning or don't understand what you're scanning, you cause disruption.

[00:01:59] Jim: But why are you doing. The scanning to prevent disruption, right? we've seen that out there a lot. I know you've seen it where it's caused some major disruption. but, we've talked to, guys that are running those machines. And when does it happen?

[00:02:11] Jim: Are you at least notified? And since I'm notified. When the machine stopped running, right?

[00:02:16] Dino: it's it applying their best practice to try to discover and learn about all the assets that are within their view of the environment that they're responsible for. I've just, unfortunately they captured the OT stuff that's been connected up over the last, several years due to digital transformation industry for the Ford auto, remote access requirements, especially with COVID.

[00:02:37] Dino: To an IT person, it's normal procedure for that, right? Pen testing, that type of stuff where they're scanning, trying to discover assets. and the challenge is when you hit an OT network and you do that, it can cause, PLCs, for example, to com fault, or go safe because if they can't communicate within a timely manner to a machine that's got very, tight tolerances on, communications, they will fault and they will go safe.

[00:02:58] Dino: And so that's 1 issue that you'll find. and so for it, even those that have learned their lesson and have created exclusionary list that say, these are the networks to stay away from. how do you keep it pristine? Because over time, things change, people change, technologies change, things get updated, new networks come out online and all of a sudden you find it happening again.

[00:03:17] Dino: And so the intent is, these ot IDS intrusion detection system platforms as advertised, not market architecture, are designed to be passive. There's a reason why they're passive.

[00:03:28] Jim: and even interesting you bring that up, that's something with the OT IDS systems, they count scanning as an issue, right?

[00:03:34] Jim: Yeah. Scanning that commonly happens and it's for the very

[00:03:37] Dino: reason, right? the vendors will tell you that, right? Rockwell will tell you that. Siemens will tell you that. Cisco will tell you that. that knows anything about industrial control system infrastructure will tell you do not scan.

[00:03:48] Dino: Actively into these OT environments, just blindly, or scanning any of it, right? You don't do it. and and some have learned painfully, they've learned to quit doing it. And we've seen exclusionary lists. we've seen a client that took us a while to get security team to sit down at the table because we recognize they were scanning the control system network.

[00:04:05] Dino: The plant and the plant was complaining about it. And everybody was ignoring them. And the networking team that was involved in the project was thinking, then it's just normal by the time the security people showed up and sat down at the table, they were showed what they were doing.

[00:04:17] Dino: It's like that stuff's supposed to be on the exclusionary list. it was at one time. the question then becomes is, how do you keep that from happening? We have the technology and the tools to do it today. It's just that when you get to the OT environment, as you go through what we've witnessed is not everybody has a good understanding of how this stuff's put together.

[00:04:34] Jim: want to take this and then go to the next step. so the guys. they know this goes on, right? And they just try and work around the IT guy. So this is another lesson from the field that you get machines connected on their own path, right? They find a way around the IT and whether it's from disruption or whether it's because it's difficult to work with or it's just easier.

[00:04:55] Jim: We've seen this where those machines are plugged in to, say, a vending network,

[00:05:00] Dino: right? Yeah, take Fastenal machines, they're out there providing PPE, for example, right? And it's down inside the plant, and it's very easy to connect it within the confines of that plant network that's right next to the networks that the control systems are that are probably also connected to the same network.

[00:05:15] Dino: And that's where you'll find those things that have remote access up and out. You'll find cellular modems in those things that have access out, but they're still connected to the plant network or the customer network per se. and so it's not uncommon, we find it all the time. It's so easy.

[00:05:28] Dino: somebody's looking for a jack to plug something in an ethernet jack, and then the next thing you know, somebody's doing exactly that. When

[00:05:34] Jim: I plug it into this, people call me and they don't help me fix my problem. When I plug it into this, they may open up other problems, but at least they get their

[00:05:41] Dino: machine back running and then you listen to some it guys sit there and go, we have network access control, right?

[00:05:46] Dino: We got knack. you plug in and if you're not recognized or you don't have the right credentials, you don't get on and it's like on whose network. Your network, what about the plant

[00:05:53] Jim: network

[00:05:54] Dino: but for them, we're going towards snack.

[00:05:56] Dino: We're looking for segmentation. It's yeah, you're not there. And even if you are there, moving that direction, how do you stay there? Because these are not static environments. They're very dynamic. we've been working with a client, the plant manager told us he's got people from all over the world accessing his plant floor every single day, right?

[00:06:13] Dino: Think about that for a second and he doesn't know what they're doing. And it doesn't know what they're doing, but they're in there every day. Yeah, They show up on site. They remote in. and so when you go down and so what's is that person doing? So when you get in there and there's unsecure protocols, there's open ports, there's host scans, you got, the login failures and monitoring that stuff, default passwords, unencrypted passwords, we find all of these insights demonstrating the vulnerabilities and weaknesses in this environment.

[00:06:38] Dino: And who's doing anything about it? If you don't know it exists, how are you going to chase it down and try to solve the problem to fix it?

[00:06:45] Jim: an interesting point from there is. You're not going to find it till you get out there on the floor, right? Because those plant managers aren't broadcasting that back through the organization, are they?

[00:06:53] Dino: no, they're not

[00:06:54] Jim: going to say, hey, just so you know, our daily reminder we had, who knows who come into our network last night and work on our machines. We don't know. That's something they don't share. But you got to find that out when you're on the floor, right? yeah, it's

[00:07:06] Dino: just like the conversation we had recently about the Apple devices, right?

[00:07:10] Dino: Now, somebody would look, when you see an Apple device, If it's connected to the BACnet network, which is the environmentals of the building system or some testing system that you have because of the type of business you're in, and that's connected to the network, then all of a sudden, yeah, you do have a Mac sitting out there.

[00:07:23] Dino: It's not being monitored. there's no EDR and there's nothing, but it's connected to your network within that environment, and somebody would sit there and go, Oh, Those tablets are not, OT assets. Okay, now From a risk perspective, who owns that?

[00:07:36] Dino: this is where you get the deer in the headlight look. IT didn't buy that and we don't use that asset. And OT is using that asset and it's valuable to what they're doing. But they're not the security team. That's not their responsibility. Nobody's telling them you've got to do a better job of being cyber security experts and professionals in your environments.

[00:07:52] Dino: Here's some money, here's some tools, so here's some expertise, skills. any of that stuff,

[00:07:56] Jim: Nah, they're just worried about getting it out the door and making sure that machine's running.

[00:08:00] Dino: they've been taught, OT's been taught over the years, because I've witnessed this, I've lived through this, what the controls people will do is they'll just put multiple network interfaces on their equipment.

[00:08:08] Dino: That's how I keep IT out of my hair, right? I'll put two NICs in my PLC, I'll put two NICs in my server, I'll put two NICs in my engineering workstation, and now I have a device that straddles both worlds. and I. T. is not responsible or accountable for the other side of that house.

[00:08:24] Dino: But yet this machine is connected to their network. So now it's a conduit either way up or down, they choose to ignore it.

[00:08:30] Jim: and, that kind of leads to this other point that I wanted to touch on, which was, I mentioned before, they're not advertising certain things, right?

[00:08:38] Jim: Because sometimes they're afraid of it going away, right? and we see this all the time. We'll be walking on the floor and we'll. Is that machine connected? right? You open it up and so what's that? And yeah,It's connected, but can I leave it there? And We're in the business of identifying these things and offering up mitigations on how to handle that. But again, they're not going to advertise. They're not going to communicate it because. Okay. They've been successful working around the I. T. controls and the I. T. network and the I. T. support. and they've been successful doing it.

[00:09:09] Jim: They want to keep doing that. they don't want that other headache that you said where everybody sits around. who is responsible?

[00:09:14] Dino: yeah, ultimately, you think it'd be the senior leadership, who I think that have been misinformed by their teams by assuming that the message coming to them is that we're air gap.

[00:09:24] Dino: Those things aren't connected to my business system network. Those things are separate, right? they must be separate because obviously there's different groups managing them. So they have to be separate. somebody needs to inform them that either they're being lied to. Or that's just the ignorance within the organization doesn't understand what they've done to themselves, right?

[00:09:41] Dino: Because everywhere we go. There is nobody immune from the things that we find that when I listen to 1 thing, I just take with a grain of salt and say, yeah, okay, I'm taking notes. I'm listening to you now. Let's go see what's really going on. Versus reality.

[00:09:53] Jim: man you're just leading, and by the way, Dino hasn't seen this list and I'm just checking off things.

[00:09:59] Jim: should have made it into a quiz show. and we hear this all the time, right? this is our policy. This is our policy. This is our policy. But until you get on the floor and find it with the real practices, does policy really matter in that case? Because that's what we take practices over policy. That's what we're trying to identify.

[00:10:14] Dino: some of the projects we get involved with, even though the scope, They want it to be narrow, right? Because as soon as you start digging, you're standing in a hole and you're digging, what are you supposed to do?

[00:10:23] Dino: I quit digging, right? And the problem is that they've got this problem. It's you've started this digging process and now we're finding all these vulnerabilities and all you want to do is shift back to the focus of to stick to the scope. I don't need. For all these other, risk activities to float up and get beyond my control to be able to manage this, right?

[00:10:42] Dino: It's an interesting dynamic when you start going into these engagements and people start realizing what they've been communicating is not. what the organization really fully understands about that environment per se, are we safe? again, we can talk about this.

[00:10:55] Dino: This is another podcast of what are the feds been doing chasing these terrorists? These hackers have been breaking in to organizations, right? It's going on, you have the feds out there. Because businesses don't know what's going on from China in their own backyard here in the States, for example.

[00:11:11] Dino: And so the feds there hunting them down, trying to remove what they're doing. And then after they do what they do, then they'll come back and talk to the client again, business owner and say, Oh, by the way, we broke into your building and, sorry, we destroyed a bunch of shop. We shut down your plant floor.

[00:11:23] Dino: Didn't mean to, we had to go get these bad guys.

[00:11:26] Jim: Yeah. I'm with you. That's just generally, my impression is because, the government has changed tactics and are those tactics that they're using. Add more risk to the OT environment if you're not managing it appropriately?

[00:11:36] Dino: I'd say they don't even realize that they're impact. They don't company's business to produce the goods that

[00:11:40] Jim: they make. They're not sure.

[00:11:41] Dino: They don't care.

[00:11:42] Jim: that very first lesson from the field, which was, you know, the scanning disruption to prevent disruption.

[00:11:47] Jim: So How does it go? The cure is just as bad as the disease. Something like that. Anyway,

[00:11:52] Dino: Yeah, we'll save that for another podcast because we've got some good stuff on that.

[00:11:55] Jim: another thing that we always hear were mentioning outside access, but we'll go into places.

[00:11:59] Jim: Oh,no. And there's no outside access. And then you got to answer the question. you who fixes that machine? well, except for this. And think in addition to that, I'm saying the majority of the machines I'm thinking back, at least in the past year, the panels we've opened up, they all have remote.

[00:12:14] Jim: Access built into the machine, right? And if you told an I. T. security person, you got six different methods of remote access and who knows who has those, I. D. s and like you were mentioned before, the plant managers know people are coming in, right? But do the I. T. security guys don't know, but

[00:12:32] Dino: So what's his job at that point when something like that happens? what power do they really have when that somebody over India has remote access to your plant floor unfettered, right? you have no idea what they're doing once they're in there. Oh, no, by the way, this machine is actually connected to the enterprise network, right?

[00:12:48] Dino: Of the organization, right? It's connected. So what's your job? at the end of the day, what are you supposed to do? this violates policy, every bit of it. the risk is ridiculous. what's more important? You're going out there and ripping that remote connection out of there or let the plant let that piece of equipment shut down and which might ultimately shut the plant down.

[00:13:04] Dino: And now you're losing hundreds of thousands of dollars or millions of dollars a day or whatever it is. what is your role? when it comes to that? We had a CIO, what did they say? If there's a machine connected to the internet?

[00:13:13] Jim: Why should I care?

[00:13:14] Jim: the thing is their scope you mentioned before the scope is okay. I'm just worried about stuff on my network. shouldn't this gets to that overlapping security resilience and operational resilience, right? and, okay, your scope should be the business, right?

[00:13:28] Jim: Not just what's connected to your network. Yes, those are vectors, but. If your scope and you're providing security services for the business, that's why you should care, right?

[00:13:37] Dino: because somebody has the ability, 15 year old kid can send, 500 camera video streams to your PLC that's sitting on the internet and shut that piece of equipment down.

[00:13:46] Dino: It's sitting at a key critical spot within your plant. If that machine goes down, your plant starts slowing and then it stops. that response was given that CIO didn't have a response to that, it wasn't like they were in a cop and say, oh, yeah, you're right.

[00:13:59] Dino: That's a bad idea. They said nothing.

[00:14:00] Jim: that's the chasm. have to cross at some point to go, what is my role? Just my network. That what I can see and what I got

[00:14:08] Dino: right about security really, what's your job when it comes to OT?

[00:14:11] Dino: If you find bad stuff, what are you going to do about it? The plant's going to tell you if I don't have this plant shuts down. Who trumps who that stage?

[00:14:18] Jim: we know who wins every time, right? Yeah.

[00:14:21] Dino: then it gets back to the point, then why isn't OT leading this?

[00:14:25] Jim: But yeah, we have to beat on OT a little bit too, right? we can't let

[00:14:28] Dino: them off the hook. Yeah. trust me. I say it all the time. They need to grow up and be big boys and girls and mature cybersecurity hygiene looks like in their plant and start caring about it. start having some really good conversations with their peers on the I.

[00:14:41] Dino: T. side who can give them governments who can give them policy can talk technology, work together on it instead of reverting back to your own camps because it's too difficult of a challenge for

[00:14:51] Jim: you to figure out. Yeah. back

[00:14:53] Dino: to the whole federal government breaking into your environment and

[00:14:56] Jim: doing what they're doing.

[00:14:57] Jim: Yeah, definitely, they need to. What? work together and recognize it. I think we've said this before. awareness and having those discussions together on how to move it forward because can be a difficult discussion and it can be difficult to get.

[00:15:10] Jim: Your proper controls in place or mitigating controls, whatever that might be, and why we always say you've got to be moving forward to think of it as a continuous improvement. they need to get together and keep moving it forward and keep getting it better. you can't just assume that.

[00:15:24] Jim: we put this in place and that should fix it. just a simple fix from the it side could take months on the ot side. whether it's. Response or mitigation or remediation, or, everybody's hatch management, right? It's not that easy. so takes a little bit of different skill set and

[00:15:39] Dino: takes a different perspective, right?

[00:15:41] Dino: Yeah, it has to be part of the culture, right? Yeah, and because people come and go. think how many people we've been working with just to go over the past year. Facilities guy's been on the job six weeks, right? he knows this much about that environment that he's working in when it comes to this conversation.

[00:15:55] Dino: Same thing on the IT. We see the same thing on the IT. these security people move around, the average tenure of a CISO? And I'm guessing for other IT security people, it's less than two years. It's 18 months, maybe 12 months. These folks are floating around. in order to be successful at this, You have to start building some type of governance and structure and make a part of the culture because you're going to have people coming and going.

[00:16:15] Dino: and you don't want these initiatives just to die because the person that's been working on the past 6 months moved on and up and out to something else. And especially the executives,they're, people that have been tasked with being, in leadership roles,they bounce around within the organization.

[00:16:28] Dino: Right? And so if you don't put something in place to make it sustained at the time of the resources tenure in the job, then you're just spinning your wheels.

[00:16:36] Jim: it has to be a business priority at that point, right? Yep. I think even getting it there, it's gonna take an individual or individuals talking to each other to go.

[00:16:45] Jim: This should be a business priority and please, who's ever the funding agent within that organization? You have to be able to, communicate what those risks are, which, some of the things that we talked about here, from are very real situations and probably half of them are going on in many different 1 site.

[00:17:01] Jim: And those are very real things that you could go back to and say, Hey, Dean and Jim, some of those lessons from the field, we found 5 of those things and we didn't really have to look hard. those are very real things to take up into your organization and say, this should be a business priority guys.

[00:17:16] Dino: And it is a core competency, right? Are you pursuing a core competency in this realm? are you going to rely on,third parties to help you with this? Yeah, you have your in house legal. Do you have in house legal or do you rely just using legal when you need it?

[00:17:27] Jim: because you're going to need that as you start moving forward. we were talking about people coming in and from all over the world at any time and any of those devices. It was a five minute discussion with the plant manager.

[00:17:37] Jim: I know that, right? And that's what they said. So those are very real things that you have to be, a detailed person to understand to go, Hey, wait a minute. Is that good for the organization? on one hand, we're keeping everything going, but from a security standpoint, we need to come together and say, let's figure out how to address this.

[00:17:53] Jim: And make it a business priority. Now that it's a business priority, let's start digging deeper. Let's find, that knowledge base that we need to move it forward. which is where we generally come in into those areas, right? Which is we think we have a problem. can you guys help?

[00:18:10] Jim: identify those things and it's whether through, combination of us being on the floor and, cracking those panels and looking at and tying it back to their network and do it, running ideas, devices or defensible architecture, any number of things. man, that's just so much to pick from, but you have to go.

[00:18:25] Jim: These are very real things and we see them all over the place.

[00:18:28] Dino: And you bring up a valid point. Who are you interviewing? You're not interviewing the IT guys. No, we realized that what are they going to tell you? You don't already know, right? you can see the technology that they use and the roles that they have.

[00:18:38] Dino: Networking person, security person, Program project management person, you can look at their ideas and look at their data center in a plan and see how they put things together. Who do we go interview? Who is it that we go and talk to, to discover this stuff? It's the plant manager.

[00:18:52] Dino: his or her direct line managers, right? The facility managers. Who takes us around and pops open those panels, right? The AJs of the world, right? The guy that's got all the keys to these, all these 50 machines that you got out here and knows what those machines do and when they run and when they go down, right?

[00:19:09] Dino: If you're not doing that level of discovery and interviewing, then you're never going to get there, right? You're just not. and that's the job. OT cybersecurity is what I just described. It's not sitting and talking to CIOs, CISO and their networking team and this much of it. That's a piece.

[00:19:25] Dino: The bulk of the work is going out there because one plant. And going and talking to that group of people, you're going to go get a whole different flavor and story at another plan, even within the same organization. It's maybe something a little different. Then this plant 7 years old.

[00:19:38] Dino: This one's 15 years old. That 1 over there is 30 years old. this 1 here is under construction. They're different. So that conversation has to be had with each 1 of those to really get to The task of what are these assets doing? What are the risk and vulnerabilities associated with them?

[00:19:52] Dino: How are they connected? How are they, interact with one another? How's the OEM involved? What's the remote access look like? there's a whole bunch of things that you got to go through in order to get this story figured out.

[00:20:02] Jim: And that's why we say this many times to all our people, we require field time is just as important as behind screen time to put all this stuff together.

[00:20:11] Jim: and I'll do my quick wrap up, so I hope. If you stuck around, we threw out a bunch of things that we see all the time, right? and it's not everything that list is long. Just what we got to touch on. But I would say, To wrap the whole thing up, as I always say, get better, and this is on the floor.

[00:20:25] Jim: These are things that you find on the floor. So you have to figure out a way to get on the floor, get somebody on the floor to have those conversations to look at what's going on to really start putting this puzzle together and how to move these moving up. Forward how to make it a business priority. So with that, I'll hand it off to you for the final wrap up.

[00:20:44] Dino: what I would say that it's not a Monday through Friday to 5 gig to do this. it's after hours, it's weekends, you got to work around plant shutdown events in order to get in there and really do this work and take the time to do it right. and if you're not willing to do that, like I said, you're not going to get there if you're not going to go spend the time in those particular.

[00:21:01] Dino: windows to go do discovery the same time you really got to start learning the plan operation of the process and the roles of the leadership and teams out there in those manufacturing facilities and who their partners are and their supply chain. that's the job. if you're not doing that, then you're not going to get to the OT cybersecurity cybersecurity posture you're

[00:21:18] Jim: after.

[00:21:19] Jim: right. thanks, everybody for attending and, until the next podcast, appreciate you listening,