How CISA Supports Cybersecurity from Critical Infrastructure to Elections

[00:00:00] Roger: working with security researchers And security analysts, whether it's ones and twos within security operation centers without a miss every single one of them, they're so fascinated with the industrial control system world or OT cybersecurity world, the idea that something could go boom, instead of just shutting down a mail server, right?

[00:00:46] Craig: Hi, i'm craig duckworth and today with me I have roger hill from hill strong group security Welcome to the show roger.

[00:00:55] Roger: Hey, great to be here. Greg. Thank you

[00:00:57] Craig: So I guess let's start with a little background about yourself Why don't you give the listeners some background on you what you've done and then we can kind of jump in I really want to Get your perspective of kind of what you've seen change And evolve In the security of the ot environment since your Infancy, I guess in the ot environment because i'm sure there's been a lot of change a lot of of of migration But let's start a little bit about your your background your history And then we can kind of go into what you've seen over the lifespan of your career

[00:01:32] Roger: Wow, Craig, I think you just asked me how old am I?

[00:01:35] Craig: I did, but kind of in a roundabout way. Yeah,

[00:01:39] Roger: so listen going back, I've been in the industrial automation world for, let's just call it 30 years, right? Okay. Long before we had the name OT. Or operational technology. So when you pose the question of what's changed, what's really interesting is, I would say I definitely predated any notion of cyber security, right?

[00:02:02] Roger: So I think, you know, if we go back to the free Stuxnet days, the idea of Quite frankly, from an industrial automation, you know, having spent over 16 years for Siemens and, you know, a couple of years with Rockwell. So I definitely have the time with industrial automation vendors, but just looking at from that system test in a pre stocks net world, your system has never even considered a test scenario of someone purposefully trying to attack a system because we come from such an engineering driven industry.

[00:02:35] Roger: World. So I think from that post Stuxnet world, that's where there was that first wave, which was really the evangelism, raising awareness of a, we really need to pay attention and secure these systems now. Right. And I think that early work in that post Stuxnet world was really Yeah. Okay. What are the, sectors that really need to focus on this?

[00:03:03] Roger: I think early on, oil and gas was really leading the way anyway. That's where most of that early cybersecurity work. I think when you look at North America specifically, you know, it's largely not doing anything offensive, but largely it was a compliance play around NRCSEP and, I think there's, pros and cons, but that's, that's, that's, that's, that's, that's Time for a different episode.

[00:03:22] Roger: I think. So I think when you start looking at the other sectors in that first wave we clearly saw from our market research that chemical sector was really leading that wave and manufacturing was not convinced retro China were out. They were right. I mean, they didn't view themselves as critical.

[00:03:41] Roger: so I think one of the things that really changed in that post docs, then we went through a secondary wave, which was all of the emerging tech startups of specific OT designed cybersecurity solutions. that was a real shiny object center wave, right? And now I think it's really interesting because ransomware has changed.

[00:04:03] Roger: That dynamics, I think, in terms of a threat, because now the manufacturing sector, you know, they've been targeted, you know, either number one or number two target for ransomware. Ransomware, quite frankly, has emerged as a completely different threat now with ransomware as a service.

[00:04:20] Craig: Yeah,

[00:04:21] Roger: absolutely. It's kind of quite the evolution.

[00:04:24] Craig: So, I were to Unpack that, I would say everything has changed.

[00:04:28] Roger: You know, you could even go back to say one of the early foundational elements was the adoption of the Ethernet based communication protocols at that field device network control system network. Layers right from the old rs45 rs32 32.

[00:04:46] Craig: Okay. Yeah, so brought up the shiny object new ot ids slash technologies whether they are Ids they are secure mode access. They are intrusion prevention Whatever the flavor is, there's a lot of them, you know but I think that As the market is maturing, as the OT environment is maturing, and organizations are accepting that there are purpose built technologies for this industrial space, we are starting to see a shift.

[00:05:25] Craig: Are you seeing the same thing?

[00:05:27] Roger: Yes, I think there's another shift that I don't hear. Many people talking about right now, and what I have purposely, or I, what I have specifically experienced myself formerly at the Koudelski Group and now within Aelstrom is a lot of the large manufacturers I think in that early shiny adoption phase, what was happening is you had acquisitions of this technology, not based upon systems, System requirements, definitions, instead, it was really everything was backwards.

[00:06:03] Roger: It was acquiring technology and then determining what can that technology do for us and then build a program around it. And that's, completely backwards, right? So I think what, when you look at the top four players in that, that I guess, you know, slash, you know, Pakistan and Rome, and I say that tongue in cheek because I think they all have very strong merits and they're very important to the sector.

[00:06:28] Roger: Sure. Having said that, each of those logos, each of those brands are only as good as how well was that system architected. Deployed, configured, and tuned. And I think that's where a lot of these companies are missing, quite frankly, in, the adoption and rollout of these technologies. And then what they find out is they're not realizing the value they thought they would.

[00:06:56] Roger: They've spent millions and then it becomes, oh, we bought the wrong technology. Well, did you? Yeah. we can't lose sight of You have to go after these engineering designed and built systems with an engineering approach

[00:07:13] Craig: Yeah, and I saw a quote that I really like and it's been in the last Three three weeks to month where visibility without context is just noise again if you're not going to contextualize the data and let's be honest These technologies provide a lot of very valuable data both from a process integrity and a cyber aspect, but Where organizations in my opinion fall a little short Is understanding, interpreting those insights and then using that data to make actionable risk decisions based on the organization's purpose, the organization's direction, and how they can better understand that data and what it means to them moving forward.

[00:08:09] Roger: So I definitely would. Disagree with that point, but I would maybe unpack that a little bit and say I think there's a bigger issue at play. So let's just take a use case of a global manufacturer with 100 sites Yeah, right I I think you know as well as I do that more than likely that company Acquired a good percentage of those sites throughout for sure So what you end up with Is Many different types of facilities with many different types of industrial automation solutions, whether it's Schneider or Siemens or Rockwell, and there's never a single logo at any given site.

[00:08:52] Roger: That's the reality, right? And I, you know, it's, it's very complex. So I think one thing that's interesting, Craig, is where did the budget come from, from that initial acquisition of that technology? whether that was a clarity, A drag is consuming arm is whatever the case may be. Was that funded through securities budget to the CISO?

[00:09:15] Roger: Was it funded through operations? Because where that funding originated might drive how that adoption and initiative. Was executed and deployed, right?

[00:09:26] Craig: Yeah, for

[00:09:27] Roger: sure. I think it's interesting because the reality is most organizations, when you talk about context, you don't necessarily have context of those hundred sites.

[00:09:36] Roger: What are the architectures? So, for example, am I just connecting to a span or a span port of that core switch? Right. Or the reality is that control system layer, that critical control layer, where these are the, in terms of the internet of things speak, this is the things that tell the things what to do, right?

[00:09:55] Roger: The PLC, motion controllers, VCS controllers. At that layer, they are typically inside of control cabinet. In that control cabinet, you've got the IO, you've got the PLC, you've got a nice little HMI in the door of the panel. Right. Some VFDs, variable frequency drops, and 99 percent of the communications east west are off that cabinet.

[00:10:15] Roger: It doesn't even go up the trunk of the switch. True. So what does that really mean? That means that the shiny object, you know, technology is not getting access to that data unless that state of server is acquiring some of that telemetry. So that's the, the real challenge because now you're, as an organization, you're making a decision, are we trying to get the technology deployed as a first wave initiative across all 100 sites first, and then go with a secondary analysis of, okay, what is our level of visibility?

[00:10:49] Roger: Are we achieving 80 percent visibility? 70 60 because if we don't do that secondary work Don't we have a false sense of security of what's on our network?

[00:11:01] Craig: yeah, and back to the purchasing portion of that we're starting to see a shift and it's i'm not sure yet what I Think of that.

[00:11:13] Craig: We're seeing a shift to the IT side, as CISOs, CIOs, are being tasked with holistically security for the enterprise, including the manufacturing. I would say the, Negative if there's a part of that that's negative, which again, I don't know that it is. It's just different is They normally run into a roadblock When they hit the plant floor and you have a plant manager You have a controls engineer or a vp of engineering that says well, hang on a minute We can't do any of that without a shutdown and our shutdown comes Next july 4th, and we've got three weeks worth of stuff to put in two days And we'll try and prioritize you but there's no guarantee so get on the list and Submit it and we'll see what happens versus the ot teams that might have procured one site because they were interested in the process integrity portion of the tool.

[00:12:19] Craig: They had OPEX budget for one site. They're leveraging it, but not so much globally for the entire enterprise because they own the assets. So that's the divide and I would say that OT ecosystem needs to really kind of grow up a little and kind of own more of this and partnership with their IT counterparts to be more successful in a large scale deployment.

[00:12:55] Craig: I think

[00:12:57] Roger: accurate. Um, What I would say maybe a slightly different direction. When you're looking at that global corporate entity, first of all, I completely agree that from what I'm seeing, yes, CSOs predominantly are on that trend of owning responsibility for OT cybersecurity. And what hasn't followed as clearly, though, is budget.

[00:13:22] Roger: Right. True. And one thing that I think is really interesting is when you start looking at the, problem globally, right? The real challenge I think is this goes beyond about the CISO, Because what you're talking about, the biggest challenge. It's really around governance and compliance and risk across that organization, right?

[00:13:46] Roger: So what there has to be alignment even above that CISO, and I still maintain the best way to move forward is the cross functional teams. So you have to put together these advisory boards that include The ot subject matter experts. Yes cross domain experts of different security domains It it number one, I think politically It's important from that alignment, but also technically that cross pollination Because it's an aging workforce problem, right?

[00:14:23] Roger: Oh, for sure. So we have to create the type of talent we need within these organizations. And, and that has been the best tool I've seen to solve some of these challenges is really bringing these communities together. Because it takes a village, it really does.

[00:14:40] Craig: I would agree a thousand percent with that.

[00:14:43] Craig: And we see that from a, from a resource standpoint. To eat, not only. Internally but but externally within organizations that are trying to To hire that are trying to grow the talent because it's not available to hire what we haven't seen yet that I hope we see more of soon are universities Education systems that are beginning to foster programs that teach OT cyber, similar to what they're doing on the IT side.

[00:15:15] Craig: But, you know, you look at IT and the 30 year head start they've got, on the OT side, we're still in the early stages. In the last six or eight years, this is all relatively new, and it's not, it takes time. You can't, Rome wasn't built overnight.

[00:15:33] Roger: I couldn't agree more, you know, and I think this is where I think you have to give some of the vendors some credit here because how they acquired their talent for the most part at least I can speak to Rockwell and Stevens in this regard, is, is the engineering development leadership programs, right?

[00:15:52] Roger: Rising talent out of the universities, academia, and then bring them through six month rotations into different functions. And I think part of that is there's another side to this where Siemens or Rockwell both have programs where they're donating their equipment to the universities because it's kind of go back to high schools back in the day, it used to be all Apple, right?

[00:16:17] Roger: That's

[00:16:18] Craig: Yeah,

[00:16:18] Roger: which was a brilliant move because then, you know, you're, you're bringing all these people. Hey, yeah, let's see if I know that's even system. That's like, you

[00:16:25] Craig: know, trained on. Yeah.

[00:16:28] Roger: So, you know, I think everyone is trying to do their part and it, you know, we're, I'm starting to see a lot more programs.

[00:16:36] Roger: So that's great news. But every time I'm talking, especially the plant manager level, you know, when I'm doing global assessments and one of the things I like to talk to the plant managers about is what they're seeing in terms of the aging workforce problem. Without a miss every single one of them.

[00:16:54] Roger: It's one of the biggest issues they face is a talent. You know, this is what scares me the most right now, because once we lose that subject matter expertise, that's an inherent of the critical, you lose the operational resilience

[00:17:11] Craig: without doubt. Without that

[00:17:13] Roger: point. In terms of the animals within the song if you don't have that context, well, then you have risk

[00:17:21] Craig: Well would without doubt totally totally agree, and and I don't know if it's a if it's a stigmatism that Manufacturing is not cool and sexy if it's not People are young talent is looking at the it side because again, let's face it.

[00:17:38] Craig: You're refreshing technology every three to five years. It's all new. It's all shiny. It's all cool. You get down to the manufacturing, you're dealing with a 30, 40, 50 year old machine. That's dirty and greasy. That had no thought of security when the machine was even made. It works. so the philosophy is Why upgrade it and fix it still works.

[00:17:59] Craig: It's making our widget and it's doing it. Finally. Why why change something?

[00:18:05] Roger: It's making money, right?

[00:18:06] Craig: Yeah

[00:18:08] Roger: There's another side of that coin though, right? Because this my personal experience working with security researchers And security analysts, whether it's ones and twos within security operation centers without a miss every single one of them, they're so fascinated with the industrial control system world or OT cybersecurity world, the idea that something could go boom, instead of just shutting down a mail server, right?

[00:18:34] Roger: It's absolutely fascinating. And so every time I have a, whether I'm doing a, you know, some type of a CMMI assessment for a software, whatever the case may be. That's always an engaging conversation where they, they're just hungry to learn more. So there's a hunger out there and fascination, but there's just the unknown, right?

[00:18:55] Roger: Yeah, for sure. So somehow, some way, if we could bridge these gaps in these worlds, the appetites there, Listen, I would say the same thing about some of the process control engineers and control system engineers that we have, give them the opportunity, I'll go back to that cross domain, the cross functional teams.

[00:19:17] Roger: If you do this in the right way, you have organizations that are hungry. We just have to break down silos.

[00:19:24] Craig: Well, and we see the same thing, Roger, across multiple verticals, multiple disciplines when organizations work together, the OT and the IT teams, and they come together to solve the problem holistically, and they're trying to work, you know, you use the talent and the skill sets from the IT side, and you apply the principles and the differences in the way that it works in the OT side, and you come with commonalities Everyone wins.

[00:19:53] Craig: The vendor wins because the software solutions providing is being utilized to its capacity. It's being leveraged for what it's designed to be. The organization wins. They get the data they need. It can pump into Multiple tools on the I. T side to feed a sock, assume a sore or whatever they want to get that data.

[00:20:16] Craig: It's being done correctly in a passive manner down on the O. T side. So it's not harmful. You begin building that trust between the two sides. Everybody wins. I think that that's the best scenario. Whether that's a small facility with one or two sites, all the way to a global organization with hundreds of sites in, in multiple countries.

[00:20:43] Roger: You know, Craig, I've always said when, when someone would ask me the question, what's the difference between OTSOC and Enterprise SOC? Three things. Context. Context. Context.

[00:20:56] Craig: Yeah,

[00:20:57] Roger: you know, Tommy cheap, but you know, the reality is it's understanding what is the impact if we're trying to look at some type of containment post triage investigation, right?

[00:21:09] Roger: If we're looking into kind of containment, what is the containment measure? Going to have an impact of that system, right? Let's take a typical type scenario of you get an alert that suspicious configuration change of a control, right? And it's coming from this IP address.

[00:21:26] Roger: Looking at your tech stack as an analyst often you're dealing with the same tech stack. It's almost like overlapping Venn diagrams. There's a little bit of the tech stack that really needs to be focused. On the OT systems. Otherwise, you can leverage other IT tools or tech. But it's different tooling and processes that are unique for the OT SOC.

[00:21:50] Roger: I've always maintained having your OT SOC as a subset of your enterprise makes perfect sense. Because if you think about it, whether you have an MSSP or you're doing this yourselves with your SOC, your primary mission is to identify threats. As they're traversing your business functions, right? So OT is just another business function.

[00:22:15] Roger: So. There has to be that cross and plus you can extend the life cycle of your teams, go from a tier one to tier two, then rotate into the tier one of the OT and you can double the lifetime expectancy of your talent. Cool. And that's a big thing on the calendar retention side.

[00:22:35] Craig: Oh, for sure it is. You know, and, I would say from a, an OT centric SOC, there has to be that content.

[00:22:44] Craig: You have to understand the why behind it. You look at the IT side and it's, it's isolate and, and remove. You can't do that safely on a calendar. On the plant floor without knowing what the implications of that are and you know for manufacturers They they recognize that so it's a a different beast

[00:23:05] Roger: And that that's the additional processes in place, right?

[00:23:09] Roger: Applying that context so I think that's what's crucial and this is where There's a little bit more engineering work up front to provide that type of context Right. For the analysts that are actually doing this work.

[00:23:24] Craig: So if I were to kind of guess to think about the next five to 10 years, given what we've accomplished in the, the OT ecosystem in the last Seven or eight years ten.

[00:23:38] Craig: Let's call it ten years since these tools technologies began emerging Organizations began thinking about this Where do we need to get? In the next five to ten to really begin making a difference

[00:23:56] Roger: Whoa, that's kind of loaded. You know, it's what strikes me there. Craig is So i've always had been a little bit of a futurist, right?

[00:24:03] Roger: Okay good Formerly as global head of technology management on the security services side of cements That was really kind of one of the key Abilities in terms of that vision, what I would say is a year ago, if you would have asked me that question. I think I could have given you a pretty decent answer.

[00:24:23] Roger: Now, I find myself much more unwilling to project beyond two years. I think what we've seen in the last year just with generative AI and the, you know, the LLMs, it's really changed. And what I mean by that is, Mainly is the,

[00:24:46] Roger: is the way adoption is happening. This is really disconcerting to me is there's an adoption without discipline. There's an adoption without investigation. And I think this is going to come to a tipping point and I'm afraid what that can actually mean. Uh, So I think looking at the potential. Of artificial intelligence for being able to identify and analyze vast amounts of data.

[00:25:21] Roger: Absolutely, this is great, but I think what we have to do is do this in a responsible way. And you know, what is the biases that are inherent within these models? Yeah. These are the things that I'm really concerned about because I think projecting beyond a year or two out now is actually very challenging.

[00:25:41] Roger: On the one hand, that's the pessimistic side. Now on the optimistic side, it's very exciting times because I think the future is all about integration. All of these silos of technology we have now will be broken down. It will all be about integration and having larger data lakes. Now what becomes interesting.

[00:26:02] Roger: At that point level and listen, we, we've seen some signs of this with some of the the cloud interruptions that we've had not to mention the latest incident here is the on prem versus cloud, right? What should be done at the edge in terms of pre processing versus moving just that high fidelity data up to the cloud, right?

[00:26:28] Roger: Yeah. Right. I think that's an interesting discussion of, all critical manufacturing and other critical sectors. What should be at the edge of the facility versus.

[00:26:39] Craig: Yeah. And, and we still see the adoption, depending on where the buying cycle is, if it's coming from an it buyer, they're more comfortable in a cloud SAS model.

[00:26:50] Craig: If it's coming from the OT buyers, they're traditionally in that I want an on prem solution still. So, so we're still seeing that

[00:26:59] Roger: I think there's an in between world right now. So when you look at the digitalization initiatives. Okay, you know whether it's industry 4.

[00:27:07] Roger: 0 or any other type of digitalization initiative What are you seeing in terms of the commonality of where that budget and where that is being driven from? operations it

[00:27:19] Craig: We're still seeing Operations in large capex budget for these type technologies where they're rolling it into other projects It is getting there You They're not quite there yet from the plant floor initiatives from what we're seeing

[00:27:39] Roger: Interesting.

[00:27:40] Roger: So what I have seen is definitely a convergence of IT and OT. OT generally having the budgeting owning the budget, the budgets, the digitalization initiatives. The part that isn't as consistent from what I'm seeing is bringing in cyber security. Yes.

[00:28:01] Craig: 100%. Yes. For digitization, updating processes, controls.

[00:28:08] Craig: Modernization that is ot they are that is by far 100 spot on

[00:28:16] Roger: it's such a look to be Competitive in this global marketplace. We have now global manufacturing sector, it's imperative that they be able to make those decisions quicker.

[00:28:28] Craig: Yes

[00:28:29] Roger: the shop floor You know And so there's enormous pressure You but if I can get one message out, it's again, break down the silos, bring cyber security in on the earliest planning and design phases, not as an afterthought, but just as a control.

[00:28:51] Craig: and I would say that for anybody listening if to echo that if there's one Commonality that that everybody that we talk to is saying you have to start working together. You have to start taking steps Now you can no longer just ignore and say, you know It's not going to happen to us or we're immune or pick it Pick a reason or an excuse is really what it is, but organizations have to start doing something to get better and they can't ignore it.

[00:29:24] Craig: that's a great piece of advice So this conversation could go for days roger in it And I don't think that we're going to solve anything but hopefully the listeners can take out of this a few ways or some Opportunity that they can look internally, look at their own processes, look at their own teams and say, Hey, do we have what we need as an organization to sit down together with C suite operations?

[00:29:55] Craig: IT and begin addressing there's going to be more questions than there are answers initially and every organization goes through that But again, you have to have the conversations

[00:30:07] Roger: Go through the exercise of a day in the life

[00:30:12] Craig: Absolutely. Yeah,

[00:30:13] Roger: and listen, I think for the most part you'll be surprised at how receptive other organizations will be When it's done in the right way.

[00:30:21] Roger: Yes Yeah because then you have to appreciate some of the things that are decided and yeah, if

[00:30:28] Craig: you're the cio or you're the cso and you're Leaving a board meeting and you're being tasked with security for the manufacturing side of your organization My recommendation your very first stop is red wing boot store and buy yourself a pair of Steel toed shoes call the plant manager and say hey, I've got this new initiative and I'd like to come sit in your office for a couple hours, and I would like you to walk me through our manufacturing process.

[00:30:58] Craig: I want to see what it involves, and I don't want to look at it from the mezzanine. I want to be on the floor.

[00:31:06] Roger: You know where it starts? Show up, have the CIO or the CISO show up at 7 a. m. stand up for lean manufacturing.

[00:31:15] Craig: Yeah.

[00:31:15] Roger: They will always be amazed, Agile, I know Agile from South Florida. It originated from lean manufacturing, right?

[00:31:24] Roger: And go through the management of change meetings. And I think there's an appreciation of, wow, there is much more to this. Yes. This, the, the lean processing. There is more in common. Then there, then you wouldn't actually think it's a great foundation to go from. Yep.

[00:31:41] Craig: Nope. I agree. very much for your time, Roger.

[00:31:44] Craig: I, I really appreciate you being on the show. the listeners were able to get some good feedback and, you know, until next time, thank you very much.