ICS Cybersecurity Knowns, Unknowns, and Actions to Take with Industry Expert & S4 Founder Dale Peterson

[00:00:33] Craig: Well, good afternoon, folks. This is another episode of our podcast, and my name is Craig Duckworth, the president and CEO of Velta Technology. And today with me, I have Dale Peterson. Dale, want to give us a little bit of background about yourself?

[00:00:47] Dale: Sure. I'm in the ICS security business, have been since 2000.

[00:00:51] Dale: Right now I spend most of my time on my S4 show, which is every this year it'll be March in Miami South Beach. but I still keep my hand in consulting and do a little bit of this and that. And I guess if anyone wants to know more about me, just go Dale dash Peterson. com.

[00:01:07] Craig: Perfect. That's a great introduction.

[00:01:09] Craig: I know that you've been in the ICS, as you mentioned for 20 plus years. What led you? Into the industrial spaces. Are you a trained engineer by

[00:01:19] Dale: degree? No. like a lot of things in like it's a dumb luck. my company digital bond started actually 25 years ago this month. So I wrote a couple of articles about how we got it, but we started as a product company to develop a smart card based authentication solutions when E trading of stocks was just starting.

[00:01:40] Dale: we failed, never got a big beta. We started doing consulting to pay the bills because we didn't want to raise more money until we got something in and we did consulting, financial insurance and all that. And then some water utility called us up and said, Hey, can you do an assessment of our SCADA system?

[00:01:57] Dale: Because we had a pretty good blog site. We had a fair amount of attention on the internet. And you know what consultants say when they're asked that question? Can you do it? Sure. I don't even think there was Google. Then we're trying to figure out what the heck SCADA is.

[00:02:08] Dale: we stumbled into that in 2000. And when I went on site, I just loved it. It was great to be able to go out in the field and see these pumping stations. That part of actually seeing the physical process was fascinating. And the, just the complete lack of security. Again, this is 2000 said these people could really be helped.

[00:02:29] Dale: You could make big differences very quickly and it was fascinating. So I just found ways to do more and more in that. And by 2002, that's all I did. That's all the company did.

[00:02:40] Craig: No, that's fantastic. A lot of people in our industry started in the industry and there are some like yourself.

[00:02:49] Craig: My background is not in skater but as the industry is evolving as the industry is beginning To mature, I look at it and we've often said that this equates back to the beginning of the internet or the beginning of switching where Industrial cyber and industrial, networking is really starting to take off.

[00:03:10] Craig: I would say, this is the infancy of it almost for where it needs to get versus where it's come from.

[00:03:18] Dale: Yeah, it definitely has moved a lot slower than what people thought. I would look at most of my projections. always say I'm directionally pretty accurate on timing wise.

[00:03:28] Dale: I'm always, almost always optimistic. So the things that I thought were gonna happen in 2010 are starting to happen now and such, but I'd agree with you, it is still a wide open space. So if you're coming into this space, you can go from newcomer to best in world in some specific niche in two to three years still, which is amazing.

[00:03:51] Dale: And a great opportunity for people.

[00:03:53] Craig: It is as organizations are looking to bolster their staffing. Again, we hear this every day. There's a talent shortage of in the millions for people that understand industrial networks understand. Cybersecurity and they can marry the two together, whether that be an individual organization looking to bring on talent for themselves or the vendors in this space that are rapidly trying to take the talent for themselves or, even the IT teams that are bringing over and entering the OT or the industrial space that are trying to backdoor into this because they see the opportunity.

[00:04:30] Craig: Talent is a big concern today.

[00:04:33] Dale: this is a war between the, various sides. A lot of times the engineers and the automation professionals, not all of them, but certain very vocal ones will say, we're the only ones that should do this. We understand this. You need to be an engineer.

[00:04:47] Dale: Sometimes the it people come in with a lot of hubris saying, you have no idea what you're doing. We're going to show you what to do. It really is about communications. And I have, don't do a lot of consulting, but back when I had a team, I could bring someone in from I. T. with an I. T. security background, and I could have them be the second person on a job immediately.

[00:05:08] Dale: And within 6 months, they could be very helpful to the O. T. community. As long as they didn't pretend they knew everything and they knew how to communicate and ask the right questions with the engineers think that's the biggest thing, the idea that we're going to create these unicorns, these people that know engineering, automation, and IT or OT security, they're very rare, they're called unicorns for a reason.

[00:05:32] Dale: I don't think we're going to get there and I guess just one other quick thought on this. I think it would be great if more engineers and automation professionals moved to OT security, but that might be unrealistic because they didn't go to school. They didn't find their passion. That's not what they got in.

[00:05:48] Dale: At the job market to do right. They wanted to be an engineer. So you talked to someone who loves engineering and says, Oh, by the way, you're going to be worrying about how the switches are programmed. And you're going to be an active directory domain admin and a firewall. And I want to do.

[00:06:02] Dale: They're certainly capable of doing it, but it's not what they want to do. So why try to force them to do it?

[00:06:08] Craig: Yeah. I would say you're spot on. So in the, years or 22 years that you've been doing this with Digital Bond and the S four, how have you seen the industry evolve? You mentioned in the beginning and even today, it's still somewhat of the wild West, depending on how you look at it.

[00:06:27] Craig: How's it evolved in the space you've been in so far?

[00:06:30] Dale: That's a really broad question. Cause as I mentioned, it's evolved slower than I thought. I would say the evolution that's happened in the last 10 years is as people have been getting more and more involved in this.

[00:06:43] Dale: The easy wins are gone. The ones that everyone understood. Okay. We're going to segment it from OT, We're going to have two factor remote access. We're going to have to be able to recover from a cyber incident. Those things are done. And now we're kind of at the. Good practices stage, and we're at a very dangerous stage where you have all this almost endless list OT and IT security.

[00:07:06] Dale: Good practices that people are saying need to be applied. And some of those are a lot of work with little payoff in terms of risk reduction.the current I can't say it's 20 years, but the current evolution of we're going to create this long list of things that you need to do. To get a good housekeeping seal of approval or to meet some set of practices is probably the latest trend and dangerous trend in my mind.

[00:07:32] Craig: and we see a lot of the frameworks, a lot of the practices on the IT side are very applicable and should be used to try and help develop and guide the industrial, the OT side with different tool sets. The frameworks that have been in practice that have worked are still there and are still a good foundation.

[00:07:54] Craig: You look at what SANS does, you look at some of the other. Organizational trades and how they apply it. It's just take the different tool sets and what works within the industrial space and try and navigate within those parameters. I guess you could call

[00:08:10] Dale: it. might push back a little bit on that.

[00:08:13] Dale: Craig. I don't think we know. For example, some we know certainly don't apply like patching, correct? Pat patching most of OT forget about whether you can do it or not, because oh, you can't patch. I think that's oversold. You can patch. I've seen it done. You can patch during you patch differently.

[00:08:31] Dale: You do it during outages. You do it leveraging your redundancy. You don't just blindly apply everything, but you can do it. But the value of it is much. Less significant for the level of effort. And it's a big level of effort. Imagine, why do I want to patch some web browser vulnerability cross site scripting thing on a PLC when I can reprogram that PLC with clear text, cause it's not authenticated, where's my value in that?

[00:09:00] Dale: So I think we have to look at that and then even the ones that are a little more. Logically sounding, I don't think we really know. I think you're partners with some of these companies that create asset inventories and do detection. And we hear this, you can't protect what you don't know.

[00:09:19] Dale: No, you can, we do protect what we don't know. We have perimeters around things that are protecting them, whether it be a wall around a safe or an office or a locked drawer or a security perimeter, we can protect. I think it's fair to say you can better protect if you know what you're protecting, but how much better?

[00:09:37] Dale: I'm not sure. How do we really know are these asset inventories that take a fair amount of time? Are they really worth it? So what I'm hoping we're going to see is some measurements on this. Is okay. It sounds reasonable. Most of these good practices sound reasonable. Are they actually moving the risk needle?

[00:09:55] Dale: Is this where we should be spending our money and our time?

[00:09:58] Craig: Yeah, great thought. And when you step back and look at it that way, you can absolutely see where that makes sense. The time we haven't had enough time and enough, let's call them. Metrics or statistics of what is happening. This is still a fairly new area.

[00:10:16] Craig: I guess you could call it in what these tools and technologies that are on the marketplace are doing. And you're right. If you look at, the organizations that are preaching asset inventory disability, All the different things that they're doing and the method of that. You can't see what you can't protect what you can't see, I can see that side of the fence because it is true that micro segmentation has been around a very long time.

[00:10:41] Craig: It's just a different way of doing things than is being looked at today.

[00:10:46] Dale: I would say it's time now. So if I were in charge of any OT security budget and with the few clients I still work with, every project has a metric of success. it's not an implementation metric.

[00:10:58] Dale: It's not how many patches have we applied. It's a metric, how many cyber incidents have affected a key mission metric, so that's what we're measuring and I think we need to start doing that. You could look at a program like crisp that DOE has or I've even asked, I've asked the CEO of Nezomi networks, like, how should someone measure if your system is successful and.

[00:11:21] Dale: Neither one of them, the director of Caesar, they have implementation metrics. Oh, we're seeing this percentage of the thing, but how many attacks have you stopped? What consequences have you reduced? Or if you haven't stopped them, how have you reduced the consequences? We have to start asking those questions and building those metrics in and we're, the metrics are going to be bad at the start.

[00:11:40] Dale: We're going to, we're going to find out all that metric stock stunk, but we need to start doing that.

[00:11:45] Craig: it's same thing as the insurance industry, they are driven and built off of, and only operate on metrics and actuaries and numbers. And as a whole, the insurance industry operates very well on that.

[00:12:00] Craig: But when they enter the cyber aspect and they're entering. Industrial and manufacturing, really are no metrics and that we recognize that the last Three to five years, they've all really taken a bath on the losses and they're trying to be very reactionary on how they adjust from a policy, premiums to coverage, to no coverage, to.

[00:12:25] Craig: The exclusion list and we're seeing it all over the board. So that kind of goes back to that There has to be a metrics a yardstick. Where am I measuring up on this? Definitely. Okay. So do you think that organizations are? Taking it far enough and are being proactive enough to relax some of the regulatory oversight.

[00:12:48] Craig: Or do you think that you're going, we're going to continue to see DHS and the various organizations continue to force mandatory. Regulate regulations in the industry because they don't feel private sectors are going to do enough. What's your thought on that?

[00:13:05] Dale: There's what they should do and what they will do.

[00:13:07] Craig: I agree with that and i'm that's a different conversation But yes

[00:13:10] Dale: i've asked my question this for years and I finally wrote an article on it I it was I think it was titled what? Regulation would I put in place if I was omnipotent, or if I was king, and what I think they should do, especially for critical infrastructure is they should say, what are your minimum effective operations?

[00:13:30] Dale: colonial pipeline, the example, everyone uses, they provided jet fuel and gasoline through their pipelines. What is the minimum they need to provide? To prevent a high consequence event for the country, it probably isn't full capacity, right? They probably need to be running at some diminished capacity.

[00:13:46] Dale: And then what is their recovery time objective for that achieving minimum a cyber incident, if they can, That's what I would regulate. I would say, okay, government you're, identified as critical infrastructure. We're counting on you to provide this much power, this much water, this much oil and gas.

[00:14:07] Dale: wE realize you might have an incident could be caused by weather. It could be caused by covid. It could be caused by labor issues, all sorts of things. And it could be caused by a cyber incident. But you're committing to us that if it's caused by a cyber incident, you can get it back up and running within this amount of time.

[00:14:22] Dale: And you have to commit to that. There's going to be fines If you don't achieve that, and we're going to evaluate both what you said you could do, and how you're going, how you have a process in place, an incident response recovery capability to meet it. And that's what I would require, because these long checklists.

[00:14:43] Dale: if you think about it, colonial pipeline could have all these long checklists. They still could be hacked and we still would be faced with the same problem, I don't think these long checklists are the way to do it. And I guess the way I'd approach those is they're coming.

[00:14:55] Dale: I don't see any sign that they're not going to increase, but I put those in the regulatory risk category. . So as an organization you have to deal with regulatory risk, which means you have to do these sorts of things, right? But I don't really see them being the way I would drive my cyber, addressing my cyber risk.

[00:15:14] Dale: So

[00:15:14] Craig: you're you view this almost as SLA, I think of the financial industry and I look at MasterCard, visa, amx. . And they have 99.999% uptime guarantee, if you wanna look at, because of the criticality of card processing. Yeah. Around the globe and how they have redundant systems, they have high availability, fault tolerant backup.

[00:15:36] Craig: All of the disaster recovery, all of the things that are necessary to maintain that critical infrastructure within card processing. So you're viewing it the same way. And that makes sense. If more organizations did that, then there would be less need for forcible. Regulatory oversight, possibly.

[00:15:56] Dale: Yeah. And you're seeing this, the simple risk equation, likelihood times consequence. Security is all focused on reducing the likelihood or 90 percent of it, or more focused on reducing the likelihood. You're seeing some cyber informed engineering consequence, what is it? Consequence driven, cyber informed engineering, CCE from INL, but it doesn't have to be some complex thing like that, you just have to say.

[00:16:18] Dale: Assume my system has been hit. How am I going to get that minimum effective operations running? And what I'm finding with a lot of asset owners after they've hit those basics, they've separation some basic remote access security, some basic detection, the next thing where they're getting their biggest risk reduction is consequence reduction.

[00:16:38] Dale: Like Oldsmart, Oldsmart was a great example, small water has almost no chance of being cyber secure, but I would agree with that, but the solution there was all they needed and they probably had it in place was if they had that pH sensor directly connected to the pump. So that if the pH got too high, because someone jumped all these chemicals, which they shouldn't have been able to do, but if they did it, it would shut off the pump.

[00:17:02] Dale: So it doesn't matter if the attacker has complete control of their system, they will not deliver deadly or hazardous water to their customers.that's a cheap consequence solution. That's 100 percent guaranteed that this thing can't happen. And that

[00:17:16] Craig: makes sense. And what we've heard over and over from an industry to Oldsmart specifically is that'll never happen to us because we don't use team viewer.

[00:17:25] Craig: And they're missing the entire point of what happened and how it got there. It doesn't matter if it's. TeamViewer or any other remote desktop application, it happened and it's not because they use TeamViewer. they're not seeing the entire picture, but I agree with that. Something that would stop it is more effective.

[00:17:44] Dale: What, and this is interesting because the EPA just pulled back the requirements for water utilities to do assessments. They had this thing that they pushed down and the states and the utilities pushed back on it. And It's going to come back, but I would say to the water utility that I just ask him 1 question if you're under this size and your control system is completely.

[00:18:04] Dale: Unavailable or under the control of an attacker. Can you still deliver potable water to your customers? And if the answer is yes, then, these checklists don't matter. But when you start to say no, now you have to worry more about these

[00:18:17] Craig: things. these organizations just need to get back to some basics and truly look at a risk score of, what are their.

[00:18:24] Craig: Processes and procedures and what's the consequences if any of those in line stop? How quickly can they recover from it? It doesn't need to be super complicated,

[00:18:34] Dale: when you were saying we're at the infancy of this, some of that infancy offers great opportunity virtualization where virtualization, so it can come back like this.

[00:18:44] Dale: And when we start to see virtualization at level one, Siemens now has virtualized so you can do that everything, but the IO is virtualized imagine you have a control center. That you can just shift to a different location in the cloud. And so you can spin it up. Now you might still want to have some basic capability on site.

[00:19:00] Dale: So if the cloud's unavailable, but all of a sudden I lose everything. Oh, no big deal. I just spin it up over here. I'm back up and in 30 minutes.

[00:19:09] Craig: we see that a lotFrom a gold standard of virtualizing older operating systems, whether it be a Windows XP or Windows 7 because it's easier to do that and put it on a server somewhere than it is to rewrite the controls for whatever process that is managing.

[00:19:26] Craig: Sure. the same process. Okay. So let's shift a little bit to S4 and what led you to kind of bring that to the industry? What, was it your love of the industry and trying to help guide it and try and see a need for what? Tell us a little bit about that and what it does.

[00:19:43] Dale: It started because I had a researcher, Matt Franz.

[00:19:46] Dale: He's not in the ICS world anymore. Unfortunately, we lost him to other worlds, but he's still tracks it, but he came up with the first. Vulnerabilities that were publicly disclosed. In the O. T. I. C. S. World. They were part of the I. C. P. Stacks. The information electric utilities send around.

[00:20:04] Dale: So pretty important vulnerability. And I said, Matt, you need to write this up and present it at an event. And he said, there's nowhere I can present this where they'll understand the security and they'll know what an I. C. S. Is. I don't even think we were calling them ICS back then. I said let's, maybe we need to create an event.

[00:20:19] Dale: So we created one, excuse me, with about 40 people at the first event and really just designed for people that understood both those worlds and we wouldn't need to spend, have a lot of sessions on what is a PLC, what is this? They came in knowing that, and we didn't have to explain what is the industry for

[00:20:36] Craig: the

[00:20:36] Dale: industry.

[00:20:37] Dale: It was very technical at the start. We had actually a book as far as the SCADA security scientific symposium. So we had papers written and a lot of the early efforts were just to grow the community. We had Whit Diffie as the first keynote cause he helped grow the community. So that's how we started it.

[00:20:55] Dale: And then as it grew and got larger, we really. Shifted it more. we still have the technical stage, but we had other stages as well. There's three stages now, and we've focused on this create the future mission. So what we try to do is we try to bring in a really forward looking group of people put them in a very creative environment to take too long to explain how we do that, but it's very different than a typical conference and then just throw a lot of stimuli at them.

[00:21:21] Dale: In sessions and in social events and inside events and just try to put them in this thing so they can cook and come up with new ideas that will, come to fruition in the next 1 to 5 years.

[00:21:34] Craig: we were there last year, we're intended to be there this year, and the conference was run very well.

[00:21:38] Craig: And I would agree that it really is like something that's completely different. It's on steroids for this industry. you and your team I would say, are very cutting edge for what is happening and helping steer this industry, maybe that's not the right word, but trying to help keep it corralled and focused on the industrial space versus, All the changes going everywhere.

[00:22:06] Dale: first of all, I think there's a lot of good events out there and putting on an event is very personal, especially if you're not putting on hundreds of you, there's some that put on 10, 20 those have their value too, because they go all around.

[00:22:18] Dale: So if you're local and you can't travel, they're fantastic. And there's a lot of good events that are aimed. And getting people into the industry and those, quite frankly, there's a larger group of people that need to attend those events, we need to bring in thousands and thousands of people for that event.

[00:22:34] Dale: But what we try to do is we really try to think of the target attendee, and it's the person that's gone to many of these events. It's been in the community for a long time, probably not the person that says it won't work in ICS. They'll probably be very unhappy at S4 but they're welcome.

[00:22:50] Dale: Everyone's welcome. There's no gate that says you have to have this experience to come, but we designed the event for someone who is deep in this field and really understands things and is saying, how are we going to change this to make it better? And that's really our goal. we all even put in talks that completely disagree with each other, conflicting ideas.

[00:23:10] Dale: I'll put in sessions that I think are dead wrong. If I think the person can represent that point of view, because do worry in this industry that we're really getting these blinders on that say, this is the way it works. I don't think we really know it. It goes back to what you said at the beginning.

[00:23:26] Dale: It's still early. I don't think we really know the right solutions to this. And I'd like to see more ideas out there. Tried.

[00:23:33] Craig: Again, that goes back to part of the talent piece is, when I came into this five years ago, didn't even know what an industrial network was. And I'm not by any stretch an expert at it, but I know I'm learning.

[00:23:49] Craig: And I look at it differently because my background is casino and gaming and finance there is not one way to get to the same result There are multiple paths to get to the same to an end result. So I always question that Is that the only way we can do that? What if we tried it this way or what if we looked at a different path because I don't have 40 or 50 years looking at the exact same thing every single day and that's it has to be that way, correct?

[00:24:17] Craig: Because we've always done it that way. And that's the only way we can do it. And my response is, but why? So maybe there needs to be more of that.

[00:24:25] Dale: think there does. And I think there needs to be more crazy ideas. And I do think we need to bring in related field type talkers. we saw with the U S government strategy, cybersecurity strategy, they're talking about shifting liability to the vendors, and we've seen a lot of people in this industry write about that. But quite frankly, most of us know very little about liability law. I actually have Stuart Baker from the Cyberlaw podcast, who's been involved with Cyberlaw for 30 years in liability to come and tell us, no, this is how liability actually works.

[00:24:57] Dale: There's a lot of case law about liability and if it's passed and if it goes, this is what you'll actually see. So we need to bring in people that understand the law, that understand economics, that understand how boards deal with risk management, who understand insurance. I think, again it's the blinders.

[00:25:14] Dale: We think we know how a lot of this stuff works. And every time I talk to these people, I'm amazed at. How little I know. How little we

[00:25:21] Craig: know. Understood. We've covered a lot of ground today. On a last note if there's one piece of advice that you could give listeners that are out there that are trying to figure out one maybe where to start or something that they could do To begin the journey at their organization or where they are today what do you think would make the most sense of here's a good nugget that if out of all the stuff that's out there from Everybody that's spouting the different things.

[00:25:51] Craig: Here's something that really could help get you started or to the next level. What would that advice be to them?

[00:25:57] Dale: One piece I would say I think I already hit one. Is I would measure everything. I would not approve a single dollar if there isn't a measurement and the measurement tied to a key mission metric. So how much does this prevent? I think that's really important, but 1 that's less obvious, I think, is I just would really open my eyes to the consequence side of the risk equation.

[00:26:19] Dale: I would talk to my engineers and I would say, what are the really bad things that can happen to our process? How are we stopping those really bad things? And they'll say, Oh, we have these systems in place. And then this is where OT and engineering can work together. As you can say is there a way a cyber attack could eliminate that protection and allow an attacker to cause the high consequence event?

[00:26:41] Dale: The simple example of this is. A lot of times you'll see people, you'll say, what's bad that could happen and they say if this temperature goes over this, or if it spins faster than this, bad things happen. But don't worry, that can't happen. We have logic in the PLCs to shut things down. If that's going to happen then you talk to him and you say a cyber attacker could change that logic, what happens if that happens?

[00:27:05] Dale: And then you start to say, oh, maybe we put in this physical trip. That doesn't require like that pH sensor I talked about or something like that. I think people could get huge dramatic and the nice thing about the consequence side. it's not a guess, like when you put it in a security control, you can say, how much does this reduce the likelihood of an attack?

[00:27:28] Craig: Subjective, very subjective. We

[00:27:30] Dale: try and there's things like fair and that, but they all seem to fall back on subject matter experts opinions. Whereas with consequence, you say, if I put this in, it doesn't matter if they own this, it's going to shut it off. The bad thing is not going to happen. So it's really good to go to executives and say, if we spend this much money, this really terrible thing can happen.

[00:27:51] Dale: I would do that. I would prioritize that more. And that also gets that engineering OT conversation going in a very productive way.

[00:27:59] Craig: That's great advice. Again, back to simple basics. What that engineers do every day, all day long. And it's questioning the norm. It's going back to say, how do we truly understand our systems and what we do and protect them in a way that may not being force fed to us.

[00:28:21] Dale: I think after you have the basics in place, after you have your firewall, your two factor MODAC. Access a few other things in place. Because those are, starters. So if you're talking to someone who's brand new, I'm still amazed because I usually talk to the leading edge, the top, the 10 percent early adopters, but I'll run across people who will still have manufacturing environments that are flat networks.

[00:28:46] Dale: The PLCs can be pinged from the. there's a few things you want to do first, but once you get those in place, then the decisions become a little more important as to where you spend your time and money.

[00:28:55] Craig: Great insights. Thank you for your time today, Dale.

[00:28:58] Craig: It's always a pleasure to talk to you. That's. Great advice. And hopefully someone listened today can take that and back to their organization and make some positive steps forward.

[00:29:06] Dale: We appreciate that. My pleasure and good luck with the show.

[00:29:09] Craig: