Federal Agencies Can Enter Private Networks to Hunt Malware. Is Your Plant Prepared?

[00:00:00] Ken: Two of our production lines are running on that server right now. We're not going to be upgrading that. But then it was like, okay, well, shoot, I gotta get OT a seat at this table.

[00:00:07] Intro/Outro: Welcome to the Industrial Cybersecurity Insider. In each episode, we dive into the world of industrial cybersecurity. Join us as we cover the latest trends, innovations, and practical insights as we talk with leaders and practitioners across the industry. Gear up and let's get into this week's episode.

[00:00:37] Dino: Hi. Welcome back everybody. I got a good one for you today. Today I'm joined with Ken Cully from Rockwell Automation, and we're gonna kick off a two-part, conversation on some challenges regarding OT cybersecurity. So Ken, welcome to the show.

[00:00:53] Ken: Hey, Dino. Thanks for having me.

[00:00:54] Dino: Give a little background on, on you and how you got to where you are at this point, where you been.

[00:00:59] Ken: Yeah. Well, man, I have been a lot of different places. My first encounter with OT was probably, oh gosh, over 20 years ago. And actually, I mean, my first job in high school was technically working in a process environment. Not automated, it was a all manual labor book bindery. that was interesting.

[00:01:16] Ken: But my first job as a student engineer back when I was going through university was with, , Husky at one of their gas plants. They've since sold it off to another company, , here in Alberta. But I spent a couple of summers out there doing little bits of ladder logic, but database programming, SCADA system stuff.

[00:01:34] Ken: Second time I was there I was helping rip out the old SCADA system, put a new one in., And since then I've just kind of always ended up in the orbit of OT, right? I worked for, you know, another student gig was at a water treatment plant pipeline company. And then when I graduated, none of that materialized into anything as a career.

[00:01:51] Ken: I actually wound up doing IT for a little while, before my wife and I ended up, , expecting our first, and that necessitated sort of a, a change-up in terms of household income. So I took a position with a company called MatriCon.

[00:02:07] Ken: If you worked in the OPC space, that's a name that's fairly familiar. And so I joined the cybersecurity group there. They got acquired by Honeywell, , couple of years after that. I left them for a while. Well, I stayed with Honeywell for a couple of years, then I left for a while and went to Inter Pipeline, back into the pipeline business.

[00:02:25] Ken: , Came back to Honeywell, and then when my team at Honeywell kind of dissolved in the post-pandemic era, I had an opportunity to jump over to- Verve Industrial, marking the second time in my career that a man named Rick Conn hired me into a role. Uh, the first was at Matrixcon. And then, of course, Rockwell came in about a year after I joined Verve and bought the company.

[00:02:52] Ken: , And so I've been with Rockwell ever since. And I kind of keep moving between roles, within Rockwell. I started with their delivery group, so actually installing the security software,, back when it was known as Verve. And then, kind of as the name change was happening, I moved on to what was called system support because we decided we needed sort of like frontline support operations, , client-facing stuff.

[00:03:13] Ken: And then that group got handed off, that responsibility got handed off to another group, and so now I am running what's called delivery readiness, which is kind of a combination of training and working with clients ahead of security installs to make sure that their environment is ready for us to actually come in and put the software in place.

[00:03:31] Ken: Yeah.

[00:03:32] Dino: And so a little background for our listeners. , Ken and I met a few weeks ago in Milwaukee at a, at OT cybersecurity, , training program that, that Rockwell was putting on. Ken was leading it. I'm actually an ex-Rockwell guy myself. I worked for Rockwell, uh, back in the, the teens, you know, from the 2015, 2018 timeframe, specifically working in the OT cybersecurity, , arena for them.

[00:03:55] Dino: Just to get everybody, um, kind of up to speed on this connection and this relationship. And the topic we were going to talk about today , is the Purdue Model, and has it outlived its usefulness, right? , I have a perspective on it, and Ken reached out to me and said, "You know, we should talk about this."

[00:04:11] Dino: And so I, I just kind of want to open it up, Ken. I, I know you, you saw what I had written and, , I would just like to get , your view on that Purdue Model., And where do you think it's, is it outlived its usefulness, or is it gonna be there for the foreseeable future? And, and I'll give you my perspective on that, , for the audience.

[00:04:30] Ken: Hmm. I'm kind of of two minds about it because on the one hand, the model itself is getting on in years. It's been around for, gosh, I can't even remember when it was first promulgated, right? Like it's been, yeah. And I mean, it sort of rose to, at least in my experience of OT, it sort of rose to prominence on the back of like NERC CIP.

[00:04:50] Ken: There was a really, one of the first like major, you know, at least in the US, government-backed regulatory requirements, "Okay, you should be, in this particular operational context, implementing some fundamental cybersecurity principles." And network segmentation was one of those, and the Purdue Model really kind of, I think- At least that's when it really came on my radar as like, you know, this is how you should be, or this is a good framework for how you sh- should be doing this, right?

[00:05:22] Ken: And it makes sense at like a framework level, right? You should be putting degrees of isolation between the things that are closest to your process and the things that are internet-facing. And there should be, you know, a couple or more levels between those two things, depending on a variety of factors, right?

[00:05:47] Ken: , The makeup of your plant, the particulars of your control environment, the geographic distribution of multiple locations, right? If you have multiple locations, all of these things kind of factor in. And with a little bit of, you know, massaging, all of that stuff kind of fit well enough into Purdue. , It always felt a little bit aspirational to me.

[00:06:09] Ken: Like it's, it was ... I don't think I ever actually saw a pure implementation of it anywhere, and that's not really surprising, , because, you know, the reality on the ground often s- necessitates making some compromises and some choices that might not, you know, that, that might move you into a place where you're not in pure alignment with Purdue, but it still satisfies, you know, what you have to do operationally and from a regulatory standpoint, and it works.

[00:06:37] Ken: And it's, Purdue enough, it's secure enough, it's segmented enough. Rounding the corner now in like twenty-twenty, twenty-twenty five, I try to refer to it less as like, you know, a strict guideline again, like it's definitely more in sort of the, the framework.

[00:06:54] Ken: If at any point in my career I was using it as an actual like guideline and this is how to do it, now it's more just like a framework, a definition. , But sometimes you have to reach for the definition even if there's something better that you could point somebody towards, because there's still flat networks out there in OT environments.

[00:07:15] Ken: And even if the Purdue model is getting on in years, even if it isn't necessarily the best approach for network segmentation, the most secure approach for network segmentation, it is a form of network segmentation, and it's easy to explain, and it's better than flat.

[00:07:34] Dino: Yeah. Well, the CPWE, you know, the converged plant wide Ethernet design, right?

[00:07:39] Dino: So there's an architectural model for how you- Would design a network, right? And how you can set up those security control points in those different areas, levels of the Purdue model or even the, at the cell zone. I view it as a people process and technology challenge, right? The technologies is not the issue, right?

[00:07:58] Dino: The convergence per se from a technology standpoint is, has moved , quite afar into the environment, right? Especially when you think about networks and applications per se. , and we've witnessed that. Even though there's some nuances with OT-specific security models and technologies, to me it's still a people and process challenge, right?

[00:08:16] Dino: I've yet to meet a, plant manager that's also the CIO, right? In, in my view, right? Because if, if you think about IT/OT convergence, the way that it's being touted, you know, from an industry perspective, , that's kind of the role that you're, you're, you're, you're positioning that CISO or that CIO to be.

[00:08:34] Dino: I know that sounds ridiculous from the standpoint of when it comes to safety and, and process and operations, where how are you gonna get an IT organization to be a- a- aligned and transparent and very collaborative in that control space without being disruptive, right? And even want to own it. I mean, you talk about large layer flat networks.

[00:08:59] Dino: Yeah, we still see them all the time. Very large companies, you know, still have that type of setup in their plant. But what we're not really witnessing is the handoff between IT and OT. That's why I use that Waymo as, as an example, right? We talk about a connected physical system where you've basically taken the driver, the OT person, out of the driver's seat, and you let technology move in society and trans- uh, uh, transport people around, all driven by technology, right?

[00:09:30] Dino: With no operator sitting there in that front seat of that car, right? And when it comes to the plant, I mean, we're just not there generally speaking. Maybe in certain industries potentially, but I, I don't, I haven't seen it yet where they're, where OT is gonna give up control.

[00:09:48] Ken: No. And I mean, at some level you're never...

[00:09:51] Ken: At some level, I suppose you're not really going to be able to, um, completely pull OT personnel out of, um, out of the way. And whether that's, you know, because, you know, QA still often has a human element to it, right? You're just literally putting eyeballs on things. It might be, you might be using tools as well, right?

[00:10:11] Ken: Depending on what you're manufacturing, you might still be using tools as well, but you still have human eyeballs on the process. You still have human eyeballs on the end product.

[00:10:20] Dino: I call it the human in the loop, right? Right.

[00:10:23] Ken: Yeah, exactly. And th-there's many other places where, yeah, the human is still inserted into the loop.

[00:10:29] Ken: , I remember being a, a young, dumb engineering student trying to run calculations on like, you know, pressure in a gas line, was one of the things that my supervisor gave me my first time at Husky. She said, "Yeah, just calculate me the line pressure on that one." And I, you know, applied all the principles I'd been learning in my classes to that date, and,, I got an answer that was mathematically correct, and then the lead plant operator's just like...

[00:10:53] Ken: And all he did is like he literally inclined his head, , and he's like, "

[00:10:56] Ken: you're off by about 15 PSI." He didn't do the math, he could just hear it in the line 'cause he'd been working the plant for 50 years .

[00:11:02] Dino: And you have that, right? But as you and I both know, I, I call it the silver tsunami, which, you know, I can say that 'cause I've got the silver hair.

[00:11:09] Dino: We're losing that, that, tribal knowledge in these manufacturing environments, , that can pick things up just by sight and sound and activity, to know what's going on in order of whether things are going well or things are getting ready to go sideways. How do you get that knowledge into a, a younger workforce, right?

[00:11:26] Dino: , Or leveraging AI, which is a whole different conversation, you know, to bring , to that, uh, forefront into a production facility,, that's, , making whatever goods that are being made, right? , again, back to the people and the process, right? If you really wanna see a true IT/OT convergence, that dynamic has to change.

[00:11:44] Dino: Again, it's not a technology issue per se. It's not. It is a people and process technology or, or people and process issue that needs to be... is the challenge for anybody, right? We see it all the time. You and I, the firms we work for,. we make our livelihood working in manufacturing environments, you know?

[00:12:01] Dino: We know who it is we're talking to every single day, you know, and what our responsibilities are to... Whether it's to put in new control systems and new machines or whether it's the infrastructure that stuff rides upon. We, we don't have a lot of interaction with IT. And I'm not saying that's right, it's just the nature of the beast, you know?

[00:12:19] Dino: And I don't know how you change that dynamic, you know, where CIOs and CISOs, the IT leadership is tapping in to your organizat- organization and my organization to help them with whatever solutions and strategies they're pursuing inside these manufacturing environments that whether it's to, , make them more secure and safe and/or also, improve, their o- operational efficiencies.

[00:12:43] Ken: You know, it's funny you mention that because, you're right. I mean, the technology is there Although, let's give at least a, cursory nod to the fact that technology tends to, shall we say, persist in OT environments a lot longer than it does in IT environments.

[00:12:55] Ken: There's multiple reasons for that. I'm sure you've talked about it on your podcast. I know I've talked about it on mine. On the people and process side, it's funny because especially when I was at Honeywell,, I often ended up in this, just like the interpreter, right? You know, because you would have, , a company, whether it's a power plant, pulp mill, you know, they're trying to do cybersecurity.

[00:13:16] Ken: And so there are standards they're trying to align with. There's audits and gap assessments and all of that good stuff that they're doing. And I'm the one that, you know, is getting brought in, 'cause say it's a pulp mill running a Honeywell control system or a series of pulp mills, right? , I'm the one who's getting brought in to, you know, do those assessments and audits and then, you know, deliver findings.

[00:13:39] Ken: And a lot of times it just comes down to interpreting between IT , and OT. It is like watching my wife deal with the frustrations that she has to sometimes as... She's a nurse, and the healthcare system here, , the health board in Alberta, keep changing names, I can't remember what it's called right now.

[00:13:57] Ken: They did a whole bunch of overhauls to, like, how they do staffing,, staffing scheduling. Whereas before it was, like, entirely phone-based, , she'd have to call in if she wanted to book a shift, they'd call her. Then it became like, well, you could like go to this portal, and then we could like text you notifications when shifts are available.

[00:14:15] Ken: And now it's driven by two different apps, and it's all hooked into their, you know, Azure cloud, Microsoft authentication, blah, blah, blah, blah, blah. , She's got me in the house, so I can sit down and be just like, "Okay, we do here, this here, then we jump to here, then we jump to here." I think I've already lost, you know, her at that point, but the point is, like, she's got me in the house.

[00:14:34] Ken: I can set that up for her. I can show her how it works. I can show her again, you know, three months later if she needs a reminder. , And then she kind of just sits back and she's just like, " My average coworker is 12 years older than me. , They are going to die." And there's something to that.

[00:14:50] Ken: And you kind of see that in OT as well because you get people like, you know, my plant operator who can incline his ear and tell me ex- the exact PSI in the line better than my math can. But if you tell him that he's got to start using MFA, he's not gonna have any idea what that even means, let alone what it entails.

[00:15:10] Dino: Why does that matter, right?

[00:15:13] Ken: Exactly. And it's one of those things where, you know, it's hard to illustrate to people why it matters until it really, really matters.

[00:15:20] Dino: And to your point, even MFA, I mean, I've worked in plants where shift change, the operators don't log out and lo- the new s- operator logs in with a unique ID.

[00:15:31] Dino: They use the same ID, you know, just some guest, - ID to, to get back into the machine centers. You know, they never log on or log off in that environment because , they forget their passwords, and then, of course, you got to get ahold of somebody to reset it for you , and to them, that's losing time, right?

[00:15:47] Dino: , The message that, that I always heard early on in this game is security just gets in the way of getting things done. That was the OT mindset, and if you bring it up to them today, they'll tell you flat out, "It, it's not my problem. That's IT's problem." , They push it off.

[00:16:02] Dino: And, you know, to me, the narrative needs to flip, and this is where I think our ecosystem, the OT ecosystem, has to get better at this, right? Meaning we have to grow and mature and start bringing in, just like we did with safety around machines, we've got to start developing, you know, designing security solutions and technologies into these machines, um, that carry through.

[00:16:25] Dino: And, you know, and if you're really working well with IT, you can give them the hooks that they need to get the information they need for compliance and risk and auditing, you know, to get the asset inventory, to get the risk and vulnerability, just so that they have a way of tracking and measuring it on their end.

[00:16:39] Dino: Not that they're necessarily gonna do anything about it. , They can, uh, lightly nudge on, in certain areas where , you might want to think about spending capital in the next year or two to get rid of these 20-year-old, , control systems you got, right? The Windows 7 machines that you still have in your environment.

[00:16:55] Dino: Standardize on your networking platform, for example, right?, Implementing, you know, firewall policies, you know, within that environment where IT has a lot of, relevant skills and knowledge and capabilities to assist, but what they're missing is what's operationally important to that plant, right?

[00:17:14] Dino: How can they do that and do those things without being disruptive, you know, , and causing headaches and/or downtime? And my experience, I say this all the time for, for IT guys, you know, for them who've already had their hand slapped or have been poked in the eye, it's like trying to pick up a porcupine.

[00:17:32] Dino: No matter what you try to do to pick it up, you're gonna get stuck. So they just learn to leave the damn thing alone, , and somehow that needs to change, right? , But that means that IT capability Because of, going back to the Purdue model, the anatomies of these plants, even if you've got 30 plants, no two are the same.

[00:17:49] Dino: To your earlier point, , those machines have been out there and deployed in different eras, right?, So you really have to understand the anatomy of the plant , and how it operates when they do line changeovers, when they do shift changes, when they do maintenance windows, you know, product swaps, whatever they might be doing, - you got to get in tune to that, right?

[00:18:07] Dino: As an IT person, to be able to implement your cybersecurity tool sets, again, without being disruptive and still get what you need out of the environment. , The area in which you work at Rockwell today , and other, , cybersecurity tools, platforms , that are geared for th- that OT environment have a lot of rich contacts and a lot of information, but I don't think IT knows how to map that, per se, to move forward, improve that cybersecurity posture, right?

[00:18:35] Dino: Which is really the end goal. How do I improve my cybersecurity posture? Mm-hmm.

[00:18:40] Ken: That's something I've talked about, on my cast before. I'm sure you have too, right? It's, and I mean, you've mentioned it a couple of times already. , You've got that old box over there.

[00:18:49] Ken: Maybe it's running Windows 7. , I almost chuckled because I've seen older, much older. , But it's hooked up to something that's, you know, making a lot of money, right? You know, that paper machine over there, well, I mean, yeah, technically , that's a Windows 98 box hooked up to it. But, you know, we got a contract with, like, two major American news magazines.

[00:19:07] Ken: This is the only paper in the world they're willing to use, you know? , Our downtime is measured in euros, , because they want their paper and it's, you know, a massive legal headache to do anything because, you know, okay, cool, yeah, that paper machine vendor, I don't think they exist anymore. , So it's not like we can just call them up and say, "Hey, do you got anything that, like, runs on Server 2022 these days?"

[00:19:33] Ken: They don't exist. It's a complete , replace capital project. That's how many months of downtime, how many months that, you know, these major magazines are not getting the paper, the only paper they're willing to print on, but they're gonna find a new supplier. , Or there's gonna be some significant legal penalties attached to that, right?

[00:19:47] Ken: , It's things like that. And then on the flip side too, you know, you talk about being sensitive to downtime. I can remember a couple of times in my Honeywell days when it was one of those cases where we had some critical changes that needed to be made. This was, like, a regulatory thing, compliance thing, they were facing fines, and we were just waiting on an outage, and you get the call and it's like, "You have 36 hours."

[00:20:13] Ken: It's like, "Dude, I'm not even in the same country- Right now, , it's 10 hours for me just to fly there and drive because nobody builds power plants or pulp mills anywhere near pleasant touristy places that have frequent flights to them.

[00:20:30] Dino: Well, and just the inherent knowledge of the information that you need to collect before you even go on site, right?

[00:20:36] Dino: To even begin tackling a problem. I mean, you know, when we're looking to set something up and we're meeting with the key, , stakeholders that are involved, and we're trying to get our hands on, you know, IP and VLAN addressing schemes. We're trying to get configurations out of switches and firewalls.

[00:20:51] Dino: We're trying to get drawings and documentation, site drawings. And back to the earlier, you know, if the plant's been around for 30 or 40 years and maybe changed hands a couple of times, that stuff gets harder and harder to find. The tribal knowledge is not there, you know? And so then you're in a huge discovery, uh, effort the first time around just because they tell you something you talked about the operator with his, his head kind of cocked to the side. I do the same when people tell me, like, remote access, for example. "Well, how do you get internet support to this?" "Well, you don't. You gotta come on site." It's like, then I open up the panel and there's a tosy box sitting in it or a Cradlepoint cellular modem, right?

[00:21:30] Dino: Some VPN concentrator. Or there's a jump box, right? Or they think because they got dual NICs in their PLCs that they're safe and secure because you can't get to the control side of what that other NIC is running on the, for the interlocks and the IO on the backside. It-- And so when you start putting it , in context like that, and it's usually with IT and OT together, , to have that conversation because it's eye-opening for IT, and it's also rec- letting OT recognize what we're trying to solve here

[00:21:59] Dino: Because this is the target, right? These control systems are the target of this security strategy that your company's decided to move forward on, and so we have to get our arms around this plant. And they may have one pulp mill in, in one on the West Coast, and then we go to another pulp mill down in the Southeast, and they're not the same.

[00:22:18] Dino: It's not like you can just - replicate what you did in one in the other. You still gotta go through that discovery phase because they don't have asset inventory in that environment. They're not really sure what kind of remote access is going on in there. They have different, , networking technologies being used in those plants, different control systems.

[00:22:33] Dino: Might be Rockwell on the West Coast and Siemens in Southeast, right? And that's where IT kinda gets lost because that's just not their model, right? Standardization, centralized approach, right? , That's not your plant floor. You know, that's just not the way it was built.

[00:22:49] Dino: And a lot of those plants run and operate independently. They have their own P&L. You know, they pick their different partners that they want working. You know, they like this particular system integrator that, you know, is from the community or local to them, you know, versus, one that might be more of , a national, where , you can get that breadth and depth across your plants to start s-standardizing and centralizing some of that stuff and put a strategy together, you know?

[00:23:12] Dino: When you go back to the Purdue model, again, the Purdue model to me is just a map at the end of the day, is really what it is. And, , it's not saying you, you have to live within that, that Purdue model. You already do. , It's just the map of your environment. I know where your IO is and stuff down at level zero and what's in level one and what's in level two and what's in level three.

[00:23:33] Dino: We us-usually don't get too concerned about what's in level four and five because that's IT's world, right? It's when you get into level three is where , it's truly an OT environment, I've yet to see any companies where they've moved any of that into the hands of an IT organization to manage and support and to strategize on how, to be successful.

[00:23:55] Dino: The business we're both in from a cybersecurity perspective, I've seen lots of OTIDS platforms, even though yours is not quite that, im-implemented, - And it's been in there for years, and you're seeing a third or less than half of your assets, you know, in your environment. You don't have an accurate asset inventory, which means you don't have an accurate understanding of the risk and vulnerabilities within your,, industrial, environment.

[00:24:21] Ken: Yeah. Well, and it does begin with that, you don't know what you have, you don't know what you need to defend.

[00:24:26] Dino: That's security 101, right? Every security framework, regardless of the industry you're in, the vertical you're in, the very first thing you have to have is an accurate, up-to-date asset inventory.

[00:24:37] Ken: And once you've got that, I mean, well then, you know, there's all sorts of stuff that becomes enabled at that point, right? You can start doing, - Risk assessments, crown jewels analysis, right? Like what are the critical systems? Can't lose these. What are the less than critical systems? Well, if I lose that, I might be okay for a day or two.

[00:24:52] Ken: , What's gonna trip the boiler in five seconds? That kind of stuff. You can, start figuring out, hey, especially if you are like in a fairly flat network, like what are the points that we can segment at? And Purdue comes in a little bit useful there as well as just like you say, a map, not so much as like, you know, the strict guidance of like, okay, well those things you must wall off as your level zero, and those must, you must wall off as your level one.

[00:25:14] Ken: But more where the communications need to go, right? The part of Purdue that I like to hang onto the most is the whole idea of one up, one down communications, right? There's never, you know, especially coming down, right, going from a level four, reaching down into the environment, you want those connections between level four and anything below it highly brokered, and really you never want them to go past level three.

[00:25:37] Ken: Or if you have a 3.5, right, that IDMZ or DMZ, whatever we're calling ,, it has many names. But, you want the communications to be tightly brokered, especially if they're flowing down into your OT environment, and to a certain degree out as well. Control communications, anything that can actually like change something within your OT environment, you want to keep that as isolated and as far down the stack as possible, and as far away from the internet as possible,

[00:26:04] Dino: , Coming from the lower levels up, we see a lot of PLCs, for example, that are trying to reach something that is outside that level three or three five. It's trying to get to four or five, or even the internet, right? In the Purdue model, I, I put the internet at level six, even though there's five levels in the Purdue model.

[00:26:23] Dino: But to me, the internet is level six. And it's not that it's getting out, but it's trying to get out, right? , It's knocking at the door, right? And when you turn around and say, "Well, why is it doing that?" The client doesn't know why that PLC is doing that. It's whoever, you know, did , the programming and development of it back in the day has left that in there, right?

[00:26:43] Dino: And it just takes a, a screw up. You know, it could be a firewall update that what was once closed is now open, a configuration change in the environment, or somebody setting up a remote access that's going around that, and all of a sudden now that PLC is able to get up and out into, you know, to be sending a signal all the way up into the internet or into the enterprise, right?

[00:27:03] Dino: , And we see that all the time. That is a regular con- occurrence that we find it devices in the levels two and one, three trying to get out of the box they're in, the box they're supposed to be in, right?

[00:27:19] Ken: Right. And all of a sudden you're on Shodan. Oh, dear

[00:27:23] Dino: Yeah. , And we've seen that too, right?

[00:27:24] Dino: An IT leader who should be concerned about that, how do they get their arms wrapped around that, right? If they're not gonna work with the OT people and pore through that information, or the integrators and the OEMs who also are playing a role in those plans, right?

[00:27:39] Dino: That's the third leg of the stool. You got IT, you got OT, and then you got those outsides and OEMs that work in your plants every day, right? Remote access, people on site, you know, they got a role to play to maintain and support, , or do startups, whatever the case may be, and they're in your plant. You're not really tracking what they're doing. , And what they have is their work, , book that they're, their laptop that they're using to communicate with that stuff, right? They bring in their own stuff. If I go to 10 plants, eight of them are gonna let us walk in with our own laptops and plug it into their environment.

[00:28:11] Dino: Eight in 10, right? Because they're not in, they're not in a position to hand off computers to people that are showing in, sh- that are showing up to do work in their environment to make sure that it's got the right, you know, EDR package on it and the right services that are supposed to be on it, and it's been scanned for malware and all of that.

[00:28:31] Dino: It just doesn't happen. And it's an expensive proposition for them, 'cause, I used to work for a very large adult, uh, beverage manufacturer, and that's what we did. You know? You come up to our, plant, you're gonna, we're gonna hand you a, a laptop, and this is the one you're gonna use. But then we had to keep expensive software on there.

[00:28:48] Dino: We had to keep Studio 5000 software up to date and on those machines, right? That stuff isn't cheap. That dynamic , is something, again, I think IT misses on., They wouldn't let somebody do that in their environment if they went in to go work in their data center, right?

[00:29:03] Dino: They're just not gonna let some contractor or consultant walk in and plug into their data center environment right onto their network. They don't allow that. You would hope not, right? It's not a, it's definitely not a good practice. But in the industrial space, we find that to be true a lot. So

[00:29:18] Ken: commonplace.

[00:29:19] Dino: Very commonplace.

[00:29:19] Ken: I've seen it live. I've seen it live. Mm-hmm. I... Yeah.

[00:29:22] Dino: We do it today. That's the business we're in. We do it today, and I know probably the Rockwell field guys that go out there and work in plants too probably do the same thing, you know, for the clients that they're supporting.

[00:29:33] Dino: Again, back, circling back around to the Purdue model, it's just a map of where things sit within your plant, right? Generally speaking. , And so by then, if you can start isolating by whether you want it at the level, the cell, the zone, the line, , and start building out your micro-segmentation, have something that is, using your words, brokering that conversation between whatever it's talking to within the environment to make sure that it's okay versus things that it's talking to that's outside that zone.

[00:30:03] Dino: You know? Just it could be even the same plant. Could be level three to level three, but it's from packaging to brewing, you know? And why does my, control systems need to talk to the brewhouse, right? Maybe the filler does, you know, to go to the release or to utilities to call for steam or hot water to clean the lines, whatever the case may be.

[00:30:26] Dino: But map those out, and then make sure you have a mechanism in place to make sure that they stay in their lane.

[00:30:32] Ken: Yeah. Versus yeah, just, you know-

[00:30:35] Dino: Willy-nilly ...

[00:30:36] Ken: wild communications. Right. Yeah.

[00:30:37] Dino: Yeah. So again, Purdue map, Purdue model, it has its place. , Until you get to, I guess, a fully autonomous AI world , that's working flawlessly.

[00:30:49] Dino: Again, back to my Waymo, even though I did catch a good story the other day that... ', 'Cause the Waymo thing's been popping in my head since I used one in Atlanta, , last week. , Was a g- there was a guy who put his luggage in the trunk of that Waymo, and then, got to his destination, I guess the airport, and got out and was trying to open the trunk to get his luggage out, and the car drove off with his luggage in

[00:31:12] Ken: it.

[00:31:12] Ken: Ah. Yeah. Oh, sorry, sir, your session has ended. I'm on to my next customer now.

[00:31:19] Dino: Again, there- there's still goofy things that can happen, but that seems like, okay, I don't wanna call it harmless 'cause, , , the passenger obviously it was problematic , for him. But on a plant floor, downtime at 100 grand an hour, or damaging a piece of equipment, or having to sewer, , a batch of beer or destroy product, because of things just didn't work out right, no human in the loop, , to catch it before it got too far out of control.

[00:31:46] Ken: It's amazing how quickly those sorts of errors add up to more than anybody who could be deemed responsible makes in a year.

[00:31:54] Dino: Right. Exactly, right. , My position is I don't think IT wants to own that, Ken. That's my belief, is

[00:32:01] Dino: They don't wanna be on the hook for things like that. They just don't. Right? Right or wrong or different, I just, I don't see it. 'Cause otherwise my phone would be ringing every day from IT leadership wanting to know what it is that we're doing for them in their manufacturing facility. I got 30 plants.

[00:32:17] Dino: You're working in six of them today. You got 15 people out there working in these three plants doing startups, doing maintenance and support. What are you guys doing? How are you doing it? You know, what tools are you using? , What's , the activity and tasks that you're being asked to do by my, company, right?

[00:32:32] Dino: So I can get my arms around this as I'm trying to build out my cybersecurity strategy, to me, that's the miss. Right now I see that as the miss, and I don't know why. , I know Rockwell's been doing a, a better job of,, getting to IT leadership, you know, to have these conversations, you know.

[00:32:49] Dino: But those are the big companies with deep pockets that really are trying to, , do the right thing and, and get involved and engage, but that's such a small percentage of , the overall manufacturers that are out there.

[00:33:00] Ken: When I was at that pipeline company I mentioned, , they had no OT cybersecurity program whatsoever.

[00:33:05] Ken: 'Cause I'd come from Honeywell ICS role. Technically I was not in an ICS-facing role at this pipeline company, but I just started building what I could, right? It's like, "Oh, here's an old server. Well, you know what?, I'll throw something on here, and I can at least start, like, inventorying the network and watching for stuff," right?

[00:33:19] Ken: Just do what you can, right? Here's an old workstation. Well, maybe I'll, like, get licenses for three different antivirus softwares and then, like, you know, if someone wants to bring media into the environment, cool, you gotta pass all my tests, right? Just lives on my desk.

[00:33:32] Ken: , Do what you can. But eventually they brought in big outside consultancy to build them a company-wide cybersecurity program. And they were about six months into that process, and all of a sudden the consultants start showing up with, like, clipboards and taking notes in the, , control system and started putting, like, tape on the servers.

[00:33:51] Ken: Like, "Okay, well, we need to upgrade this, we need to upgrade this, we need to upgrade this." And, you know, I was doing some switch maintenance. It's like, "What are you doing? What's going on here?" And they explained it, and I'm all for those kinds of programs, but obviously they were coming in with absolutely no context, right?

[00:34:07] Ken: You can't just be like, "Oh, well, here's this server. We should upgrade it to, like, the latest and greatest supported model." Uh, that runs two of our lines. Two of our production lines are running on that server right now. We're not going to be upgrading that. But then it was like, okay, well, shoot, I gotta get OT a seat at this table.

[00:34:23] Ken: And at the end, I mean, I think they did find a good balance between, you know, what was IT's responsibility and what e- what skill sets they needed to ensure they had on the OT side to supplement or complement what IT would also be assuming responsibility for. But a lot of times it just comes down to communicating those realities, right?

[00:34:42] Ken: The, the complexities of OT, the roadblocks that exist to just doing things in an IT way. even if it is something as simple as, you know, upgrading your servers every few years or upgrading your desktop PCs every few years or pushing all the updates that you can, , when you can.

[00:35:02] Dino: Well, I know you've heard this term.

[00:35:03] Dino: It's think in global, but you gotta act local, right? And again, if you got 30 plants, you, you-- it's a resource challenge, right, internally. Because that, depending on how fast you wanna move in this journey, will determine how many resources you need to be able to execute that plan, right? Um, because you need people that are, that are close to the process, who really understand how that plant operates, and are committed and willing to do that, right?

[00:35:30] Dino: And so now 30 plants, it's like, okay, well, you might need, you know, 25, 30 some odd people working on this instead of just two, right? Unless you plan on spending five to 10 years doing it, right? And by the time, you know, that time rolls around, it, it's a whole 'nother set of problems that you got, you know, versus I wanna get this done in the next 18 months.

[00:35:51] Dino: Well, you're gonna need an army and, and you're not gonna get that internally, right? Those organizations that self-perform, which I know you and I talked about in Milwaukee, that, that's a heavy lift. A heavy lift. And, and regardless of the technology, to be able to do it right and be consistent and stay on top of it and, and, and, uh, turn, um, you know, and be successful, you, you got a resource issue.

[00:36:17] Dino: Which where I, again, flip around, it's like, why aren't you leveraging the system integrators to help you with that, right? 'Cause a lot of them have practices. They have the partnerships with these technology companies. They've been working in your plants for years, decades, right? They got, they got a lot, they got-- they know a lot, right?

[00:36:33] Dino: And, and tap into that.

[00:36:36] Ken: It's, I, I mean, it seems like the best way forward. And, and maybe that's, you know, why it's a really, really great thing that, you know, like, your group and some of the others that were there at that training, right? Like, we're actually now getting our product line, right, the SecurOT product line, asset management and inventory, right?

[00:36:53] Ken: Find what you have so you know what you need to defend. We're starting to get that now into SI partner channels, and that could be a really, really good thing. But you are exactly right. Like, these are the people who are, you know, they have almost that same level of plant familiarity as Mr. I Can Listen for 15 PSI Variations.

[00:37:11] Ken: Um, but they understand the technological side, and it comes back then to that interpretation layer, right? They're the ones who can speak to the IT side of the house and take what the IT side of the house says and speak to it in plant operator on the OT side, um, to people who are, you know, from the highly technical to the people who are very process-wise, but not necessarily, you know, computer savvy.

[00:37:39] Dino: Well, yeah, organiz- yeah, organizations like ours are a trusted entity to that OT organization. I mean, you're, you're not there if you're, if you're doing a bad job, right? You're, you're just not gonna be allowed. They're not gonna ask you back. They're not gonna let, let you get involved. If you, if you're, um, a lousy integrator, you're very disruptive in the environment or your, your technical prowess is way down on the scale, you're not gonna be in their work in any way.

[00:38:04] Dino: Right? And so to, to, to the point, if you really wanna be successful, and you wanna get that asset inventory, and you wanna get your risk and vulnerabilities well understood, and you wanna start de- forming a plan on how you wanna execute this, you're gonna have to tap into that resource, so that third leg of the stool.

[00:38:29] Ken: Yeah, we are. Um, so I don't know if all of y'all out there... I mean, if you got hair the colors of mine and Dino's, you might remember, like, some of those old cop shows, uh, where they would do, like, the crossover, crossover episode. You know, the, the whole plot would start on show A, and then you'd have to, like, tune in to show B the week following to, to finish out the storyline.

[00:38:50] Ken: So we're trying to do something like that here. So, um, yeah, Dino will be... Well, once this episode goes live, uh, Dino will also then be joining my crew over on the OT After Hours podcast. Little, uh, passion project I've been running out of Rockwell for a couple of years now. And we're gonna keep this conversation going because, you know, clearly there's a lot more here to unpack.

[00:39:18] Ken: Um-

[00:39:20] Dino: Yeah. And quite, you know, we- I don't know how many, how many podcasts are actually on j- at, at the level that we're, we're talking about. I mean, there's consulting related, you know, the big broad brush stroke of you... It's like the consulting companies. This is what you should do, and you spend a lot of money to get a, um, a, not even a playbook, but just, yeah, you need to find an OTIDS, fill in the blank, um, to help you get a better idea of how, how to improve your cybersecurity posture, right?

[00:39:52] Dino: The question then is how do you get that spinning tire on the road, right? Because a lot of companies out there, they'll, they'll sell you all the parts to build a car, but if you don't know how to build a car, you're not gonna get... You're not gonna have transportation, right? And that's what I've witnessed with a lot of clients.

[00:40:10] Dino: They try to self-perform, bought these tools through IT channels, generally speaking, and, uh, and um, and the stuff just, it either never gets out of the box 'cause the plant won't implement it, and they don't know how to force it down the plant's throat. Or they'll get it into the plant and they'll capture 20, 30, 40% of their assets maybe in their platform

[00:40:33] Ken: Well, yeah, 'cause if you're trying to do it bespoke-like or, you know, with, shall we say, less than complete buy-in, um, th- there's just gonna be areas that you're just not allowed to wander into, right?

[00:40:43] Ken: Whether it's, you know, "Oh, you can't, you know, connect to that switch and put a tap port on it," or, you know, yeah, you're not deploying agents. Um, it's, it's just not gonna happen.

[00:40:54] Dino: Right. Yeah, if you don't speak their language, they're, they will... I- if you show up without the proper PPE, I mean, take your pick, they're gonna put you in a box, and you're gonna stay in the box until you decide to leave the plant, right?

[00:41:08] Ken: That's the joke I always used to make. It's like... A- and that's a hard ask too, right? Because now you're adding a whole 'nother layer. Now I need somebody who's highly technical, understands the plant intimately, but also can show up at the plant and, you know, or at least, you know... Well, I mean, I- with an SI, you've been working on the plant for years, so hopefully you've already built that relationship, right?

[00:41:31] Ken: But imagine someone coming in from the outside, right? You gotta show up at this plant, you gotta be able to speak the language, you gotta be highly technical, but you also have to be enough of a people person that you can sit down with that plant operator and, you know, within about a day convince him that you should be allowed to come back the next day and have significant levels of access to his kingdom.

[00:41:53] Ken: Um, that's a... It's not easy to find people that check all those boxes, you know? You can find highly technical people. Yeah. You can find highly technical people. You cannot necessarily find highly technical people who can think on their feet and are, like, kinda outgoing and conversational. That's a different thing entirely.

[00:42:16] Dino: Well, you know, a lot of times we'll go into the plant and the IT guys are the ones that are initiating it, which is great. They're the ones that crack the door open to have the conversation. But as soon as we get on the plant floor, Ken, and get ready to go, you know, we put the uniform on, as I call it, we show up with the uniform on and getting ready to go out on the plant floor, that's when the IT guy is like, "Yeah, no, I'm not going on the, on the tour.

[00:42:35] Dino: I'm not going out there and crawling through that plant for the next two or three days." They just, you know, that's just not something they wanna do, right? They, they'll work on the network. Again, the network that they can control and manage, it's their responsibility, m- firewall, network. Um, but after that, everything from that IDF down into the control, again, Purdue Model, anything at level three and down, they're out.

[00:42:58] Dino: They're like, "Yeah, you guys have at it. You work with the electricians, and the operator, maintenance guys, and engineers. You got, you, you're gonna have to go deal with them to get the things you want." Which is fine for us. We're, we're good with that, right? You know, that's not a problem 'cause we do that every day anyway.

[00:43:12] Dino: Um, but for them it's like, "Yeah, I have no interest in, in doing that. I didn't bring my safety shoes today," you know?

[00:43:19] Ken: Must be nice. To not have to always travel with, uh, checked baggage 'cause I've got, you know, two pairs of steel toes and all my arc flash gear, yeah.

[00:43:29] Dino: Yeah, I had to laugh at Jessica when we were leaving, um, after that training, um, because she was, she was on a startup in, in Vegas at a food plant, and she flew from there to Milwaukee.

[00:43:39] Dino: And when we were leaving to go, because we would travel together to get to the airport, she had her hard hat, you know, with a carabiner connected to her backpack. You know, because she had her safety shoes in there, she had her hard hat with her, you know. And that's how we travel when we go, you know.

[00:43:54] Dino: Because I don't want to, I don't want to ask the plant for anything, you know, because some plant managers or the, or the engineering team's like, "You don't have your own stuff," they're not going to give it to you, so you're stuck. You know? You can stay up on the... You can stay on the mezzanine or stay up there where the tourists hang out, but you're not coming down here.

[00:44:14] Dino: Right.

[00:44:14] Ken: You can't see it, it's just off camera over here, but there's two suitcases and two backpacks, and it's basically like, you know, yeah. It's what kind of... It, it, it comes down to like what kind of trip is this, right? So like for Milwaukee, cool, I will take the backpack, like the, the smaller backpack and the smaller suitcase because, well, I'm just in the office, I don't, you know, I don't care.

[00:44:33] Ken: Um, but the other one is for, yeah, that's where the PPE's already tucked away ready to go, the extra pair of steel toes, the bigger backpack that holds the extra gear. Um, you know, and it's, yeah, like Luray said in the chat there, it's just how you roll because where are you ending up, right? Um, but being cognizant of that fact and being able to show up looking like you belong there can go a long way.

[00:44:58] Dino: It, it definitely goes a long way with those OT guys.

[00:45:01] Ken: Bringing them treats also helps. Find out, find out what they like and, and bring them a little bit of that, you know, the next day. That's, it's, it's good.

[00:45:09] Dino: Well, the other thing, you know, the... And plant people are hard to get out of the plant. You know, you just go on lunch or whatever bec- because it's, it takes 30 minutes to get out of the plant, wherever it is you might be, to get...

[00:45:19] Dino: And then 30 minutes to go someplace to eat because the plant's out in the middle of nowhere, and they can't be gone for two hours, you know? So they, they... So I ca- I, I eat cafeterias and vending machines a lot at the plant.

[00:45:34] Ken: I, uh, I think the best rapport I ever had with a plant ops guy, uh... And it was a long project but, you know, it was about Thursday of the first week and he's, uh, he's walking us out and, uh, he sees the rental car I'm driving.

[00:45:57] Ken: And it's a, I think it was like a Ford Edge, something like that, right? One of the, one of the earlier Ford crossovers. And he's looking at it and he's like, "Hey, wh- which hotel you staying at?" I'm like, "Uh, it's this one." He's like, "Okay, I'll be there tomorrow morning. Uh, I'll meet you outside. Can I drive this thing?"

[00:46:18] Ken: Um, I probably shouldn't have let him because I don't think my employer's insurance... This was pre Rockwell, so I figured statute of limitations probably fine on it. He didn't get into an accident anyways. But, um, you know- He had a daughter, and she just had a kid. And he was looking to set her up with, you know, a vehicle.

[00:46:35] Ken: And he liked the look of the Edge. It was kind of low, it was wide, it handled well on the road. And, you know, so he's just like, "Can I take this a spin?" So I let him drive us to the plant on, uh, Friday morning. Um, but, you know, just things like that. You know, we also figured out which restaurant in town was his favorite, so we went and had dinner there one night, and then we had way too much leftovers, so we just brought him a, you know, uh, brought him lunch the next day.

[00:47:02] Ken: Um, this was like this big Asian buffet place. Or not buffet, but like big Asian place, uh, just order large trays of food to your table. But just things like that, right? Just, you know, finding those little human elements. And like you say, yeah, they can't always get off plant, but, you know, if you can bring a little humanity back to them, um, it's, it goes a long way.

[00:47:23] Dino: Yeah, it does. All right. Well, um, so we'll meet up again, and this time we're gonna flip the script, right? , I'm gonna head over to your podcast, right? So, for our listeners, you know, please,, review the show notes, and we'll catch you next time.

[00:47:37] Intro/Outro: Thanks for tuning in to the Industrial Cybersecurity Insider.

[00:47:41] Intro/Outro: To stay up to date with our latest episodes, be sure to click the follow or subscribe button now. And if you found this podcast helpful or have a topic you'd like us to discuss, please leave us a review or let us know. Thanks again for listening. See you next time.