What Actually Works in OT Vulnerability Management with Dan Cartmill, TXOne Networks

CSI - Ep.96 - Dan Cartmill ad: [00:00:00] Welcome to the Industrial Cybersecurity Insider. In each episode, we dive into the world of industrial cybersecurity. Join us as we cover the latest trends, innovations, and practical insights as we talk with leaders and practitioners across the industry. Gear up and let's get into this week's episode. host: Hi, my name's Dina Bki, and today I have Dan Cartmel from TX one. Welcome, Dan. guest: Thanks, Dino. Great to be back. Good to chat again, looking forward to chatting today about some vulnerability management stuff. host: Yeah. , Why don't you give us kind of a, a brief rundown of your background in your role at TX one. guest: I do, , product and content marketing here at TX one. My background, I, I took a little bit of a different path, I guess, to get to where I am now. I actually started on the [00:01:00] practitioner side about 17 odd years ago, , in the uh,, the IT security space. And, , did that for a while, held a, a number of different roles as a security practitioner, security analyst moving into, , consulting, and then I moved to the vendor side. guest: I did , some pre-sales work, product management, and now moved into product marketing, and that's what I do for TX one. I help describe how TX one's products can help with, the challenges that organizations are facing in the market. host: TX one's got a very unique offering with the technology, I believe. host: I mean, we've been looking and chasing in our role, , putting in a lot of these OT intrusion detection systems, right? Which was kind of the first phase of this, you know, OT cybersecurity journey. , And in TX one coming along several years ago. Has been able to say, yeah, we get that and we're a part of that, but then, well, how are you going to deal with these vulnerabilities and risk as you uncover them in the, , manufacturing environment around OTs specifically.[00:02:00] host: So what do you think is one of the biggest misconceptions about OT vulnerability management? guest: Yeah. It's the perfect starting point , and your setup there , is perfect for it. , I'm often seen a misconception between. Gathering that, , list of vulnerabilities, doing a comprehensive scan, even if you're able to scan every asset, , that somehow that is the end goal. guest: , That once you have that list, once you have that good understanding of, , what vulnerabilities are there, what exposures you have, what , your risk surface is that your significant amount of the way, , complete on your OT security program. When in fact you've just created lists, there's still a lot of work to do and , the real work of actually reducing that risk is, , still to be done. guest: And we see a lot of organizations that have, , deployed various tools and they do a great job at helping identify it, but they find themselves stuck in that gap of, what have we got, what are we gonna do about it? host: Well, [00:03:00] I think a lot of mistakes that organizations make is , they think that OT environments can be treated the same as it, right? host: And when it comes to vulnerability management, and, you know, the reality is that, , they run on legacy systems. They have uptime requirements, you know, , and limited scanning and patching. So, and, and they're sensitive to changes. And that's where we, we see this, you know, the, , gap as I call it, between IT and ot. host: And we can get into the whole IT OT convergence conversation as, as we go through this discussion. Mm-hmm. And our, and our thoughts on that. So, and, what do you think are some of the blind spots on, , OT vulnerability scanning? guest: Yeah, so, so OT vulnerability management as a whole? It's just less mature than, its it counterparts. guest: , So in terms of the technologies that exist to be able to, , to go out and scan various devices , in different, , deployment patterns and whatnot, it's just less mature. It hasn't been around as long, there's not as many vendors in the market kind of driving a lot of that [00:04:00] innovation and whatnot that moves technology forward. guest: So from a starting point, it's just less mature. , Another, I think, significant part behind that is. , How we actually do the scanning. So no, there's the ongoing debate of passive versus active scanning , and all of that. But how do you make sure that all of the devices that exist in your environment, , and whether or not they're connected all the time, sometimes they might be intermittently connected, sometimes there might be air gap. guest: How do you make sure that the approach that you're taking. Actually collects all of that information as it's doing. Its, let's call it a sweep. So the blind spot, I think that comes from that is the more perhaps it-like approach of, we expect these systems to be generally online and responding to that, , that scan, that sweep. guest: When in ot, a lot of the times, either that device isn't going to respond for various reasons, it's not able to, 'cause it's not [00:05:00] connected. What that can give you is. An incomplete picture, I'd say, where at a point in time you may have 60% of your assets, you scan again, the next week, you might get another 60% coverage, but it's a different 60% because different devices might be online or available, some might be, , bought online for a specific part of a process that's happening. guest: , and that means that you're gonna have outdated vulnerability information from the devices that didn't get re-scanned. You're gonna have new information that you are gonna have to try and, , consume and work out what to do with. So I think , the big blind spot is treating it like OT is you, your picture is never going to be complete guest: there's still too many gaps, immaturity, and tools in their ability to see all these devices. So, , there's more steps required, more due diligence required in making sure that you do have all of the information so you can make your decisions on, , what you're gonna do about that. host: Yeah, you're spot on. host: The [00:06:00] OT assets are not easily discoverable, that's for sure. Right. It's just their architectures around those control systems are, are different and you know, it's also an area where it is, it's a blind spot for them. 'cause they're truly unsure of how that stuff is put together within those environments because they didn't have a hand in it primarily. host: Right. And nobody's. Nobody's wrong here. It's just if you weren't involved with the development, design, implementation and then the ongoing maintenance and support of these systems, it creates a blind spot for it. It really, really does. And we have this saying, it's like you can think globally, but you gotta act locally, right? host: Meaning that you have to be able to. To work with people within the four walls of these facilities or at these sites that have inherent knowledge, right? The tribal knowledge of these control systems and how they operate, how they work, potentially, how they're put together. And even sometimes the plant people aren't really sure how everything is put together. host: They know where it's at. They don't necessarily know how it's connected or where it's connected. And we run into that a [00:07:00] lot in the work that we do. , So let's talk about host: what's the finding vulnerabilities different between OT and it, you know, what makes those different? guest: Yeah, so, so perhaps to level set how a lot of these, , a lot of tools that do this kind of scanning work is the vulnerability has to be known. So, so there has to be some kind of intelligence, some kind of detection, logic filter, something like that, that, , can, , identify, scan a particular device. guest: Run a set of, detections for, lack of a better word, against that, to understand what is it vulnerable to? Is it vulnerable to these particular exploits? , Does it have this missing software that would make it vulnerable to these other ones that are listed? , But it all comes back to it has to be known. guest: , So in our IT list, of course we have a, a ginormous list that's continuing to grow all the time, but it's, it's very large in what we know. Exists out there in terms of vulnerabilities. OT is just smaller. So there's been less [00:08:00] research done on these vulnerabilities that exist. There's less researchers, less threat researchers that are doing active investigations into, , the vulnerabilities that exist. guest: So, ,, for the first part, it is it's more difficult to do ot, vulnerability detection, vulnerability management, because there is less. , Threat intelligence around what are the vulnerabilities. So the result of that is even if you were able to do a complete scan, , you had a, an excellent tool that was able to do a complete scan of your environment. guest: Every device, let's just assume for a moment that that was possible to do. There's still a fairly good chance that there's a whole bunch of vulnerabilities that it just doesn't know about, that we just don't know about yet. They're just waiting for someone to exploit them. So in terms of finding. guest: Those vulnerabilities. , Our initial mindset should be, it's unlikely that we're going to have the complete picture. We're not gonna have as granular knowledge and level of protection as we might have in that IT environment, simply because [00:09:00] there's not as much research there how to. Kind of move toward doing something about that though. guest: So it's, it's one thing to say, yeah, you're gonna have less visibility. Yeah. And they all know they happen. host: Right. The clients know that they've got risk with these assets. They, they, they guest: know that host: Exactly. E guest: exactly. , The way to, to move forward with that knowledge, with that acceptance that there is going to be a good level of unknown is rather than. guest: Rather than act purely reactionary firefighting mode when, , say a new vulnerability is announced, , these devices are vulnerable. It's a, priority, , 10 have to go and take some kind of action. And, and jumping to the response there, 'cause that type of announcement is only going to increase as more research is done Right. guest: As we see it in the IT world where every couple of days there's a new high criticality one that needs to be done. Mm-hmm. , That's a pretty quick way to just drain your available resources, right? Is just jumping through and, and trying to go and do , more scans to identify what is vulnerable [00:10:00] instead, taking a more proactive approach. guest: And, , that comes from understanding more about the environment, as you were saying, getting that plant knowledge of how these, , systems are put together, what they connect to. But, , instead of looking at individual vulnerabilities, look at the classes of vulnerabilities that exist. So, an easy one to visualize, cast our mind back just a couple of years, eternal blue. guest: , So we know that, , that worm went through, , SMBV one. So instead of trying to find all the devices that are vulnerable two, that a more proactive approach to that would be to simply disable SMBB one. Not saying that that's necessarily possible inside of ot, just as the example, but that blocks the entire class of attacks. guest: So all the, , derivatives that came from that or used eternal blue, you have, , a mitigating control in place already. So that can, help you focus your efforts and the limited tools and resources that are going to be available to block [00:11:00] very large. Portions of the overall, , vulnerabilities and exploits that exist without having to look at each individual one and knowing that when the new one comes, there's a good chance you might already have some good protection there, buying you some more time to do that analysis with all the other work you have on your plate. host: No, good point. And we're gonna talk, you know, specifically about how TX one's tackling that problem. 'cause some of the other things that I would add is, you know, the OT teams, they can't afford taking downtime. To take on , these updates and these patches. , There could be regulatory restrictions that are involved. host: It could be vendor lock in, you know, where the OEM's telling you, Hey, you can't touch my machine if you want to, you know, if we wanna keep your warranties, maintenance and support in place, you know, and so outside of. The risk, the fact that the vulnerability is out there, it's a 10, it's being exploited. host: They still can't move quick enough to do anything about it. And, , because of the things that I just mentioned. So that being said, let's talk about how TX one. The technology that they bring [00:12:00] to the table can actually help solve some of those problems. So how does TX one look at going after these vulnerabilities we're talking about without being disruptive into the environment? guest: Yeah. So, , there's a couple of different ways and, and we do it slightly different between some of our products. It's the same philosophical approach that we take. So, , in edge, our network security product, , we use virtual patching is what we refer to it as. Mm-hmm. , There's a number of vendors that provide these kind of technologies. guest: It's, a time tested, very robust system, , that's just very good at what it does. So the way that that works is essentially for network exploitable, network detectable, I should say, network detectable exploits, , using deep packet inspection, various other inspection capabilities. It looks at all of the traffic going through. guest: If it matches that, , particular exploit, it simply holds the last packet. So legitimate traffic continues to pass through the device, but if it is an exploit, it's blocked. , It is as simple as that. So [00:13:00] if, , if that particular vulnerability, if the exploit for that particular vulnerability can be detected via the network, virtual patching allows you to put that patch. guest: In place off of the device so you can offload it from the device itself. That's virtual patching in a nutshell. For, uh,, the endpoint. So Stellar takes on Stellar our endpoint agent based product. So this, , uses a slightly different approach. So because it's an agent, it's based on the actual device itself. guest: It's more, , app lockdown based. So, , it does a baseline scan of that particular device. The endpoint that's hosting that agent does a scan of that understands what does it normally do, what applications generally run. What network connections is it making? What system IO is occurring on it, and it creates that baseline pattern. guest: So then instead of specifically looking for security, , events and taking action based on a security event, [00:14:00] detecting a threat, , patent filters, things like that, it instead detects any deviations from that baseline. So we have the known state of what this device should do. Just don't let it do anything else. guest: So by doing that, it can prevent those vulnerabilities, whether they're known or unknown from being exploited in the first place. So,, this is TX one's, , philosophy. So the operations first philosophy, that's, applied in both of these products where. Security. , The security perspective is very important and that can help us find targeted threats of more advanced things. guest: But the operation perspective is equally important, where the deterministic nature many of these actual industrial processes, by their very design, they are far more deterministic in nature. If we can accurately map that, we can restrict. The network activity, the endpoint activity to just that. guest: [00:15:00] So, it's more taking the approach of eliminating process variations and special process variations as opposed to detecting security threats alone. That's how we, look at , the Prevention aspect of it. , To circle back to your original question, how do we deal with vulnerabilities in this? guest: How do we identify and detect them? So, well, there's host: two things to this and there's that. There's detect and, and it gets back organizationally who is best suited to really, to do this right, to be able to map that out and make sure that you're not being disruptive by putting these technologies in line. host: Right. Able to, , you know, to provide the vulnerability or virtual patching, if you will, and not be disruptive, right. To make sure you're not breaking anything, you know, and safe operations process integrity, as you mentioned, you know, organizationally. And what have you witnessed with the TX one rollouts on who's best at handling the deployment of your, your technology guest: on the [00:16:00] plant floor? guest: Yeah, that's a really, really good question. , The answer not, not so simple because I think it comes down to, a few different parts. So it should be joint. Both teams should, , should be having input, but it's, it's more in the realm of what a security engineering team would do as opposed to what, , someone on the plant floor may do in their normal day-to-day job. guest: It's more related to the security team, but it has to be informed by the restrictions and guardrails. From the actual operations side of it. But that said, , there's always the risk of, , inputting a policy, applying some kind of a configuration that causes unintended side effects. So the technology itself, the actual tools themselves have to be well designed to limit that just by their very design itself. guest: So making sure that, , they have the ability to do bypass, , if power is interrupted or a configuration change pauses, [00:17:00] scanning that the actual traffic itself doesn't get interrupted for network based appliance. And then, , inside of an endpoint agent, and we can use Stellars as the example there. guest: It has inside of the, the console the ability to override it's, , like an emergency stop button on the stellar agent to make sure that. If something time critical, safety critical, operation critical is happening, the person right in front of that particular device doesn't have to pick up the phone, try to get to the security person to get them to revert a policy or change something. guest: They can stop it right then and there. So it's about having the products and tools designed to support the way that the operation. Needs to function, , with appropriate controls around that so it's not, , abused and whatnot. But it's a combination of both. It probably does sit more with the security team, but it needs to be well-informed by the operations team really working very closely together and well-designed products to make sure that as many things that can go [00:18:00] wrong are removed before they could go wrong. host: Most of these control systems aren't designed with security in mind. Safety is security not so much. Right. And so now we're talking about, you know, how do you get that local plant team? Knowledgeable enough to know what's normal. You know, it's like a control system within a control system with security wrapped around it, right? host: And having multiple players potentially involved, which we see with firewalls, right? We see that with firewalls and we see that with networking where you have those external parties, , that have the ability to. Manipulate, the flow of traffic and communications amongst those control systems, which to your point, disruptive can cause, you know, loss of product could be safety issues, people can get hurt, get killed, you could have damage, whatever it might be. host: And so that relationship has to be really tight between those two groups. , And you get into that ITOT convergence conversation and there's usually still gaps. Right, because the IT team, that [00:19:00] InfoSec team, that we want them to be more inherently knowledgeable of the processes, right? Mm-hmm. host: Within that plan. , And that takes a different level of knowledge , and skill within that manufacturing facility. Even if you have 30 plants in your fleet, no two are the same necessarily. Mm-hmm. Even though you might be making the same products, but you have different control systems that could be of different, , versions, different makes and models of automation technology out there that, that are being used to make those same products. host: So that local team, in my view. Has to have a role in being able to really pull this off, in my perspective. guest: I absolutely agree. And, , as you were describing that, something popped into my mind that the situation that, a lot of security teams, it, security teams find themselves in at the moment. guest: It is, , the shoes on the other foot, the analogy I'm looking for there. Where for, and, and I was guilty of this when I was a practitioner. I'll admit that. We wish that the organization knew more about security, knew more of why we were doing things, why that control was [00:20:00] in place, and why they shouldn't try to work around it. guest: We wish that they knew more about that and we got frustrated when they didn't and they worked around the control, caused an issue, clicked that link in the email, whatever the case may be. What we're seeing now, and as you just described. It's, , the, IT security team now needs to learn more about the process. host: Mm-hmm. guest: So they're being asked to do what they were asking others to do. Right. And, , it's not an easy journey to make it is a completely new domain of knowledge to learn. It is taking you outside of your comfort zone, right? So you don't get to IT security and high positions in IT security without having been in the industry for a while. guest: You've kind of maybe a little ego kind of coming into that, and making that transition and, admitting and saying, you know what? I don't know enough. I don't know enough about that. I need to go and seek out that information. Of course, they're, , a lot, most IT security folks, very inquisitive people, just by their nature. guest: But it is [00:21:00] a, significant step to take to say, I don't wanna start doing things here until I've learned more about it. I'm gonna have to go and be vulnerable with these other teams of experts essentially saying, Hey, I don't know what I'm doing here. I need you to teach me. host: One of the things that we've witnessed is that I'll say two things. host: I'm a believer that, you know, a CIO or ciso, the InfoSec teams need to probably get a better handle on who's working in their plants, right? Because in any major manufacturer has got several or dozens of different OEMs and system integrators working in their manufacturing facilities at any given time, right? host: And so they need to start thinking about building a relationship with those groups. , And I'll tell you why here in a second. , You know, because. , How else are they going to know what the, overall, , OT cybersecurity strategy is that the organization's trying to pursue? How do you get , that message into these, third parties that are in there working in your plant? host: And so that's one, right? They [00:22:00] need to build relationships with a lot of those folks because a lot of manufacturers today, from a controls perspective don't have controls engineers. They have process, they have operations, but they're not the ones necessarily programming those control systems per se. host: They're relying on these third party engineering companies or OEMs to do that. And so that's where that relationship really needs to get tighter between IT leadership from a CISO role and an InfoSec role. The partners that they have to determine what are your capabilities, what partners are you working with? host: Like is TX one, you know, part of your partner community? What practices do you have from an OT cybersecurity perspective as an engineering company? You know, so you can leverage that within these environments to help close this gap that you and I are talking about. guest: , What just popped into my mind there is in terms of finding that and the, and then. guest: Finding, understanding if you have all of the information.. So we can map out a, large list of, , all of those that we're [00:23:00] integrating with our third party providers, the different technologies tools we have, who is looking after what particular parts. A great way , to test that is to do things like tabletop exercises. guest: Mm-hmm. So develop out plans on, , okay. Well we want to. Do a simulation for this particular system, our, , inventory management system going offline. Do we know everyone that we should be calling? , Do we know who their backup is when they're not available? Do we know what systems this is integrated in, or if the loss of that system will cause impacts in other systems? guest: Mm-hmm. So, , I mean that, that comes up all the time. There's a. A big auto manufacturer dealing with, , something probably similar to that where it wasn't necessarily their OT systems that were impacted, but the IT systems that may not be connected to it were impacted and it caused a flow on effect, right? guest: Making sure that all of that is mapped and being able to have those, . Plans disaster and recovery and BC plans in [00:24:00] place. It's really important. , Tabletop exercise as well. They may seem a little goofy, especially the first time you're doing it. If you're doing a good role play of it, they can be really good at finding the gaps in your knowledge and helping you patch that and really get a good security program that's battle tested in place. guest: So, , when the time comes, not so much if, but when the time comes, you really know , what to do about it and you're not. Running around with your head on fire trying to work out who do you call first? host: Yeah, it's interesting incident response. You know, we work in manufacturing facilities every day, all day. host: It's what we do. And I, I can't think of any client that's actually said, here's our IR plan. If something of a material breach was to unfold while you're here, none of them have it. Now they have, what's interesting is when you think about safety before you go on site, yeah. Safety protocols or safety training, where do you go if this alarm goes off or if this alarm, there's one's for tornadoes, there's one for ammonia leaks, there's, there's one for fire. host: You know, where's your mustard [00:25:00] point? They go through all of those types of exercises before you even get to start on site. But I've yet to see one that says, here's our incident response plan. If we have a cyber breach within our, within the four walls of this manufacturing facility. guest: It really should just be a part of that. guest: Maybe , the process for developing , the safety based responses. Maybe it's the same kind of a process that's a, a well tried and tried and tested. There's a lot of standards out there to help guide that. It's the same thing that we're doing. In creating that process is just a different, , topic that it pertains to, but that's how the frameworks and standards are written, so you can use it in different situations. guest: Maybe that's a great way to look at it, and that's a great way to get started on it. Look at what you've already done, try to adapt that to the security part. host: So, Dan, as part, part of it is we're wrapping this up. What would be one piece of advice that, or takeaway that, you know, for our listeners. host: Take away from this conversation. guest: Yeah, so, , no one's claiming that, , vulnerability management is easy. , it certainly is a task [00:26:00] that needs attention and, , it's not gonna be something that you can knock out in a day. The important thing, I think one of the most important things to keep in mind is always the why did you start doing this in the first place? guest: , What happened in the organization? What triggered you to start doing vulnerability management to track the vulnerabilities and then remediate them? Likely answer to that is some kind of, , security program was, designated. So maybe you're being tasked with OT security. The big takeaway of that is never forget the original objective of that security program. guest: So if that security program is about reducing risk, reducing, , the chances of outages, keep in mind as you are deploying various technologies that. You may not be as close to that as you think you are. You might now have a, complete vulnerability list, but that isn't the original objective. Your objective wasn't to get a big list of vulnerabilities. guest: It was to reduce the risk. So how do you make sure that you are continuing [00:27:00] to take those steps and you're not, , seeing the big list leaning back in your chair and saying, we are good to go? You have to tie that to the action to keep it in line with your original objectives. It's very important. Don't get lost along the way. host: Yeah, I, I have found that, , when these tools go in, the events and alerts can be overwhelming. I was talking to a very large life sciences, , client the other day. They average 8 million events and alerts. Think about that. Right? 8 million, you know, and so, , how do you deal with that, right? host: Maybe AI is something that, you know, is, is a, a mechanism that can come in to help you really to try to prioritize and organize such a massive amount of information you're not used to seeing, right? You haven't seen this before , and now you've got a pile of it. There's a mountain high of it, and how do you start tackling that problem? host: Right? guest: , That's a problem that, , yeah, absolutely exists. It is one that there are some interesting solutions that we can probably take from the [00:28:00] IT world in just detection, engineering kinds of processes. AI might be the answer, but it's definitely not gonna be a silver bullet. guest: , So look at how have you dealt with huge influxes of events, , in other deployments of tools? You deploy a new tool, suddenly you're getting. Hundreds of thousands of new events. How do you deal with that? Maybe apply that same knowledge in how you're dealing with these new events. Identify what information do you need to make sense of these. guest: Go and gather that from the experts. Then implement that same, , engineering process to start filtering them out. Then you can, , move forward on that prioritization. Don't try to prioritize a list of 8 million. Try to prioritize a list of 50. How do you get it down to that list of 50? Right. host: Very tough. host: Well, there's anything that we didn't cover that you wanted to cover as far of the topic? Maybe you have something around zero day vulnerabilities or anything that maybe we, we overlooked or, guest: Yeah, very quickly, perhaps on, on zero [00:29:00] days. Is similar to, , that last topic there, there's gonna be a lot of noise about 'em. guest: Always. So the media. Vendors, , they're gonna make a lot of noise that hard. This new zero days here. Everyone panic first, take a step back, take a breath before you do anything on there. Don't buy into the hype. , A lot of these are industrial systems, especially as we were saying, they have a lot of safety built into them. guest: Mm-hmm. And a lot of cases that can help protect you. So everything's not likely to explode the second that, , zero day comes out. TX one. So we do, we do some work with, , the Trend Micro Zero Day Initiative. Sure. It's really a great source of threat intelligence. But as, , I was describing in how the products themselves work, where they're looking at the deterministic nature and they essentially lock it down to that and just detect deviations from what the operation should normally do that can help prevent even zero days from causing impact to your operation. guest: So Vault Typhoon is a good example of that. The vault [00:30:00] typhoon attacks came out and, , everyone was panicked and worried. Well, we already had protection against that. We'd already, , had a few detections of that happening in, , customer deployments where it was a non-event. , They were stopped. The activity they were trying to do was simply stopped before it could start. guest: We didn't have to have a signature or a pattern for it. It was something that the operation didn't normally do, so it blocked it. So taking that operation first approach. It can really be helpful in handling zero days because again, they're never gonna go away. There's always gonna be a new one, and you can bet your bottom dollar that it is gonna be hyped to the moon every time one of them drops. host: Well, thanks Dan. It was a great conversation. guest: Thanks for coming on today. Thanks for having me dinner. Hopefully I come back again soon. ad: Thanks for tuning in to the Industrial Cybersecurity Insider. To stay up to date with our latest episodes, be sure to click the follow or subscribe button now. And if you found this podcast helpful or have a topic you'd like us to discuss, please leave us a [00:31:00] review or let us know. ad: Thanks again for listening. See you next time.