Staying Ahead of an Industrial Cybersecurity Attack with Carlos Buenaño, Armis CTO

[00:00:00] Carlos: when you talk to a plant manager.

[00:00:02] Carlos: there's only 2 things in his head, you know, keeping production, improving, production, but also safety of the people. So then when you actually think about a control systems, especially it's actually critical systems. if you ever lose visibility of a naughty, process like operators, the displays of the operators go frozen or there is no communication to the controllers, that's actually reason enough to go do a shutdown.

[00:00:32] Carlos: if you actually going to install agents from a Nazi perspective to protect those devices, Then there's a risk, and there's a risk that increases very, rapidly. on top of that, you also have consider that these devices are maintained with very strict SLA when it comes from the vendor perspective. Hi, my name is Dino Busulaki, and today I have the Arbus CTO, Carlos Buenano, if I hope I said that right. Welcome, Carlos.

[00:01:32] Carlos: Thank you so much for having me. So this is great to be here. I'm very pumped.

[00:01:36] Dino: no, we're glad to have you. You're one of top cats we wanted to talk to cybersecurity. So. tell us about your journey into the position you're in and how you got there and why you're there. And obviously we want to talk about armistice too. So I'll turn that over to you.

[00:01:54] Carlos: Thank you so much. okay. Well, I started my career as a, Electronic engineer focusing on process control systems. back in the day, I was actually working for integrators, um, you programming, PLCs, programming, this year's, typically it's both manufacturing environments, and, um, packages and things like that, but then I moved into, working with a big vendor with Honeywell for a while.

[00:02:20] Carlos: actually saw the transition. From TC 3000 into Experian, R 100. that was actually a really interesting experience. you know, moving from a, completely, massless, control systems, where you have to tap, tap, tap to actually, to get to the parameters and change it, to, windows based, D Cs. It's quite interesting, moving to a consultancy, you know, integrating several control systems, but focusing mostly in control networks. That's actually when I started really understanding, how networks work and firewalls and designing, security. Around, control network, especially, integrating, Yokogawa systems, Honeywell systems, of course, packages like, air compressors and generators and so forth, connecting all into each other, of course, then programming, control systems to do the work that I need to do, right?

[00:03:15] Carlos: and I think my, most interesting experience was, to build up a control network from the ground up in Australia, an energy company. basically, I was actually the principal, engineer there, and I had to have to design a green field, uh, control network. and this is actually, LNG network, which mean means that we have, 5, 000 wells. gas plants. We have water treatment facilities, compressions facilities. control rooms. And we had to basically integrate all these interior, all the devices. Integrated control systems. And in that case, it was actually Yokogawa as well as, building management systems.

[00:03:57] Carlos: Altogether, we have, lands and several wins. And we actually work with microwave L. T. 5, you know, you name it. I was actually able to play with every, a technology known to men, but the interesting part in there is that that was actually the part where I had to design a control network.

[00:04:18] Carlos: And, we were trying to, because it was an energy company, to basically based the design around, ICC 2443 framework. and it was very interesting because every design that we did was actually just going through all the functional requirements or the security requirements.

[00:04:35] Carlos: And then from the ground up, we created a really good. network segmentation where we have, to define what the zones were, all the conduits and, you know, power policies to ensure what listing and so forth. So I learned a ton from a cybersecurity perspective. The time came and I had to move on, but I want to continue on, focusing in cyber security for control systems.

[00:05:00] Carlos: , and then I move into, , clarity where, the dates, I could separate security consultant for ICS. for for a year and a half, but then, I realized that, there was always a gap for me, because when you look at control systems, don't look at just actual control systems.

[00:05:17] Carlos: It's a lot of interactions, in and now the OT network, call OT IT convergence. and then when we're actually analyzing the information there, we always. Needed to understand, from an IT perspective, what devices were actually connecting in and out of the OT network.

[00:05:34] Carlos: And then I came across with Armis, the Armis as an enterprise wide solution, where I could actually gain the visibility from an OT perspective and all the devices within the OT network. See what they do and how they communicate to other devices and reasonable abilities and so forth.

[00:05:49] Carlos: being anonymous, and I also was able to see all that information from the devices, communicated into the OT network so that I can create profiles of the devices that were communicated in and out. So that's actually motivated me. And then I move on. into armies and now I've been actually 6 years in armies, armies, you know, tea environments.

[00:06:11] Carlos: And it's actually having fun. Like we were discussing earlier.

[00:06:15] Dino: I didn't realize that actually. We kind of started similar spot, right? I came out of the beer industry also, right?

[00:06:21] Carlos: Yeah. Okay.

[00:06:22] Dino: adult beverage manufacturer in the world for 20 years, working in engineering, doing a lot of what you described that you were doing for the, uh, power plant in Australia.

[00:06:31] Dino: that's interesting. And then how we went off and came back, as far as focusing and recognizing that there was a need. When it comes to cyber security around these control systems, right? which kind of leads me to my next question, and you touched on it a little bit with the IT OT convergence.

[00:06:47] Dino: I had somebody present it to me in this way. When you think about IT's role and OT's role, but a balance that's off in the decision making process, right? And I equated a little bit, the way that they framed it, I thought was pretty good. It's like there's 15, electoral votes, and IT's got 12 of them, and OT's got 3.

[00:07:07] Dino: And the question is, how do we balance that out? Because everything you were describing is you were going through the work that you're doing at the power plant. What was really IT's role? And I see it as a system integrator firm working for a very large SI. You know, a lot of times we're not getting phone calls from the CIO or the CISO or the IT leadership, having us come in to talk about the project that we're getting ready to do, whether it's Brownfield, Greenfield, lifecycle replacement, recon control, you know, integrations where they're really digging into what it is that we're bringing to the table for them.

[00:07:40] Dino: Do you, are you seeing that from an arm's perspective, I know you guys you work with it quite a bit, but what does that, that move towards OT look like?

[00:07:48] Carlos: it is very Difficult roster. And then that's why I come in, in the conversation because, having that. Operational experience from a naughty perspective. I can actually understand the positions from another perspective and also, have the opportunity be a system administrator network engineer.

[00:08:05] Carlos: Then I can also see the perspective from the I. T. And I try to kind of join forces and see the positive between the 2 and the conversions between the 2. But the 1 thing that we always need to understand is, IT networks and OT networks are completely different and they're designed for different reasons.

[00:08:22] Carlos: The IT networks are typically designed, to improve, productivity and provide more tools, to the users to be more productive and the more tools the better in a way. and, of course with great power company responsibility, and then I'll put more security in place, and add agents and, to monitor the traffic and to monitor the information that's going in and out of the buses.

[00:08:47] Carlos: So, it becomes very process intensive when it comes to understand them. The IT networks, the OT networks, on the other hand, is completely the opposite. The OT networks are defined for a purpose, and it's the purpose of communicating, ICS devices and perhaps, So, um, other IOT devices like, cameras and, TVs and, handhelds, but dedicated and purely dedicated to the process, the OT process, when you talk to a plant manager.

[00:09:17] Carlos: there's only 2 things in his head, you know, keeping production, improving, production, but also safety of the people. So then when you actually think about a control systems, especially it's actually critical systems. if you ever lose visibility of a naughty, process like operators, the displays of the operators go frozen or there is no communication to the controllers, that's actually reason enough to go do a shutdown.

[00:09:47] Carlos: if you actually going to install agents from a Nazi perspective to protect those devices, Then there's a risk, and there's a risk that increases very, rapidly. on top of that, you also have consider that these devices are maintained with very strict SLA when it comes from the vendor perspective.

[00:10:06] Carlos: So, what that means is that, if you have a system that is driven by Yokogawa or Honeywell, they will actually have very, very strict requirements of, how do you configure the active directory? You know what? Either you can actually install in the machine, what in points and so forth.

[00:10:24] Carlos: So there is a lot of really requirements that, needs to be met. and if you let it to start Installing that you could actually potentially break that is the lay and then lose the support that you need maintain operations. So it seems like an impossible task join them together.

[00:10:41] Carlos: and in fact, going back my role energy company. So we were able to work together one effort. And that was actually to build a new, operator control, area, right? So, you know, that, uh, uh, with the control room, And the way we did it, it's like, we have the SMEs from the control system and we have the SME from the IT system.

[00:11:02] Carlos: And then we actually create common design. but the common design, it wasn't merging the two networks together. It was still, we have, our, Subnet, if you wanna see it that way, of the OT and which are actually managed by us. And then the IT network, and then we have of course the DMZ and then the two firewalls, where one firewall was controlled by the OT team and the other firewall, the OT team.

[00:11:26] Carlos: But the good thing is that we created. design where all the drawings were the same, I. T. N. O. T. Have the same drawings. All the procurement was the same. So we actually use exactly the same hardware across the board. so that everything was actually a lot cheaper. and, everything that we design was Taking that into account, Hey, we know what devices we need to connect across from the OT to the it. And then we designed the policies in the, the firewalls, with a what listing, procedures to make sure that all the communications that we needed from, PI, for instance, to this car service and so forth.

[00:12:04] Carlos: we're in place in both files and so forth. So that worked really well because it was very efficient. Because we didn't have to actually ask for permission because of the design was from the ground up together. So there is ways that we can actually work together for sure.

[00:12:18] Dino: Yeah, that's a good point. I call them the aftermarket parts. when you have IET, for example, as an SI, we don't implement EDR and design EDR into the control systems that we're providing to the client, right? Because to your point, they can be disruptive, right? Yeah, we'll find in a lot of cases after we've done that work that I T will come in a later date and install a CrowdStrike or Sentinel one in the environment and all of a sudden you start having odd problems within your manufacturing environment can't quite put your finger on, which is really good when you have tools like arm is in place because it's going to show you that disruption.

[00:12:56] Dino: Right?

[00:12:57] Carlos: exactly. Yes.

[00:12:58] Dino: That's a good point, right? And for the IT people out there listening to this, think about that before you start just throwing a CrowdStrike or Sentinel 1 or some EDR package into that OT environment around those HMIs because it can be disruptive if it's, forwarding communication, scanning at the wrong time, doing its AV and all of a sudden that HMI freezes for even 30 seconds and all of a sudden my process at risk.

[00:13:23] Carlos: think the first question I need to ask themself, and this is actually something that I will actually communicate to the IT people is, is it qualified by the vendor? the manufacturer of the system that want to install it. And if you can't answer the question, you need to investigate, investigate which vendor being used and then whether or not that the vendor supports the EDR.

[00:13:46] Carlos: That's actually a very important part that, IT people tend to ignore.

[00:13:50] Dino: Right, right. Well, I want to focus a little bit on a term that I heard it asked for this, past month in February is cyber informed engineering. Right, which to me goes along with if you have a client who's it's chosen arm is their OT cybersecurity platform and you've got projects going on in your manufacturing environments to be looking ahead of saying, well, we're an arm is.

[00:14:15] Dino: Shop and I've got all these new machines and skids coming in over the next year or two, three, whatever, what should they be doing in order to ensure that they're improving and increasing their cyber posture within the design build of those new machines?

[00:14:30] Carlos: Well, there is a lot of things that you should be doing first and foremost is that you have to make sure that you have that visibility in place. Okay. So, problem that have is that you actually adding. Things into the network, packages and new equipment and new lines.

[00:14:47] Carlos: and you lose focus of, what devices you have and you are maintaining and, not seeing the devices that you have, what is going to happen is that going to have risks. And that's that you're not aware of, and they could actually be introduced into the systems without noticing.

[00:15:07] Carlos: Okay. So if you have continuous monitoring or what devices are actually into the network, and then you start looking at, brand model framework versions that they're using, and then match that with the vulnerabilities and the risks. on top of that, then what these devices are tend to communicate?

[00:15:24] Carlos: Are they configured properly? so that's actually very important. Okay. and one thing that I find as well is that sometimes when you actually bring equipment, depending on what they have in the stores, they might be already old. and meaning that, you have device, a PLC that, even though you're buying a brand new, you might have, an out of date, firmware, for instance, before you're actually put it into production, you need to actually check whether or not you can patch it and you can actually get it up to date.

[00:15:57] Carlos: Because one of the things that typically happens you're in production is that, you can't shut down, whenever you need to patch or you find vulnerabilities, That's why you have shut down windows, right? So that you can actually start looking at, what you need to fix.

[00:16:12] Carlos: But what people don't understand that shut down windows project altogether, meaning that you have a project manager that is in charge of all the. Activities across that shutdown window. So sometimes you can't even actually patch during that shutdown window because, they might be using a piece of equipment for testing some changes that I need to make in the control system, on the program, and then you need to test it. And then if you want to bring that down, just because even in the shutdown, window, and they won't be able to test, then the project manager is going to tell you, unfortunately, I'm using it and you can't patch it.

[00:16:48] Carlos: So that's one of the things that, you know, make sure that when you're bringing things in, They're up to date with the latest, patches, and also understand, what the behaviors, from a device perspective, what are they tend to communicate if there is a misconfiguration, for instance, they try to connect to the Internet, or try to, connect to the event, the website, just to grab updates or things like that, like cameras, for instance, that typically happens, just make sure, All that is actually, taking care of before you start introducing new equipment into your plant.

[00:17:19] Dino: Great point. Yeah. Yeah. And again, for the I. T. leadership out there, you know, kind of sum this up 1 just make sure you're aware of your company's capital plan. just be aware of it. Right timing of it. what they're Focusing on as far as their capital spend, the others becoming in tune with their maintenance windows, Because that's your opportunity to go in and make some more. They're going to have a whole bunch of other activities going on, which is why they take those maintenance windows. They usually don't take them down just for a patch update. They probably got activities geared up for that downtime window.

[00:17:52] Dino: Just be aware of those because that's your opportunity to get in there and do some of this work. That being said, so along those lines, from an industry perspective, what should clients be doing, right? I mean, we still see a big chunk of the manufacturers out there. Maybe 50, 60 percent of them that still are that unaware or just beginning to learn arm is his name, right?

[00:18:15] Dino: And then you got. Maybe 30 percent that have actually started doing something, maybe at a POC with you guys, and maybe brought you into it, did a demo, and then you got 10%, you know, that have actually got a plan, picked a product, and running down the path. What else should they be doing?

[00:18:29] Carlos: Well, first of all, they should be doing something because that's something that we actually find is some strategies is do nothing. that's actually pretty sad because, maybe they don't have the investment or they don't see The criticality of, improving, , the visibility into the plans, the inventory and understanding the risks and vulnerabilities that they should do something.

[00:18:52] Carlos: and the consensus has always been, Hey, I designed my network. I know my network. It doesn't have as many risks as other networks. then you ask the question, how many devices do you have? And it's like, I've got, I don't know, about 400 devices, you know, give or take. it's pretty well isolated.

[00:19:12] Carlos: we don't have communication, in and out of the OTT network, you know, it's air gapped. And, we said, Just give us a try, let's just take a look, let's do a POV let's see what would come out with,

[00:19:25] Dino: TrustButVerify, we call it, TrustButVerify.

[00:19:29] Carlos: and the reality is that, you know, instead of 400 devices, they have 1200 devices that they didn't know about because there's lot of projects that happens in parallel.

[00:19:40] Carlos: And. they said, Hey, I'm just going to install a new generator, but what they don't understand that inside the generator, they have switches, PLCs, sensors, VSD sometimes, you know, they have a lot of equipment that it comes as a vendor package and then they don't understand that it's a lot of other things has been added to the network and they're not aware of and because of that, so what they should be doing is they should be always, doubting themselves in the, in a way that, Hey, if I think that is 400, it should be, it should be more than that.

[00:20:13] Carlos: Okay. and on top of that, of course, there should be, if there is no money for investment, try to figure it out, try to see the trends, try to understand and get the budget so that they can start exploring. new ways to increase the visibility of the devices, see the risks, see the vulnerabilities and then start doing something about it.

[00:20:34] Carlos: Right. So, and sometimes I understand in OT environments, you know, mitigation is actually key. sometimes you can actually have to create exceptions. but one thing that they need to be doing from a perspective is to understand. the behaviors of the devices, and again, armies to, understand how the devices are communicated in and outta the network.

[00:20:54] Carlos: I mentioned that, the customers believe that they, the networks are isolated, but in fact, nine times out of 10, we found that there are devices that are directly connected To the network and 7 times out of 10, they are connected directly to the Internet and things that, sometimes the engineers.

[00:21:13] Carlos: needed some, support from a vendor and they open up a channel, a VPN channel to the vendor, and without realizing they didn't put the, the, the controls in place and then they open the control system to the internet without realizing. Right. so, and then if you get.

[00:21:30] Carlos: I mean, then to manage information, then we will tell you, what devices you have, what they're doing, whether or not they actually reach and offloaded in a support, that they connect into the external networks, whether it's out the network or internet. But it's important to do something to understand the actual, cyber security posture of your network for sure.

[00:21:51] Dino: A hundred percent. what you're describing is our findings too, That they just don't know what they don't know about their environment. Because it's, what I say is very difficult to manage what you're not measuring. And, and the only way to be able to do that is, is to bring in the technologies that exist today.

[00:22:06] Dino: be able to get that asset inventory, understand those relationships and risk of those assets and the vulnerabilities associated. Remote access is a big one. We all know COVID really kicked that door wide open to provide remote access in there. And so what was once air gapped or isolated is not the case.

[00:22:21] Dino: Even if you think that it is, there's still some tech that shows up with a laptop and goes out there and plugs into that machine, or they plug it into the network for that period of time. To gain remote access to your point, and then they forget to unplug it, right? And the state's connected to the network.

[00:22:36] Dino: So, if you're not watching it, you're really not sure what the dynamics are in that particular space. tell us what your experience has been. What we've witnessed is clients that have had a material breach, and have paid the ransomware, Just in order to get their stuff back. Versus those that have decided not to pay it.

[00:22:54] Dino: And then fight their way through it for the next several months to a year, you know, at a huge cost. Are you starting to see a shift where organizations are just not trying to roll the dice anymore and just rely on their cybersecurity insurance? Which is only going to pay for the remediation mitigation work.

[00:23:12] Dino: It's not going to pay for the business disruption that you had.

[00:23:16] Carlos: Yeah. We're seeing more proactive, definitely customers are becoming more practice for sure. and I think customers are starting to realize, okay, how much efforts does it take to recover from a ransomware? And we had, several, reports that tell us, the average downtime is.

[00:23:33] Carlos: Two two weeks as a minimum or two months, to six months and the cost of production, it can actually become, millions of dollars, I'm talking about two figures that are intense the 10th or 20th, depending on the production that they're actually doing. So they started to realize that that's the case and they're putting more budget into, into the cybersecurity programs.

[00:23:56] Carlos: And They started slowly, with a very limited, objectives that what they want to achieve. You know, starting from a visibility and risks, then the vulnerabilities and vulnerability management and beyond. But we actually see a trend that customers are actually more and more proactive to get to that point.

[00:24:15] Dino: it's kind of getting close to the end here. So what advice would you give to somebody who's listening to our podcast or who is looking in an OT cybersecurity strategy journey? what would you convey to them?

[00:24:28] Carlos: So, cybersecurity is real, the threats, that at the moment from our perspective is there, and it's real and it happens. And the biggest issue that we have is as soon as we open the understanding the risks and vulnerabilities on the OT network, get overwhelmed.

[00:24:46] Carlos: and then they really feel like there's a lot of work to do to actually get to a safe cybersecurity program. The advice that I give, and it's something that I, suffer myself when I was actually doing this project with ICC 2443 is that frameworks out there to help you through the process, what actually suggest, the audience is.

[00:25:09] Carlos: Don't be scared of a framework. Whatever framework you pick, they are there to literally take you by the hand, help you, develop the security program so that, you are actually Um, one day, more prepared be proactive to reach the operational resilience.

[00:25:28] Carlos: What you don't do do nothing, right? So do something, a framework will actually help you through the process. If you pick. I see for 3, for instance, you have foundational requirements security requirements, and you start from the 1st, 1, all the way to 7. sometimes it's complex, especially when it comes to them of cementation.

[00:25:49] Carlos: But do 1 step at the time don't feel overwhelmed, set objectives, very clear objectives. what it makes sense to you. Network segmentation is always the first step of mitigation, because if you close, the in and out of the OT network, you're actually minimize exposure. but the 1 thing that you need to do is just gain the visibility understand behaviors of these devices and then how they.

[00:26:12] Carlos: communicate in and out of the OT network or even within the OT network. then from there, they keep going following the steps of the framework.

[00:26:22] Dino: your experience, as clients are going through this, this journey again, what do you see that OT cybersecurity makeup of, right? You got the IT OT convergence, we have this thought process. You got to think globally, you know, make a decision on a tool and technology to use, but you got to act locally.

[00:26:40] Dino: Well, how do you see these teams organizing within the organization? Are they, is it, are they doing things internally to build out their practice? relying on their partners to some extent? how do you see that? ITOT cybersecurity team shaping up.

[00:26:56] Carlos: a bit of both to be completely honest, but the trigger is always regulations. We can see more and more regulations that are actually popping up. Like, for instance, if you are in Europe, there's now a regulation for us to, you know, to a specific security and develop a security program.

[00:27:12] Carlos: and, and because of that, you know, typically the CISO then triggers, process and, and, Some customers, they go directly and then they build their own cyber security team, and they start actually working towards the security program. Some other customers, bigger customers, they already have relationship with partners, and they bring the partners in, people like yourself, with a lot of experience, Basically, a clear path to succeed in developing introducing and deploying and these security programs, then rely on that experience to go through that process. but, to answer the question is I've seen both and depends. A lot on the maturity, of, the team and, how quickly the team wants to achieve the goals and what goals they want to achieve,

[00:27:58] Dino: is there anything that you would like to add that we didn't cover in our discussion?

[00:28:03] Carlos: you know, very quickly, in some trends that I have seen, more and more, people are new technologies when it comes to cyber security, especially when it comes to, deploying quickly and escalably, Thank you. across all the different sites.

[00:28:18] Carlos: so they actually start using a lot of cloud, solutions, to make sure that they have more effective way to get, information from advisories and threats information and vulnerabilities and so forth. you know, I'm actually surprised that we haven't mentioned it yet.

[00:28:31] Carlos: So it's a lot of the use of AI, um, we can't really do a podcast without mentioning AI. but that's actually proven that helps, there's a lot of data as you know. So when it comes to. Connections, vulnerabilities, the risk. There's a lot of data. and, creating that, using a model, to manage all that data is actually something that, is a new trend.

[00:28:53] Carlos: And of course, you know, ransomware and how organized the ransomware is. becoming and, how much money and budget they have to do what they do. So we actually have to basically, attack back and then make sure that we, use new technology to protect ourselves.

[00:29:10] Dino: yeah, I think what you said earlier do something, I mean, at the end of the day, you need to be, thinking about your manufacturing environment and your industrial assets. And have to start doing something, right? thanks, Carlos. Appreciate it.

[00:29:25] Carlos: thank you, Dana. So it was fun.