Supply Chain Risk: What Manufacturers Need to Know

Dino [00:00:00] And you're not working with OT on a pretty regular, consistent basis on, what that tool's uncovering, expanding its visibility, reaching out into that entire plant to try to gain as much asset inventory to determine vulnerability and risk, what remote access is going on in there. You are missing the boat, But you as the CISO are gonna get hung out to dry, the third-party vendor that they got in there, they're not the ones that made the news. The global beverage manufacturer made the news

Intro - Outro Welcome to the Industrial Cybersecurity Insider. In each episode, we dive into the world of industrial cybersecurity. Join us as we cover the latest trends, innovations, and practical insights as we talk with leaders and practitioners across the industry. Gear up and let's get into this week's episode.

Craig Let's talk about [00:01:00] the two global dairy producers that had breaches for third-party vendors.

Craig How's that? I think that's timely. I think it's on topic. It's something that we've addressed. We've tried to communicate how third-party risk is a large part of an organization's strategy. What are they doing? How are they, how are they looking at it? What are some things they can do, some takeaways? let's just dive right into it.

Dino Yeah. So let's frame it up. So what do we mean by remote access, right? Everybody kind of... It's like cyber security, right? It's a big damn word. So remote access into manufacturing environments, there are multiple remote access methodologies and technologies typically deployed i- just in a single plant, right?

Dino Yeah. And that remote access, is there for wide range of reasons, one of which is support, remote support, wh- whether it be, the client's internal teams [00:02:00] or their external vendors. Right Um, there's also information that's, that they try to pull out of those machine centers for maintenance purposes, you know, to, to do...

Dino be, be more proactive and predictive in their maintenance activities. and the thing is, is a lot of these plants have cobbled together some remote access that was either installed by the plant, not necessarily by IT. Sometimes, many times it's your system integrator or OEM has put something in there so that, you know, they're, they're being asked to support the plant, and so they'll put something in there also.

Dino We've seen the Cradlepoint cellular modems. We've seen Secomias and Ewon's and, Toasty Boxes, you know, TeamViewer, all the things that you shouldn't be using on your plant floor, right? 'Cause if you're still using network-based network access, right, MAC, IP, and not applicational level, y- you're not really moving the needle forward to improving your [00:03:00] cybersecurity posture in your manufacturing facilities.

Craig Yeah. Well, especially with all of the available options today that are top tier, OT-centric, that bring best in class from an IT perspective built on OT protocols and standards that allow for access into, safely, into the OT environment. there's not an excuse any longer. The excuses are done. I've had all of them I want.

Dino Yeah. And, and there's uniqueness to those industrial protocols that aren't standard IT protocols, that you have to punch holes in your firewall in order to let them gain access to those machine centers running those sometimes, in many cases, proprietary control system applications, right? And, and, and they can do the...

Dino the client can do a really good job of trying to improve their [00:04:00] cybersecurity. In most cases, IT is unaware of a lot of the stuff that's out there because it's not in their purview. It's not the responsibility of those assets to even know when those machines were built or put in there, is one. But then the fact that they're connected to your network, you have no idea what that third-party network security looks like on that side, what their cybersecurity posture is, which is what happened to one of them, right?

Dino The,the bad actor was able to go through the third party who doesn't have good secur- cybersecurity practices, and was able to come down into one of these large global, beverage manufacturers and come right into their plant And once they land in there, again, if you're just using a VPN, now they got free rein inside your manufacturing environment in most cases.

Craig Well, I think that same global beverage manufacturer announced that their dairy products would halt production for the foreseeable future. Don't know if that's several hours, several [00:05:00] days. It's gonna depend, while they address this breach, if you wanna call it. Yeah.

Dino if you're, if you don't have visibility into those OT applications and systems, those control systems, then you don't know how far the breach has gone, right?

Dino You know, IT can usually do a pretty good job of finding out what's been infected on their side because they've got a lot of that technology in place. Manufacturing doesn't. Even if you're, you know, you've bought one of these, top-tier OT intrusion detection systems, Claroty, a Nozomi, Cisco Cyber Vision, a Dragos, for example if you may not have the full visibility, and we see that a lot, right?

Dino If you only have one collector or sensor in that plant, and that's not enough. You might be seeing 20, 30% of your assets, and that's about it, if you're lucky.

Craig you get- got organizations that take separate views. Some are, we are import- we wanna see the north-south traffic. That's important to us.

Craig And others are, we wanna see it all. We wanna [00:06:00] see that east-west. We wanna see that lower control layer traffic, which is significantly more difficult to get to, more expensive, and more time-consuming, but it's valuable down on those lower level networks. You can't rely just on the north-south traffic that hits a firewall to protect you.

Dino But it's not a technical issue. This is a people, right? I mean, we still see OT organizations that do their darnedest to keep IT out of these very critical machine, systems, machine controls, right? They, they do their damnedest to keep 'em out, because whatever reason, whether it's history, they don't have a great understanding of the, how this operate in this environment, and so they don't want 'em meddling in there.

Dino You know, it's, it, if you, even if you think that you're well-aligned, it doesn't necessarily mean that OT wants anybody meddling in their particular world, right? Safety reasons, [00:07:00] environmental reasons, product integrity, q- quality. A whole bevy of reasons why they don't want 'em in there, you know, which is why I always say the OT industry has to get better at this game, right?

Craig They do. You know, we're, we're finishing up a, an analysis on a, for a client, and one of the common themes was there has been an inherent distrust of IT in the environment over the past several years, and now they are intentionally blocking IT from the network. Don't know what spurred it, don't know what it's gonna take to fix, but at this point, that is a cultural issue.

Craig That is not easily resolved.

Dino Again, it's not a technical issue, right? I, I've seen the same thing at a automotive, manufacturer just last month, same thing, right? Failed an audit. You know, the CISO's doing his, is working his hardest [00:08:00] to try to, to get, the asset inventory that was required out of that audit, a- and he's running into a, into a brick wall with the OT guys.

Craig Back to my famous, very famous- Statement. Responsibility versus authority. And I would say most CISOs have the responsibility, but very, very few have the authority to go in and effect change. That has to change, period.

Dino Yeah. And, and, and if you are a CISO or a CIO or an InfoSec leader, I would implore you to try to find out who are the top integrators that are working in your manufacturing plants today.

Dino Your OEMs and your system integrators at your plants rely on the people that those OT people do trust, right? The, the, the ones they spend millions of dollars with to help them keep those plants running, buying equipment, buying software, installing networks. You know, all this technology, the [00:09:00] integration that's going on in that plant, you're gonna have to make some new friends.

Dino If you want to get past Craig's statement of you have the responsibility but you don't have the authority, but at the same time, you're not tapping into probably one of the top,influencers that are working in your plants today.

Craig Well, so my question, you know, if we look back and, ask a, a very hard question of the two global organizations that just got popped in the last-

Dino 48 hours, 72 hours

Craig in the last, since Monday. So if you look at those, how many of them in their incident response plan and procedures, you have a CIO or a CISO calling the number one system integrator to ask them, "Hey, can my plant operate safely in this environment?" Or, "What recommendation do you have so that my people are safe [00:10:00] if we continue to produce?"

Craig Or, "Do we have to just close?"

Dino Colonial Pipeline, you know, was one of the originals back five, six, seven years ago, however long it's been now, was a classic example of that. And, it still happens today. every day. this, this is not unique to what happened to these beverage companies in the last 48 or so hours.

Dino it, they, they, and they've got the tools. Most of these companies I know have these intrusion detection systems

Craig But again, what's their philosophy? Is it just north-south? Is it we wanna see everything? Is there, are there cradle point cellular modems that they are not aware of in their environment that the tools are not picking up on?

Craig What... You know, there are so many questions that we don't know and that probably won't be available to the public, but at the end of the day, the technology is there. This is a people concern.

Dino Yep. Yeah, and some of it's, it's [00:11:00] budget related, right? Where, you know, IT, who, who should pay for this? At the end of the day, who's the board, right?

Dino I'm tired of hearing it's a budget issue, because if I go and I look at a K1

Dino or if

Craig I look at a

Craig SEC filing for quarterly or annual, these organizations are doing record business with record earnings. it's a, it's not a budget. There is funding available if they prioritize and choose to use it.

Dino Yeah. But, you know, and the, but the money that they're losing because of the breach-

Intro - Outro ...

Dino is costing them millions of dollars a day.

Dino Instead of investing, something that pays dividends, you know, if you were to, to put a, a technology in place where you have full scope and visibility of your manufacturing environment from a cybersecurity perspective, to know that whatever breach that you got, to know exactly where it is, to your point of being able to continue to [00:12:00] operate, doesn't cost near as much as the millions of dollars you're losing every day because you're down.

Craig But you know that that analysis was run pro- if I were to ask both organizations last week, "Hey, what would it cost you if you were down?" They're like, we're not gonna be... It's, it's not gonna happen to us. We're fine." And somebody said, "Our cost to do preventative was X." Let's pick $1,000. Well, our cost of Y, an after breach incident to repair downtime, reputational brand damage, everything, is gonna be $100,000.

Craig It's a, a minimum of 100X almost. So it's, it's not a fair comparison, but it's a, it's a mathematical game, same as insurance. I got friends that refuse to buy insurance on their car because, "Ah, I'm a safe driver. It won't happen." And when they get in an accident, then what are they doing? They're buying a new car.

Dino Yeah. you know, [00:13:00] and, and, and back to, you know, doing this, the, the people that, that pull back and would say, "I don't need to track the east-west traffic in order to build an asset inventory to determine what my vulnerabilities and risks down there and to continuously threat monitor," that's one opinion.

Craig Right.

Dino And, and it's like why does that person's opinion outweigh the entire organization, right? And, and, and, or the OT guy just as guilty who's saying, "Ah, we... What, what's... Why do we need this, these cybersecurity technologies in my air-gapped environment, right? Or it's not connected to anything." Meaning that just because your PLC's sitting in both worlds, you think that's your firewall is your PLC that's gonna protect you, right?

Dino So it, it's, it's definitely, you know, a conversation, and it's not being had, you know, at, at a level of what I call get all the liars in the same room and going through one of these exercises, right? Mm-hmm. And have somebody in there that actually understands both worlds to [00:14:00] determine where your gaps are.

Dino You know, why is it okay for you only to have visibility... you either have none because you're still navel-gazing, you haven't even started. Or you put a tool in and you're only seeing 25% of your assets in your plant, and you're missing out on 75. You know, why is that okay? You know, I think it goes back to money, you want my honest opinion, because IT got money to buy the tool, deployed it best they could with whatever capabilities and influence they were able to, extend in that plant.

Dino And then at that point, when it comes to the other side of the fence, not knowing you need to spend, another several hundred grand maybe to be able to outfit that plant with the things that you need in order to get 100% visibility in there.

Craig Well, so if we take a step, half step back, and we look at maybe some practical exercise that a client could do, whether that's vendor audits, [00:15:00] that's incident resource planning, response planning, it's segmentation, what are

Craig what would be some guidelines or recommendations or at ... let's call it table stakes that organizations need to do or to ask or to figure out to at least start protecting themselves?

Dino you take these large beverage companies that just recently got hit, and knowing that their global footprint and the number of plants they have, you have hundreds of people working in your manufacturing environments that are not part of your organization.

Dino They're not, they're, they're not internal, they're external, And again, going back to incident response plan, we do a lot of proposals and we win a lot of jobs. there's never an IR plan discussion on what would happen if we were in the middle of what happened to them on Monday. And here you've got this breach going on, and we've got four projects going across five of your plants, right?

Dino what's the IR plan? 'Cause you're not sharing it [00:16:00] with the system integrators. And if I was to ask that and put it in my proposal, what is your IR plan? What should we do? What's the roles and responsibilities if you have a material breach while we're working on site? What are the steps?

Craig Well, we should look at it from a, a one step removed and say, "If this happens, how are you gonna reimburse me for liability on all of my laptops or all of my assets that are connected to your network when it was breached?"

Dino Well, and we have that in our proposal, right? For things that are beyond our control that, that keep us from continuing to move forward for whatever reason, you know, at some point financially, we, we get compensated for that, right? If we have to say put for a month while you're trying to remediate and mitigate and then try to re-engage us, you're gonna

Dino There, there's a cost associated with that, and it's built into the, into our, our proposals, into the contract. So we do have th- those, those types of scenarios where, you know, again, it's just more money you're losing,

Craig But, you know, you l- you, we run the risk of if we're there and we have a [00:17:00] laptop plugged in doing some remediation or some work, and that laptop's infected and it accidentally gets brought back to our office, again, we are potentially at harm as well for their environment.

Dino Well, and our corporate IT people have their safeguards, you know, with their EDR packages that we have on these machines. And, the things that not, not to get on the plant network, use the Wi-Fi network, you know, try to be as light a touch as possible while you're on site, but that's not always the case.

Dino It depends on what, what you're trying to do and what you're trying to accomplish there. But, you know, these third-party vendor breaches are widespread 'cause you got a lot of mom and pop integrators out there that just do not have the sophistication. They, they're really good at, a filler or a pasteurizer or a palletizer or some mixing and blending vessels as far as the control aspect of it, but they're not your top-notch, you know, cybersecurity firms.

Dino Or networking firms. That's not them. They're just really good at building skids and machine centers and programming and integrating that equipment, [00:18:00] right? That's not their forte. And that's the kind of stuff these CISOs and CIOs, these organizations need to flush out. There are those of us in the space that are, and then there are those that are not.

Dino And you're not working with OT on a pretty regular, consistent basis on, what that tool's uncovering, expanding its visibility, you know, reaching out into that entire plant to try to gain as much asset inventory to determine vulnerability and risk, what remote access is going on in there. You are missing the boat, But you as the CISO are gonna get hung out to dry, you know, to your point. The third-party vendor that they got in there, they're not the ones that made the news. The global beverage manufacturer made the news

Craig That was my point I was getting ready to make, is in neither of these, you didn't hear third-party company X breached one of these global manufacturers.

Craig You saw their headline splashed across the five o'clock news that said ransomware [00:19:00] breach, w- whatever the title was, for these two large global manufacturers. That is brand deterioration.

Dino One of them had to file with the SEC, right? Right, if you're an American company and you're publicly traded, you gotta tell the SEC, right?

Dino Yep. So I mean, that- that's, that's part of the game, right? To the extent of it, and this, that's how you end up in the news. So, and, and you may say, is it har- harming your reputation, your brand? I don't know. These companies are pretty big, and they may survive some of that, but they are gonna take a hit on downtime.

Craig They will survive, but you can't go back in the way back machine to get your time back.

Dino Right, and, and I don't think somebody's not gonna buy their product because that happened. But you might have investors that might decide, "You know what? I'm gonna sell that stock because they're not pr- They're not operating at the level that I thought they should be operating at to, to secure and protect my investment

Craig Both of them did make public statements that said the [00:20:00] integrity and safety of the products, even though we're experiencing a downtime, is not compromised.

Craig So at least there was no recall. There was no integrity on what they had already made, but they can't make anything.

Dino I mean, that, that's a lot of, people that you're paying and utilities that you're paying and product that you're getting rid of and trucks that are waiting and taxes that you're paying and

Craig It's a distraction, Dino

Dino all the things of just rack and roll it up. If you're a billion dollar a year operating revenue plant and you've lost a week's worth of running time, do the math,

Craig It's a distraction from operations, and it takes time away from what they are super skilled at doing, and that's making product.

Dino And even all the professionals you bring to the table, both internally and externally, you're still bringing the OT folks to the table to help solve this problem.

Craig Agreed. [00:21:00] Yep, totally agreed. '

Dino Cause if you, if you were, we would know because our phones would ring 'cause we're operating in these plants that we're talking about.

Craig Yeah. true statement. I think that was a... Hopefully, organizations learn from this. Hopefully, they can take something back to their leadership, to their teams and say, "Hey, how are we addressing this?

Craig Who are our vendors? What is their access into our facilities? How are we managing that? Do we have one program? Is it 15 programs? Is it different for every plant, and we have 30 plants?" What... So organizations, you have to start asking the tough questions. you... This can't be a check the box anymore. It's not gonna go away.

Intro - Outro Thanks for tuning in to the Industrial Cybersecurity Insider. To stay up to date with our latest episodes, be sure to click the follow or subscribe button now. And if you found this podcast helpful or have a topic you'd like us to discuss, please [00:22:00] leave us a review or let us know.

Intro - Outro Thanks again for listening. See you next time