Talos Takes

Patching in the dark: Managing unknown threats in complex environments

Cisco Talos

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 23:15

If you're tired of being told to "just patch," we understand. The threat landscape is evolving at breakneck speed, with AI-driven tools enabling adversaries to uncover and exploit vulnerabilities before defenders even know they exist. In this episode of Talos Takes, Amy sits down with Threat Intelligence Lead Pierre Cadieux to discuss how to defend against these unknown threats. We move past the simplified advice of "just patch everything" to explore the logistical, technical, and business realities that make patching a complex, high-stakes operation rather than a simple button click.

From the necessity of testing your patches to the importance of building strong partnerships between security teams and business leadership, this episode breaks down the things defenders often miss that build true resilience in organizations.

Speaker 1

Welcome to the Talos Takes Podcast, where we discuss Talos' latest research and security news. This podcast is for everyone, from the C suite to the frontline. Hello and welcome. We are living in a time where the ground beneath our feet is constantly shifting. With AI being used to uncover vulnerabilities at an unprecedented speed, the gap between the known and the unknown is closing. And it's a race that defenders are finding harder to win. What happens when the adversary knows your environment better than you do? How do you defend against a threat that hasn't even been named yet? I'm your host, Amy Ciminnisi, and today I'm joined by Pierre Cadieux, threat intelligence lead, to dissect this topic and give security practitioners practical tips. Pierre, thanks for joining.

Speaker

No problem. Happy to be here.

Speaker 1

So let's dive right in. As I said, we're operating in a world where threats are increasing in both speed and complexity. With Claude Mythos and recently announced Fable, we're seeing AI being used to uncover previously unknown vulnerabilities very quickly. We've heard from a lot of our threat source newsletter authors and blog writers that time from disclosure to exploitation has uh shrunk immensely. Can you kind of speak to this challenge a little bit more?

Speaker

Sure. Um we've always seen this type of a risk in security since as long as I've been in the security industry, um where adversaries are aware of vulnerabilities at the same time over for the defenders and then are enabled to exploit them rapidly. And this requires a lot of infrastructure and commitment by the adversaries to have a willingness to leverage new found threats and and uh vulnerabilities immediately and not just sit on them, not wait for them to be uh leveraged by others, and to have the the dynamic capability to create those exploits and to then leverage it for purpose. So um a lot of our adversaries have focused on this kind of tooling over the years, and sometimes it's been a slow trickle, sometimes there have been spikes where adversaries have access to um an exploit ahead of defenders and are able to then leverage us. But with those things you just mentioned with regard to the AI development and identification of threats, we have to expect that this is going to accelerate. We have to expect that the passive defense is no longer really a successful option. If we're going to be defending infrastructure of any complexity, of any business sensitivity that requires uptime, that requires availability, et cetera. It's our responsibility as defenders to be aware of this and to create those contingency plans and understand the assets under our purview, as well as the technologies we have at our advantage, the logs that we have availability to, and how those might be leveraged to identify things that we aren't aware of yet. This is similar to some other times where we've seen a knowledge shift where the adversaries have a significant advantage potentially. I would say that uh defenders also have a significant advantage if they are leveraging the same tools to identify risks within their environment and then take the appropriate actions to mitigate them. But that last part is the hard part. Knowing about a vulnerability is great. Fixing it or mitigating it, that's where it really counts.

Speaker 1

We often hear the advice to just patch. Um, but patching is a much more complex process than just hitting a button, right? So, what what are those critical steps in the patching life cycle?

Speaker

Early in my career, I also was of the well, they should just patch kind of uh camp. And on paper, it sounds if you have an environment that you understand explicitly, you know the risks you're getting into when you commit a change to your code, to your environment, to the way things behave. One of the challenges of this is if we have custom applications, custom code, custom devices, or maybe little uh like unique devices. And I'm gonna specifically talk about like medical devices at this point in time. I've worked as a consultant a number of times for healthcare organizations throughout the world at various capacities. And there are very specific devices, let's say an MRI machine, which may be network aware and may sit on there and may be running a shadow version or small version of a Windows-based operating system or Linux-based operating system, but it may then incur vulnerability. You can't just patch that without the vendor being involved and without the vendor validating the uh the installation and the change, and then you validating that it still works in your environment. So you need a test environment to test it against. Otherwise, you may have adverse situations where you've created a risk to the availability of this device. And now I mentioned one device is used more for kind of detection work when you're talking about the medical side of things. But if we also had something that's also like an insulin pump or something else that directly impacts a um healthcare uh situation for a patient, that's even more sensitive. Conversely, you just look about like ATMs. ATMs also often run embedded versions of operating systems. You can't just patch it necessarily because it has to be vetted by the vendor, you have to figure out how you're gonna deploy it. They're often air gapped, hopefully, uh, et cetera, et cetera. So there's a lot of considerations about the logistics of actually applying the patches, and there has to be a level, an appropriate level of testing based on the business sensitivity of your environment. I'm gonna repeat that again real quick. We need to have an appropriate amount of testing for patches. You can't just deploy a patch without testing it because it may behave differently in your environment than what the vendor who's deployed the patch may understand. You may have a more specifically configured environment or expectations of things working in a certain way that may be changed or altered unexpectedly during a patching. Over my many years of dealing with patching vulnerabilities and that sort of stuff, I've seen that happen many times where you deploy a patch and everything looks great except for this one thing, which is maybe like the canary in the coal mine, and it is now not going to work until something else. You need a revision of the patch or you need to roll the patch back, and it will still remain vulnerable. And you need to think of other mitigations. How can I protect this device? How critical is this device? Do I need to leave it online? Can I take it offline? If it's critical, how do I protect it against these attacks that are possibly going to be happening to it? And you have to think about its role in your business, you have to think about its sensitivity to the information that it has and the ways of accessing it. Is it internet-facing? Is it internal only? Is it unrestricted network? Is it a restricted user list, et cetera? Those sort of things will help to reduce the likelihood of someone casually finding it. But if it is an edge device or an interfacing device, there's a consideration there you have to make.

Speaker 1

Oh my God. I'm just thinking about the sheer scale of asset management that has to happen there. That's crazy. How can people make it more sustainable for themselves to do this? You know, it's a monumental task.

Speaker

It is. Um, a lot of it goes back to some of the fundamentals that I've been working with and trying to promote for 30 years now. One of the easiest fundamentals is uh the idea of change control, having some sort of process for, and a lot of executive uh environments have that have a very strict mandate for compliance, have some sort of a change review board, whether it's an automated or email-based process or it's a virtual process through like Jira or something. There's a process where you submit these changes, someone reviews them and authorizes them and says approved. That way you have some tracking about what was changed when. So you can roll back. Also, you know what was changed, by who, when. And if you see a change out of a cycle, you know that you need to track this down and find out why this was deployed, patched, whatever. In environments where there's a high sensitivity of data integrity and of security of the environment, big like financial institutions, et cetera, this is usually the way that they handle that sort of thing. It's slow a little bit, but it provides you the window for the discussion, the review, the testing, and the approval that need to happen for this sort of thing to happen. One of the things that I'm gonna also mention, which I've started to talk about a little bit here, is that this is not necessarily a technology problem. This is a or a technology is a technology problem, but it's not going to be solved with technology. It's gonna be solved with people and process. So if you think about those three aspects of the stuff we do, people, process, and technology, technology problems have to often be solved with a combination of technological solutions reviewed by people through a process that's established. So establishing those governance processes is crucial to having the oversight to answer your question about the asset manager. How do you do this? You have to have some sort of a gate where purchases of software, hardware, et cetera, are tracked and logged internally in a central location. There needs to be some sort of a uh validation capability, which exists in many different software uh tools, to look for assets on your environment that you are aware of and that you aren't aware of, and to match those up with users, with business needs, et cetera, and to identify those systems that don't fall into those categories. This is also very useful in the event of an investigation or an incident or hunting. You can find very easily which systems are related to this big important financial app. Oh, hey, it's these 14 systems here, these 10 over here, these seven over here. And this is how they're connected. If you have that, you know the scope of what you're dealing with. And you see something outside of that, you can easily say, this doesn't make sense or this is unexpected, and then focus on that. Having knowledge of that for many of our defenders would be an improvement on where they often sit today. And thinking about the vulnerability leads to compromise, you need that map so that when you get to the compromise phase, you know how you're defending yourself.

Speaker 1

Yeah, I feel like that kind of answers my next question. So I'll I'll just kind of say it. Many organizations operate under the assumption that they know their own network better than the adversary. So, like when we think about worst case scenarios, what you just said, I guess, is how people should be adjusting their strategy when they realize that you know someone else knows their environment better.

Speaker

Yeah. And in a time, over the past like 10, 15 years, I've seen a big shift towards uh, and it's always been a race condition between new technology coming out and our our ability collectively as an enterprise to deploy it. Hey, this new thing came out, I want to deploy it today. And this has created a challenge within the internal IT organizations, speed of evaluation, validation, security controls, and deployment of things that are new that the business is asking for. The time to deployment for a lot of these applications and tools has shrunk, where there's an expectation that we can deploy this within days or less of this new thing coming out. The reality of the evaluation process and the testing is higher than that. So often this creates a need for parallel IT capabilities, what we often refer to as shadow IT, where a business organization has people that do IT-related tasks and deploy technologies outside of the governance that's deployed by most of the IT organization or the enterprise. This is not uncommon in many places. And this is one of those things that you know, if the need arises, business will find a way, sort of thing. Uh, in this case, the need arises, someone who says, okay, we'll just buy the thing and deploy it. This is a way of doing it. This creates a lot of risk because you're not necessarily conforming to the organization's security problem uh program or following the configuration guides and the configuration requirements, and not maybe deploying the monitoring tools, not deploying it in a secure environment. And this creates a huge risk. I don't know how many times I've seen environments that were created as a test environment or as a temporary thing that became production and lasted forever and were never meant to or scaled to be production and should never have gone that way. That happens all the time. And the changes in technology are very rapid. We have to expect that what is an asset today will become a legacy thing tomorrow, and there'll be a new one or new five things deployed. And who owns it, who's assigned to it, what business purpose it provides, those change dynamically. A laptop that I was assigned um a year ago may now be running a localized LLM on it and be running as my own little AI thing that my rest of my team is using and send its data back to our larger corporate environment. That's a risk if it's not managed correctly. And so there needs to be some governance, especially about the creation and deployment of new tools. And I'm not focusing just on LLMs, but that's one of the new tools out.

Speaker 2

Yeah, and don't put just the default settings on it, please. God, please. Yeah, you got to make sure that you adjust it.

Speaker

Yeah. Default settings are almost always vulnerable out of the box. There used to be a metric, a time metric that we used to track for how long would it take to compromise a default install of Windows on the internet? How long would it be put on the internet and how many days, weeks, months would it take for it to be compromised? Last time I remember that's being done, it was like six hours. You put a a default configured box on the internet, and within six hours, someone, not you, now has ownership of it and is doing bad things on it. Yeah. I bet that number is much slower. Yeah.

Speaker 1

Oh my God. So, like when an unknown threat or a zero day emerges, the pressure to react is ginormous. I mean, within your team, and then also you have people pressing you for information. What does a prepared organization look like? Maybe, you know, take me through the first period of time, maybe hour, several hours or day after a disclosure.

Speaker

Yeah. And a lot of this starts obviously before the disclosure. So there's a lot of assumptions I'm going to mention here that I would hope that an environment that is mature and prepared for this would have in place. So you mentioned before knowing your environment. It's important to know your environment. At least have an inventory of your top and number of critical environments. This is our main business maker, money maker. This is our system that our customers use. This is the back-end service thing that does the shipment of our product. This is the thing that an engineer uses. Those might be the top three different environments. Each of those contains assets, servers, cloud hosted systems, software as a service systems, et cetera, knowing what those are and then who's members of this team, who the owners are of the various assets. Having that mapped out ahead of time helps you link things together when you're seeing activity. And so being having that understanding, even if it's just for the most critical system or the top N number of critical systems in your environment, gives you a leg up in your detection process. And also when you have to ask about can I patch this device, you know who to ask about it. You could say, hey, I see you're the owner of this device. I have this critical patch that's coming out. Can you help work with me to test it andor deploy it? So that's one of the first things. Understand what the vulnerability actually is. And so reviewing the vulnerability information, the text, not just the number as far as the provided CVE difficulty or you know, the uh the uh likelihood, uh, often that's listed out appropriately based on sort of a generic uh assessment. If you have environmental configuration things that may reduce that, you can consider that as an uh an altered thing. So this is you know meant for if this was on an internet-facing system, this would be a 10, for instance. But on systems with restricted access and an isolated network, et cetera, this falls lower. And so maybe you can then deprioritize that or have mitigations in place that can help to answer the question to your executive leadership. Did you know about this? Yes, we did. What do we do about it? Well, we've got controls in place and we'll patch it within the normal patch cycle at the end of the month. Oh, okay, that sounds good. And we're gonna have additional monitoring, for instance. We're gonna spin up additional monitoring looking for this specific traffic at our edge and at our network uh egress points and maybe across uh various different uh monitoring positions within our network. That helps you have that heightened level of visibility during the time where you may be vulnerable. And so in case something does happen, you can be ready to take action when you see this. I think part of it's also gonna be if you believe that you have been compromised by that, doing your own hunting. And again, if you have that map of those systems and the environments and how they interconnect, you can then have a more targeted hunting capability and say, I'm gonna target the financial system, I'm gonna look for any indications of any compromise here that may have started with this vulnerability or related to this vulnerability. That's important again to know. But if you also have a mapping across those critical systems, you can say this vulnerability does not come into play or does not impact these systems because they don't use that technology. That's also a good piece of news. Or each of them use this technology, maybe in different ways. This one is deployed in a in a test environment over here, but it's not in production. They can prioritize accordingly based on the risk there. That's one of those things that again requires partnership with the business owners to understand the governance requirements, the compliance requirements, regulatory issues, et cetera, so that you're prioritizing correctly. This is the hard work of IT security and is is not as fancy and fun as finding a malicious actor and doing this cool stuff. But this is the stuff that keeps an environment safe when there's unknowns out there.

Speaker 1

And often often a very thankless portion of it as well. Yeah.

Speaker

Yeah, I I've done uh GRC consulting for a number of years, so you're 100% right.

Speaker 1

Finally, I I really want to touch on the internal pressure that people are facing, maybe from their leadership. It's one thing to, you know, look at the release CVE and fully understand these threats, but you know, it's a whole other game to explain them to executives who just want to know if they're safe or not. So, like, how how should we as defenders be communicating this reality of unknown threats and the need for this resilience to the leadership teams?

Speaker

A part of it is establishing leadership as a stakeholder in any security program and security plan, making sure they understand the process you go through to defend the environment and what you would do in a situation where you have an unknown vulnerability or a new vulnerability being disclosed. Walk them through that process so they understand where they will get updates, what they should expect, and the diligence for doing as far as evaluating the threat, looking across our business critical systems, validating if they are impacted or not, considering mitigations, considering testing and deployment, and validating that they are okay with the timeline on a nominal vulnerability. And then explaining that we also have a capability of doing an emergency deployment. But that creates a different risk. If you deploy without testing, you could be bringing the whole environment down. But if they accept that risk, then they put their little executive stamp on it, then perhaps we have to take action on that with that notification that they are accepting the risk and this is done at the behest of the CEO or whatever. But having a backup plan and being a way to roll that patch back is crucial. If there's a patch that you can't roll back, making that clear to them as well, like we aren't testing or we're doing extra testing on this because we can't roll it back. We deploy it, it changes things fundamentally, and we will have to revert back to a disaster recovery backup or to rebuild our whole environment, not just roll the patch back. That also changes the metric. Some patches are indeed like that, where they change uh something significantly to the point where you have to actually rebuild an environment. So being clear with your executives, I realize that they're not going to understand all the technical details behind the vulnerabilities, but they do understand timelines, they do understand commitments, and they also, if you have again in the security program, you have a periodic change review process that they are part of and they have visibility into and they can participate in. I think that uh helps along the way. Again, this is administrative requirements and work that need to be established, but bringing them in as partners is crucial for them accepting the process you're doing.

Speaker 1

Exactly. And I mean, this is going slightly off topic, but I'm thinking about an episode that I did several months back now, where if leadership is frustrated with the pace and you know, they they just want things to get done and they want to know if they're safe, like point to you know, all of the trails that you've previously left. And I mean, thinking about budget and headcount and things like that, like this is an area where you can advocate for yourself and for your team.

Speaker

I percent I mentioned before people processing technology, and this is where you need to, as a leader of a defense organization, you don't just spend all your whole budget on technology, you don't spend it on people or the process. You need to have an equal share across all those peep those features. You need people to execute the process against the technology. So this is a chance for you to say, you know what, to do this in a way that's gonna be meaningful. I need some some people who are experienced defenders who have done this vulnerability management type of stuff in the past. I need them to be uh here on the team to help to evaluate this one's gonna be dedicated to the financial services system, this one's gonna be dedicated to our shipping system, et cetera, or maybe they're on a rotation or whatever. And you know, you'd make those arguments. I think that in my mind at least has a lot more of a place now in this conversation because we know that there's this wave of incoming unknown or soon-to-be-known threats that are about to hit or will be hitting over time. I think ramping up a vulnerability management process is appropriate for any environment uh that's as a defender. And that also includes refreshing and re-evaluating your disaster recovery plan, refreshing and reevaluating your incident response plan, making sure your leadership knows the roles that they have in these things, and that they know the process for getting a status check from you, the security team, on these vulnerabilities and giving them a more periodic readout of here's the vulnerabilities we evaluated over the past week, here's our plans for deployment and testing. Please validate that you're on board with this. And again, involving them in the process. Executives don't necessarily like to spend a lot of time thinking about security risks, but they'd rather be informed than be in the dark.

Speaker 1

Well, Pierre, thanks so much for joining me.

Speaker

Certainly. There's a lot to be said on this topic. And so we may have to revisit it down the road as well and come back to see how things are going. Um, but uh I encourage all defenders to take a deep breath, think about their relationship with their business leaders, business owners, and then to refresh this.

Speaker 1

Yeah, absolutely. And I'm sure everyone here at Talos wants to thank you for all the work that you do in keeping people in your organization safe. Um and thank you so much for listening. We'll see you again in two weeks for a new episode. And until then, stay safe out there.