Amy Ciminnisi: 0:00

Welcome to the Talos Takes podcast, where we discuss Talos' latest research and security news. This podcast is for everyone from the C-suite to the frontlines.

Amy Ciminnisi: 0:12

Hello everyone! I'm Amy and today on Talos Takes, we are talking about proactive cyber defense, specifically threat hunting. Cyber threats are becoming more sophisticated every day. And networks and systems are becoming more complex. So it's really more important than ever to identify risks before they escalate. We're going to be talking about a few things today. First, what threat hunting is why it matters and common mistakes that people make. Next, the peak threat hunting framework, as well as a new open source tool that Cisco Foundation AI recently released. And finally, how security teams can better structure their threat hunting approach.

Speaker: 0:57

Today we have David Bianco, a recognized leader in threat hunting and also an AI Cybersecurity Researcher with Cisco Foundation AI. David, thanks so much for coming on.

David Bianco: 1:09

Hi, Amy. Thanks for having me.

Amy Ciminnisi: 1:09

Yeah. So to get started, let's establish a baseline in this conversation. When we are talking about threat hunting, what kind of threat hunting are we going to be referring to?

David Bianco: 1:24

So a few years ago, my team and I published the Peak Threat Hunting framework, which actually, has two things in it that are relevant to your question. One is a definition of threat hunting, which is basically any human driven kind of, exercise that you might go through to identify bad things on your network that your automated systems did not like. But the key there is human driven. So fully automated or even AI automated and AI driven doesn't really count because we just call that a detection platform at that at that point. Right? And then the other thing is we also developed three types of threat hunts, or at least we we documented three types of threat hunts. Probably most people who have even heard a little bit about threat hunting know the first one, which is hypothesis driven threat hunting, which is, I think, what you were asking about. But we also have, the baseline threat hunting, which is more along the lines of figuring out what's normal for your data set or your environment and then looking for, deviations from that, that are, that, you know, might, might be indicative of some malicious or suspicious behavior. And then the third that we defined was the model assisted threat hunting or we just call it math for short, but it's that's where you bring in statistical or, machine learning or, or maybe in some cases even, Lims or something like that to help you, but it's there all three years, still driven by humans.

Amy Ciminnisi: 3:15

Got it. You have already talked about PEAK a little bit, but for those who don't know, what is it? What does it stand for? Why did your team develop it?

David Bianco: 3:24

So we developed it and with my two coauthors, Doctor Ryan Fetterman and Sidney Moroney, we developed it to basically modernize threat hunting. So around 2015 or 2016, I actually was working for a different like a startup company that was in threat hunting space. And we created and published the first widely adopted threat hunting methodology or framework. It had been almost nine years since that when I joined the search team back, at this time, we were part of Splunk. Even though the search team is now part of Cisco Foundation AI we had a great threat hunting platform, and we wanted to modernize the way people approach threat hunting with the lessons that we had learned in almost the previous decade, because that's a long time not to say any of the older things were wrong, but we added new things. We got better at doing the existing things. So in a way, the PEAK Threat Hunting Framework is a modernization of the older frameworks, that existed. It's called peak because it is a three phase threat hunt process. The first phase is the prepare phase, in which you go from I have a general idea of what I want to hunt for into. I am ready to start executing data analysis tasks or whatever they are. Right. So it involves researching on, you know, open source research, like internet research about your hot topic, your general hunt topic, how it is performed, what it is, why threat actors use it, etc. but it also involves research in your internal environment. So things like, has my hunt team ever hunted for this before? Have we encountered any incidents that had this technique involved that I could learn from? Does our CTE team have some information about threat actors who might be using this, or, you know, almost any kind of internal, nonpublic data that you combine together with the public research, to get a full picture not only of what that is, but also how it affects your environment and you end up, the preparation phase with a, not only all that research, but also the actual hunt plan. Like, here are the steps that I'm going to do. Here's how they work. And you know, this, the order I might have to do them in, etc. And then you transfer into the next phase, which is execute, which is where you actually execute the hunt plan. You might have to iterate on it a few times because maybe your plan wasn't exactly accurate. Or maybe there's some wrenches that you didn't know about that were going to be thrown into the gears, that you have to work around. And then you move into the third phase, which is the act phase. You feed that knowledge that you just gained from the execute phase back into your organization. So it can have an effect on security. That could be as simple as, I'm going to document the hunt steps that I did and why and what my results were. I might if I found some active security incidents or things that look like they might be incidents, I might escalate them to the SoC analysts. There might be other things to stakeholder briefings to let them know what you've been doing recently. Kind of showing the value of the threat hunting and of course, tracking the metrics for for the threat hunts. With all those three you've moved from, I have a general idea to a concrete plan, which you then execute, which you then turn around and take those results from that execution and put them back into your, security program to strengthen your security posture. There's a one additional layer. It's the knowledge. So the full name of it is prepare, execute and act with knowledge. Because in each of those phases you have some interaction with knowledge. Either you're consuming knowledge. A lot of times in the prepare phase, you're doing the research and you're bringing that knowledge in. But also especially in the act you are creating net new knowledge and putting it back into your environment. So you're getting better over time. So it's it's prepare, execute and act with knowledge.

Amy Ciminnisi: 8:04

So there you go. That's PEAK. So a lot of organizations struggle to get budget and headcount for even defensive cybersecurity programs as it is. And threat hunting is primarily proactive. Right. So for those organizations that have finally gotten the budget and don't have established programs are doing their best to get it up and running, how does the PEAK Framework help them start?

David Bianco: 8:34

I usually recommend people start in 1 or 2 places if they're starting from zero hunt, capability at all. And that's either, for those three types of, of threat hunts that we defined. We have detailed process diagrams to show you how to do them. And at each stage, what your goal is, and maybe not exactly how to accomplish that goal in every situation, because different hunting hypotheses will be different. But we guide you through the process. And so if you have no hunting at all, that is a great place to start. Like start with hypothesis based hunting. Go through the process diagram and the process descriptions and understand how to perform hypothesis based hunt. And just start with some simple hypotheses. The other places especially helpful if you already have a established threat hunting program, starting with the metrics, the the one metric that that springs to everyone's mind at probably the first thing is how many how many incidents did you find when you're hunting? And that is the worst garbage metric for us throughout hunting program ever. Like, it is so totally unfair. Your threat hunting program is going to fail if that's your primary metric, you're not going to fail, but it's going to look on paper like you fail. And the reason is we don't control what the threat actors are doing, how they're doing it, or when they are doing it. But when we hunt, we hunt for a certain, constrained set of behaviors or activities in a certain location on the network and it through a certain time window. Right. And all of those things have to line up for us to be able to actually catch an adversary in the middle. But that is not the purpose of threat hunting, the purpose of threat hunting. First, first of all, figure out how to find these things better. So we can automate them ideally, but also to drive continuous improvement across your security posture of your entire organization. Because when we are threat hunting, we are poking around in places that a lot of times people don't poke around in too much like the automated systems that we have, like our SIEMs and IDs and whatever they consume, the data that they consume, they don't. They inspect it, but they're not really looking for gaps or mis-parsed things or something that, that, all that time. But we run into these, we run into visibility gaps, logs. We thought we were collecting that we weren't, or logs that we didn't try to collect that we should. We've run into old software versions. Nobody noticed. We run into configuration issues where it's, you know, insecurely configured or something like that. We just a whole ton of things. And every one of those things that we find and report back to the organization is an opportunity for the organization to improve their security posture. We might not be the ones that actually solve those issues, but we can at least present the opportunities. Like if we don't present them, they mostly won't know and they won't be able to improve. But the more we present the, the more opportunity we're providing to the organization to improve. And so our, our PEAK metrics are all built around telling the story of the impact that you've had. Sure, some of them are like how many incidents did you open? But key differences might be things like, we're no longer counting only the incidents that you opened during the hunt, but when you developed for example, a new detection mechanism that you automated and then it found a security, issue eight months later, you get to count that because you never would have had that detection in place had you not figured out how to do it during the hunt. It's it's really about telling the story. And we only have five high level metrics. So it's actually really easy usually to start collecting those metrics. So those are the two places if you're zero hunt or close to zero hunt capability. Start with hypothesis based hunting with the defined processes in the framework. And if you already have a functioning hunt program, maybe consider starting with the metrics to show the impact that you're having.

Amy Ciminnisi: 13:09

Yeah really smart moves there. So one of the reasons why I invited you on was because the search team recently came out with the peak threat hunting assistant. Can you talk a little bit about that? What inspired you guys to create it?

David Bianco: 13:24

Yeah, absolutely. So the the peak framework to back up slightly has nothing in it about AI or agents or actually any specific products or whatever. I mean, it is designed to be general, any organization can use it on any technology stack, etc. and it is human driven. But one of the things that our search team has been heavily involved in for years now is, research about how to effectively use AI in security rather than just throwing AI into the mix randomly and hoping it somehow magically improves things. And doesn't break anything. Right? It doesn't break anything. Now we have, a lot of research that we've published about how to do that kind of thing effectively and safely. Let's be honest, we've we've come up with a number of key guidelines. The most important one is human in the loop at all times. You can give some autonomy or some agency to your AI systems, but not too much. You want them to be able to do things for you, but you don't want them to be able to go out of control and making that balance. Is, is very critical. So the human is the most important thing in that loop. And also threat hunting is defined as human-driven. So the human has to drive it even though they can use the AI to help them. But some of the other things that we uncovered in our research is the, like general approaches that you might think are kind of a common sense, like find your big pain points, probably the repeated things in your core workflows, and figure out what AI is actually good at and then match those up. We wanted to try out some of the things that that we had been uncovering in our research and writing about. So we actually had a couple of different things come out. We also had an LLM honeypot that used LLMs to simulate systems rather than a honeypot to simulate an L LM, and then that kind of proved it out for like straight up, I kind of I'm going to make a request and get a response and kind of things. But when we started going more into agents, we needed to do the same thing again. One of the big things with threat hunting is a lot of hunters like to jump right into the data analysis without doing a proper plan for sometimes without even doing any plan, just going to go out there and look at some data and see what I can find. You're setting yourself up for failures. The best way to ensure the success for your hunt is to do a good a good plan. But a good plan can take a long time, maybe days or possibly longer, right? So I looked at the frameworks process diagram for hypothesis space threat hunting and I thought, well, this part is the big pain point. It takes a lot of time. So much time that people skip it or don't do it very well. Maybe, they skimp on it, but it's something that really matches well with AI's strengths. So it just became a matter of like, how would we want this workflow to work if it was AI assisted but human driven? And so that's what the Peak Assistant actually does. Right?

Amy Ciminnisi: 16:51

So how do people actually go about using it?

David Bianco: 16:55

The easiest way to use it, it's an open source. So you can download it from GitHub. And and it's a we call it a bring your own model. So you can hook up any Lims in the back end that you want. We've tested it with all the frontier models. You know, OpenAI and anthropic and those, but also you can run it with, local models. So, foundation AI has our own foundation seq land that's tuned for cyber security tasks, which works great in some parts of those. Each of those little tasks has its own agent or sub agent. So you could actually mix models to see who's doing better at which ones. But you really don't have to. You can just say, hey, we're going to use all Claud, or we're going to use all GPT five or, you know, whatever you want. And then when you run it on your system, the the default interface is a local web app. So you can easily then go through and it gives you a chat kind of based interface for most of those things that you have to go through for preparation, you can start by telling it, hey, I'm interested in hunting for Kerberos staging, or I'm interested in hunting for, I don't know, log in anomalies or I am interested in hunting for, lapses or whatever, and it will go out and do the research with the addition of MXGp servers for your local environment. It can actually also do the research in your local environment, but does that in a separate agent team. So it's careful not to accidentally mix in local information in your internet searches and leak data. Right. And it and it'll lead you through the, the whole process. But at any given time you have a little chat interface. So you can ask it questions, you can give it feedback, you can correct it, you can tell it generally what you want to do. Oh, hey, you forgot that I have some Linux servers. Add those in and it will go out and, you know, do extra research or revise the plan or whatever you need to do. It even has, which is one I think is one of the coolest functions. If you provided an, an MCP server into your SIM, it can understand what data is in your SIM that would be relevant to your hunt. Identify like if it's, you know, use we test a lot with Splunk of course. So which indexes are relevant, which source types in those indexes like the types of records that are in there. And then even the key fields that are going to be relevant. So it will tailor the hunt plan exactly to your data and give you the exact searches that you need to run, even if you never told it what data is in there like it will discover it. If you do tell it, that's great. You can give it hints like that, but you don't actually need to and it can find the right data. Our test environment has something like four terabytes of data in it, and it's it's actually able to almost always find the right data. So it's really fun. And at the end, what you have is a couple of documents. Actually you have three documents that you would give to a real threat hunter to start executing. You have the internet research, you have your local, research summary, and then you have the hunt detailed hunt plan. In future, we're looking at expanding into execute and act. But right now it is a prepare phase focused tool. And it does a great job, I think, at taking the pain out of that workflow and you can complete it sometimes you can complete it in like 15 minutes. It's it's really fun. We also provide a couple of different alternate ways that I don't talk about as much because they're not as visual. But we have a full suite of command line utilities for all the different phases, so you can script them if you want. And but we also have an MC server that runs it so that you could actually put it in your own LLN systems and call the PE agents that way if you wanted to.

Amy Ciminnisi: 21:08

That's really incredible. I didn't know the details of how that worked. For listeners, I will put a blog article about the peek assistant and also the GitHub link in the show notes. But, David, for listeners who are looking to improve their threat hunting workflows, regardless of the tools that they use. What is one key piece of advice, whether it's technical or operational, that you would give them to help get started or advance where they're at?

David Bianco: 21:38

The first thing is get started thinking if you're not already doing it, like don't overthink it. You can take something really simple and start hunting for that. And then maybe the next hunt would be slightly less simple and slightly less simple. And then, you know, eventually you're you're just gaining experience. It doesn't matter. You don't have to jump to, a really complicated topic. Write it and right up front and you don't have to be like, well, I don't know how to make my own machine learning models, so I can't do threat hunting. You can start slow and easy and simple and build from there. Whether you are just threat hunting yourself or if you're trying to establish a hunting team, think of it as adding tools to your toolbox.

Amy Ciminnisi: 22:27

Yeah, the best expert hunters started where you are now, so just take that next step, get started.

David Bianco: 22:35

Nobody starts as an expert , but they all start.

Amy Ciminnisi: 22:39

Yes, absolutely. Thanks so much for listening everyone. I'll be sure to put the resources that we discussed in the show notes tune back in. And another two weeks for the next episode. And until then, stay safe out there.