This webcast was originally published on March 20th, 2024.
In this video, Whitney and Eric discuss the powerful capabilities of Velociraptor in threat hunting and incident response. They delve into its ease of deployment, versatile use-cases, and how it significantly enhances cybersecurity measures without high costs. The discussion also covers the continuous improvements and updates to their training program to keep it relevant and effective.
- Velociraptor has become a significantly easier tool to deploy and use compared to other similar tools like GRR, making it highly accessible even for organizations with limited resources.
- The transition from VM-based training environments to cloud-hosted solutions addresses the increasing challenges with running traditional virtual machines, enhancing accessibility and reducing technical barriers for students.
- Effective incident response benefits from tools like Velociraptor that can process forensic evidence directly on the endpoint, streamlining investigations and enabling quicker responses across multiple systems.
Highlights
Full Video
Transcript
Eric Capuano
So let me go ahead and just kick it off with a little of what we came here to chat about today. So, last year Whitney and I put together and ran a, training at wild, wild hacking fest called threat hunting and incident response with velociraptor.
And, if you’ve followed our work for, any, any amount of time, you’d know that we’re both huge, huge fans of velociraptor. I think by now most folks know what that tool is, or they’ve at least heard of it.
But we’ve been using it since the days of it being this silent launch of an open source tool, by Mike Cohen, out of Australia. but it’s since gained enormous popularity, got acquired by rapid seven.
And I think, again, most folks are probably, pretty familiar with it, but I would still say that even folks that have heard of it still oftentimes have no idea what this tool can do.
And when they figure it out, they just start scratching their head saying, I can’t believe this thing is free. I can’t believe this is better than my, my really, really expensive, endpoint agent.
And, and so one of the things that we love to do is spread a little bit of awareness of this tool and and also really put, put together a training environment where folks can get hands on experience with using it in an adversarial situation.
And I want to give a little bit of a backstory here because since we’ve already run this training, folks might be wondering what’s changed. Is it still the same? No, we’ve been working over the last several months to significantly overhaul the original training, which, the format that it used to be in was, it was a VM that you would download onto your system and, the VM had already been compromise, pre compromised system.
An entire ATT and CK chain has already sort of unfolded on that machine. And you would go through the steps of deploying a velociraptor server up in the cloud, generating a little MSI installer to get the velociraptor agent onto your VM and then you would start your hunt process.
And we learned something pretty, important in running that training. And it’s, I think it’s something that a lot of other trainers can probably relate to. It’s that it’s becoming harder and harder for students to run traditional virtual machines.
whether it’s because of the m m one m m two MacBooks, or just not having enough resources, not having enough disk space. But it also could have something to do, or will soon have something to do with what.
I, won’t name names, but parent companies of said products, starting to make some weird decisions about how they support and sell that product. So I’m just, I’m a little skeptical about the sustainability of the traditional virtual machines for trainings.
So one of the things that we’ve done is we’ve completely overhauled this, this course to be complete, cloud hosted. So the vms are cloud hosted. The students simply need to.
Somebody else said it. I’m glad you said it. I didn’t want to put it on the air, but yeah, so that everything’s cloud hosted. So all the student needs to be able to do is remote desktop, to the environment, and, boom, you’re right in it.
Because again, something that Whitney and I are super adamant about is lowering that bar, making it as easy and as accessible as possible. So removing the requirement to run a Vm just makes it that much easier, that much more accessible.
And so that was, a big motivator for us as we rebuilt version two of this, velociraptor course. so, what I would love to do, and by the way, I saw a message in discord that was actually very accurate, very relevant.
Now I want to come back and speak to that. So, ghost dragon. Yeah, I was talking about, ger. Google rapid response. Yes. So you’re not wrong that there is an indirect relationship between velociraptor and ger.
So Mike Cohen, who’s the creator of Velociraptor, previously did work over at Google and he was working on the GER project.
so, yeah, anybody that’s familiar with Ger, a lot of similarities between velociraptor and GR, but a lot of very distinct differences in that. I would say velociraptor is 50 times easier to deploy, easier to maintain, and easier to use than gran.
and I’ll just leave it at that. I’m not going to say that one is better than the other, as far as power and capabilities. But it is. What it really boils down to again, is lowering the bar.
If the awesome tool is very complex, hard to maintain, then nobody gets to really get value from it except for the big, big organizations with tons of resources to throw at it.
One of the things that I fell in love with about velociraptor is the bar could not be lower to use this tool. Matter of fact that, we actually demonstrate that in this lab that we spun up for our course.
You could literally deploy a velociraptor server and client on the same exact system if you needed to. No need for complex infrastructure. Now, naturally, in a more production scenario, I would deploy a little lightweight velociraptor server and then have all my clients connecting to it.
But, I mean, you’re talking about a single binary there on the server. Not some big complicated install process, but one binary. Boom, you’ve got a server that same binary, also your endpoint agent.
So it doesn’t get any easier than that. So I’m a huge fan of that. But, what I was hoping to do here today, folks, is I’m going to bring up a screen share here real fast.
And what I wanted to do is actually step through a little preview of some of the things that we’ve got going on, in this course.
This also doubles as a little bit of a, Looks like zoom just quit. I hope it’s not about to crash on me. I’m getting some zoom pop ups here saying that the client’s crashing, but I still appears.
Yeah, I still appear to be connected, but, yeah, it’s, it’s okay.
Am I back?
Whitney Champion
You’re back.
Eric Capuano
All right. I don’t know what is happening there, but that’s, that’s a good time. I hope it doesn’t do that again.
Whitney Champion
Yeah, the Q and A, and they’re saying, are you in Citrix?
Eric Capuano
Yeah, that was weird. And we did the audio video checks, screen share, and all that stuff is working just fine. Okay, let me get the screen share going again. Okay, crossing, crossing fingers.
Okay. Yep. so, all right, so, the reason I kind of wanted to step through some of this is, one, to give a little preview of kind of what’s going on in the training, but two, to also just share a little bit about the awesomeness of Velociraptor.
Right. Like, I didn’t want to bring in any slides and talk about things at an academic level. Wanted to actually demonstrate the awesomeness that is this tool. and it just so happens that this is all coming directly out of our training.
So, there’s a, there’s a little bit that we’ll have time to cover here. but, obviously a lot, lot more in our tHVR, our threat hunting with velociraptor course which is 16 hours of content. So, as you can imagine, there’s a lot happening and just for a little bit of backstory, again.
So the vm that we provide to our students is basically just a cloud hosted VM that you remote desktop into. it has been compromised. Right. So this is something that, kind of putting on our role play hat for a minute.
This VM was owned by an it administrator at an MSP Somewhere. And, this, this individual was looking to get into the field of cybersecurity.
So they were doing a lot of, research, on the Web, Reddit, following different NetSec students and cybersecurity subreddits and downloading all kinds of fun tools.
And, they’re just kind of playing around like a lot of folks do. Like a lot of students in cybersecurity do, is experimenting and trying to learn. Well, unfortunately, our victim user here, decided to post out on Reddit that they’re looking for a certain Tool and, looking to see if anybody could help them get their hands on that Tool.
And, what ends up happening is some unsavory Redditor decides to give, this poor user, some malware.
fortunately, our user here realized that something was amiss. Like, okay, I downloaded this thing this person told me was a cracked copy of Burp Suite, which is a web app penetration tool.
and that just seemed like it was too good to be true. I ran it, but nothing happened. So I went ahead and took a screenshot of that chat, because I know I’m about to have to involve the Security Team.
so we at least have this much. We know that our poor user, Bill Lumberger, downloaded this really shady Software from the Internet from a Redditor, a Reddit user, of all things.
and now he’s a little worried that he might have made a big mistake. We get a little bit of a backstory on what happened. There’s a lot more to the backstory, but, we at least have a general idea of what’s happened.
one of the cool things that we do, with this lab, because, again, our motto is make this as easy, as streamlined as possible. We didn’t want our students to have to spin up other infrastructure like a velociraptor server.
So we kept it as simple as possible. We literally just have our students deploy velociraptor right here on the windows subsystem for Linux on this VM, which is not the way that you would generally do this in production and actually whit you want to talk a little bit about why did we do it this way and how would we generally do this in a real world situation?
Whitney Champion
Yeah, as Eric mentioned briefly, we were previously running, we gave the students a VM to deploy and then they were subsequently deploying velociraptor in a cloud hosted VM, which is the typical way you’d go about this.
You would have it deployed somewhere else and not going to be sitting on the same box as your endpoint agent in most cases. but for the sake of simplicity we wanted this to be able to be running as close as possible to the VM that we’re giving the students.
So we didn’t have to provide two vms. And it’s also not running as another Windows service. It’s a little bit cleaner this way just because it’s now running in WSL as he mentioned.
And so you’re able to see the server and then you see the client check in and it’s a separate agent, whereas before these were kind of one and the same. so it made things a little less straightforward and we didn’t want it running as the agent on the same box that had the malware on it.
so it kind of has a little bit of separation there. So one of the things that we did in the lab is we provided, an optional lab that walks through our original way of doing this, which is deploy a vm in AWS.
We provide a cloud formation template that walks through, literally cut and paste steps or copy paste steps so that you can just deploy the template, run the things, have velociraptor up and running, and then deploy your endpoint agents as needed.
So that is an option still. And there’s obviously a cost associated with that because it is AWS. So it’s like a dollar a day, it’s pretty small.
but we didn’t want that to be a requirement for our course either. So you have options. but yeah, to his point, this is definitely not how we’d be doing it in production. And there’s some suggestions down the line in there I think somewhere saying, well you’d obviously want to get certs and co signing certificate certificates and put these behind protected measures and all those things just because the default is clearly not enough here.
So that’s kind of our thought process in that. but for the sake of simplicity, and this is a lab environment, it is deployed in a very insecure way. So do not do this at home when fraud.
Eric Capuano
Yeah, good stuff. Thanks, whit. So, yeah, to summarize, we made it as easy as possible, but we’re such completionists that in the course lab material we do provide instructions on, hey, if you wanted to do this the right way, the way you would do it in production, here’s an optional lab.
We’ll show you exactly how you could go about doing it again, just to keep things simple. Because really, what we wanted to focus on is less of the DevOps side of it, which fortunately, there’s not really a ton of it.
Like I mentioned to you, Velociraptor is so easy to deploy, you’ll think you did it wrong because you spin up one executable on that server and boom, it’s up and running. It’s ready to go now. Yeah, there’s a couple optional steps if you want to harden and secure that thing if it’s exposed to the Internet.
But anyway, I digress. But once we’ve got that server up and running, we got that client connected. Folks, what you now have on this endpoint is basically a blue team c two shell.
Right. I’ve got full command and control. I can gather forensic evidence. I can, process forensic. I can actually process forensic evidence in place where it sits.
Right. Because traditionally, the way we used to do that is you would gather your evidence off the system with one tool, take it offline, take it somewhere else, and then process that evidence somewhere else, which is great.
There’s, a little bit of separation of duties there. But the problem is, that’s a difficult process to scale. It’s very time consuming. How optimized can I be if I can literally process the evidence right where it sits?
Just bring back the post process data, and begin my analysis immediately. And factor in the fact that I can do this across hundreds or thousands of systems at a time.
Whitney Champion
related to what you’re saying, there’s a question in the chat about running a tool like velociraptor in the middle of an incident, and being able to deploy agents.
What problems have people had trying to run a tool like VR in the middle of an IR day job was quoted saying they’ve got 24 hours to deploy or similar post detection, but seems like a bad idea.
So it’s really easy to push these out. And it doesn’t even have to be the client server agent. We could do offline collections as well.
Eric Capuano
100%. 100%. so let me give the two part answer to that. the only real problem I’ve seen when folks are trying to quickly deploy a tool in the middle of an IR is not having the existing mechanism to deploy that software or not understanding the mechanism to deploy that software.
I’ll give an example. Some organizations have nothing more than active directory. That’s how they push out software. It can be done. You can use active directory and group policy objects in order to mass deploy an MSI and if that’s all you got then that’s what you use.
other organizations might have some kind of SCCM or configuration management tool, something like case or Bomgar. There’s a lot of these different tools, but really the problem is having gone through those steps ahead of time and proving the capability and saying ok we know that in an IR we can deploy this EDR or this velociraptor or whatever using Bomgar or whatever your rmm m agent is.
go through the steps and test it, prove that it can be done. So that when the IR happens really I’ll tell you, your target for deploying an agent in an IR is not 24 hours, it’s, it’s, it’s going to be 30 minutes or less.
Like that’s generally the speed that you want to be able to move it. But but to Whitney’s point, the other part of this is what’s really cool about Velociraptor is if, if time is of the essence and I want to take advantage of velociraptor without having a server deployed ahead of time, you can absolutely run velociraptor in a standalone way.
Basically all the cool things that I can do with it. All these interesting hunts that I’m about to show you, like gathering running processes, yara scanning processes in memory for cobalt strike processing, prefetch files, or gathering all the triage evidence into one zip.
These and many many other things can be done with the standalone velociraptor executable without ever deploying a server. Even if I’m back in the nineties walking up to a system with a thumb drive, I can still get a lot of awesome things done with velociraptor running it as a standalone binary, which is just incredible that everything that it can do with its client server deployment it can also do by itself standalone.
I’m not installing anything, I’m just running a portable executable, gathering up the evidence and packaging it up and taking it elsewhere. So here’s what I want to actually go through.
I want to step through a handful of hunts that we’ve run across, a number of systems in this compromised environment to showcase some of the awesomeness of this tool.
So, what we have in the course is we’ve got, obviously, the one vm that you connect to. is this vm that I’m RDP’d into right now.
That’s the only one that’s online. But we actually have, imported hunts from about nine other systems, actually, a total of ten other systems.
And that’s going to give us the opportunity to emulate a little bit of what it would look like to hunt across many systems at scale, because that’s one of the really cool features of Velociraptor, is not only can I interrogate a dozen systems or more, but I can come back and process the data in a really easy way.
And so I wanted to be able to showcase that. And so that’s why we’ve got several hunts in here that were run across ten systems. You can kind of see over here on the right. I just want to ask real quick, is my screen share coming through smoothly and staying smoothly?
Because I’m getting some flickering on my side. I just want to make sure that that’s not coming through to you guys. Okay, perfect. So, so notice we’ve got a handful of shares, a handful of, tens over here that’s indicating that these hunts were run against ten systems.
But really, only one of these systems is online. And I’ve got a couple of hunts that I only ran against that one system. But I’m going to start with these hunts that were run against multiple systems, because what this allows me to showcase is the power of the velociraptor notebooks, for instance.
This hunt that was run here is called the windows system. artifact. Just to give you an idea of what’s going on here, this artifact is probably one of the simplest, one of the easiest ones that you’d run.
When you’re interrogating endpoints, you’re literally just asking it to list all those processes, all the running processes and their associated binaries. But it does add some cool enrichment to it, because not only is it going to bring back the process list, but it’s going to enrich it with information like the hash of each of those executables, as well as the authentic code status of those executables, which threat hunters already know where we’re going with that.
Right. that enrichment enables me to very quickly kind of make heads or tails of which of these processes is less likely to be malware versus the one that might be more worthy of scrutiny.
Right. By checking if it’s digitally signed, is it signed by a trusted publisher? Is the signature even trusted by this system’s root cas and all that kind of good stuff?
but that’s generally what this artifact is doing, so it’s about as simple as it gets. Right. Just basically show me a process list. So when I cruise over here to the notebook, this is where all the results for this hunt come back, which, with my font size right now, my zoom level.
It’s a little painful to look at this because the columns are kind of squeezed together. But I’m going to show you kind of how we, how we work with this data here in just a moment. What I really want to show you, though, if I scroll down to the bottom here, is there’s, there’s.
There’s many, many, many pages of this, because I basically asked it to show me every running process on ten different systems. If I remove this limit here and then scroll down here to the bottom, you can see the total number of results.
So it’s just under 800. So there’s just, under 800 processes running on all these systems. So, as this is what makes incident response a challenge, is that you are going to drown in the data that comes back from just about any question that you could ask.
If I say, show me your running processes across 100 systems, I need to have some means, some method to reduce that data down to what I probably most likely care about.
This is what we use the notebook for, because I don’t want to scroll through 800 rows of data. So what I’m going to do is I’m going to use the notebook to tweak what data is being returned to me and how it’s being presented.
I’ve got a couple notebook examples here queued up. These come right from our lab guide. It’s ways to reduce the data. So, for instance, this next notebook here is simply saying, hey, look, first of all, let’s clean up the columns that we’re going to return, because I don’t need all those columns.
I only care about the columns that give me the most valuable information. Things like the name of the process, its executable path, the command line arguments, only the SHA 256 hash, because I don’t need three different hashes.
and whether or not it is authentic code trusted. That’s really the subfield of the authentic code metadata. That’s going to tell me the most important thing about this process from an authentic code status, which is, is it digitally signed or not?
That’s really all I care about. And is that signature trusted on this machine? And sure, let me know the username it’s running as and the fully qualified domain name of the machine. But notice this next little, filter criteria here is saying only return the rows where authentic code trusted is equal to untrusted.
So in other words, I only care about the unsigned processes across all ten of these systems. Now this is super cool, because remember, we had nearly 800 processes across these ten machines.
when, I asked the question of which of these are untrusted or unsigned, only one comes back. So that immediately tells me that unsigned processes are fairly uncommon on modern Windows systems.
These were windows ten systems, but still, they’re even less common on Windows eleven. So this is important background knowledge for anybody doing soccer IR threat hunting work is just knowing that the prevalence of unsigned code is becoming less and less and less and less, which is why we love to use it when hunting for malware.
Because guess what? Malware will almost always. Okay, we don’t like to speak in absolutes in this business, but malware will almost always be unsigned.
Why? Because it’s a really, really difficult thing to get your hands on a code signing certificate. Not impossible, it has happened before. But I’m talking folks like, unless you’re generally in the organization that’s targeted by advanced threat actors, you’re not likely to ever encounter digitally signed malware.
This is just a very rare thing. Something like less than 3% of malware samples out there. But that said, what I’ve got here is I’ve got a single unsigned process here, and we could do a little osint on that.
I could throw that hash against virustotal to kind of ask the question like, is this malware? and what I’d find is, no, it’s actually not. This one untrusted process is not malware.
It’s actually a well known, windows background binary. It just so happens to be the only one that’s unsigned.
Whitney Champion
But.
Eric Capuano
Ok, got it. So I’ve now answered a key question here. I’ve understood the prevalence of unsigned binaries, and I understand that I’m not looking at an unsigned binary across these ten machines.
Now that doesn’t mean I don’t have malware. It just means that it’s not an unsigned process that I’m looking, for here. continuing through this notebook, though, let’s ask a couple additional questions.
This next one here is clever, because since I didn’t find anything interesting by looking for untrusted processes, the next type of activity that I would use is what we call stacking.
Now I’m coming at it from, show me the most rare processes. The processes that out of all of these systems is only running on maybe one or two of them.
I’m basically doing a, grouping exercise here where I group all the processes based on their executable path and their command line arguments. For all the ones that are the same.
Go ahead and stack those up and just count them so I know how many of them there are, and then we’re going to order it by that count, which is going to put the most rare entries at the top of my list here.
So it’s a quick and easy, question to ask. Show me the least frequently occurring thing across all these machines, because we’d consider that an outlier.
we get some progress here because, we’re down to, we’ve got 145 rows. But really, what all I really care about, if you look at the column all the way on the far right, all I really care about is the count here, because if I, if I see a process running six, seven, 8910 times across ten machines, it’s probably not the thing that I’m looking for from m an anomaly perspective, right?
Or from a least frequency of occurrence perspective. So I’m really interested, though, in anything with a count of one. Now, as you see here, there’s more than a few of those, there’s, more than a few of these entries that only came back with a count of one.
But one of these is ripe for removing from my results here. And I’ll talk through that for a moment here. We see a lot of these svchost k, entries here.
These are svchost executions from the correct path where svchost lives. And by the way, folks, if you’re ever unsure about that kind of data, I don’t know where svchost lives.
A, really cool resource. I got a plug is, a tool called Echo trail. Echo trail is kind of like, the inverse of virustotal. You go to virustotal to see if something is malware, but you go to echo trail to learn more about a legitimate thing, like something like svchost.
And when I search for svchost here in echo trail, it’s going to show me, hey, here’s the top path that svchost is observed in. So generally, as you can see, 99.9% of the time svchost lives in C Windows system 32.
So that’s a quick way to answer that question, like, okay, well, is this the normal location for svchost as you see it is. The other thing here though, that I’m looking for is how well do I understand what the expected command line arguments are for svchost?
That takes a little bit of time and background and understanding of saying, okay, well, svchost most often is used to run services on the system, and you’re oftentimes going to see this, almost always going to see this dash k command line argument followed by a network group name.
And there’s some other, tribal knowledge, things that you’d want to spend some time with. But what I’m getting at here is the reason that these are appearing to be unique or anomalous is the fact that these command line arguments can vary quite a bit across these systems.
So those are not really the ones that I’m looking for. But it doesn’t take long, folks, though, to focus in on things in here that might be worth digging into.
but for sake of time, I’ll press on. But one of the next things we’d probably do is filter out some of the noisier, false positive prone things like those svchost, hits.
I’m going to basically just tweak my notebook here. Notice we’re filtering out the svchost exe ks because these are just not immediately actionable to us.
But I’m also filtering out, based on what I would have had to spend a little time doing, some of the other benign and noisier things, things like edge and Windows apps and things like that.
Adobe, but this again would take some time to baseline and understand what’s normal for this environment. But once I filtered down and I still have my frequency counts over here, it’s pretty easy now to rule out.
Okay, so now I’m down to just a handful of entries that came back with a count of one, and we’re kind of staring at one right here that would probably be worth looking into.
and again, I’m building off of a lot of assumed knowledge that we cover in this course, right? We talk about anomalous, executions of lol bins like run dll 32.
And what becomes anomalous about this execution right here is the path to an oddly located, dll file. So we got a DLL and a user directory and appdata local temp, and run DlL 32 is being used to execute that, which would be worth additional scrutiny.
Now, notice, the reason we didn’t see this previously, even though I’m telling you that it’s anomalous, is because it’s an LoL bin. This is a living off the land binary. Run DLL 32 is a trusted process.
That’s what makes things a little bit tricky, right? Because if only I could say all adversary tools will be unsigned, or almost all adversary tools will be unsigned, that would be great, because then I could have just used that first notebook and we would have found adversaries.
It’d been easy and we’d be done. But sadly, you have to account for the fact that adversaries will often abuse living on the land binaries, which are digitally signed and they’re digitally signed by Microsoft in most cases, which can make this even trickier.
To do so, it takes additional understanding of, okay, what are those most common lol bins? How are they abused? And, another project I’ve got to plug is the lol bas project, because this is like echo trail in a way, where I can go and learn about legitimate binaries and I could better understand how they can be abused by threat actors.
So I could go down here and find run Dll 32, dig a little bit deeper into, the known attacker tactic, techniques, procedures that have been known to abuse, this lol bin.
But I can also get a little bit of background understanding as to what is it used for. It’s used to execute Dll files. I’m not sure how well my zoom is looking here, but it’s used to execute Dll files and generally in the command line arguments for run Dll.
I’m going to see the path to a Dll, or just the name of a well known Dll. And then I’ll see, an entry point or a function name here next to it. I can very quickly learn a little bit about this, even if I don’t know much to begin with, and then I can come back and understand.
Okay, so this run dll 32 is executing this Dll, and this right here would be the function name or the entry point that it’s calling.
But the reason that I’m going to draw some scrutiny to this is that’s a really weird place for a DLL to be. DLL’s generally hang out in a handful of well known locations. I don’t generally like to see them in temporary locations or user, profile locations.
And this is both. This is a user profile and it also is a temporary location within that user profile. So I’m going to spend some time I’d be looking for. I want to go get my hands on that DLL and better understand it.
But this is cool because just by using the stacking, looking for the least frequently occurring things, we were able to identify an outlier, an anomaly, which is going to be a thread that we can begin to pool on to continue asking more targeted questions like is this DLL present on any other systems?
let’s go pull the hash of that DLL, let’s see if we can learn about it. Maybe we download it, do some reverse engineering, and the list goes on. But ok, I’m going to move on to another artifact that we talk about in our course.
We also get into some of the traditional forensics, artifacts that we can get a lot of important, valuable information from, like evidence of execution artifacts.
The appcompat cache, also known as the shim cache, is one of those types of artifacts where we can learn about the presence of binaries on a system.
Now, there’s a lot of nuance to this artifact that I don’t have time to really go over, but we talk about it in our training. But one of the cool things that I can learn from this artifact is I can find out relatively quickly, what binaries were added to this system, were introduced to this system during the time that the adversary was most active or present on the box.
And I actually have kind of a precooked version, of this right here. So what we learn is by looking at the shim cache, we learn there are a number of suspiciously named or suspiciously located executables that were all introduced to this machine around the time of the other initial findings.
We learn about some weird single letter executables here sitting in c windows temp notice. let me just show you how simple this notebook is, folks. I didn’t do anything fancy to this notebook.
this is the default notebook for this view, for this artifact. All I did was I cleaned up the rows, I just had it bring back just the position, the modification time and the path.
That’s all I care about here. And notice I don’t have to go very far to find some really interesting things in here, like the windows backup location, apparently containing an executable here, another executable, which by the way, the name of that executable is probably making any incident responders itwitch right now.
if what our clone is most commonly used for, very, very popular exfil tool, folks, I’ll just say that. So when you start to when you start to see things like this, right, it’s going to make, the hairs stand up on the back of your neck because you immediately start to know what we might be dealing with here, right.
We also see, interestingly enough, we see a what you would consider to be a normal binary spchost exe. What’s really interesting about this one is its path, its location.
So again, if I didn’t already know about this through my other previous analysis, I’m instantly getting additional pivot points. That’s one of the topics that we spend a lot of time talking about are throughout your investigation you were constantly extracting little nuggets of data that become pivot points for further expanding your view of what’s going on.
So with each one of these interesting findings, I’m now going to go ask questions of the rest of my systems. Show me any other system with the C Windows backup folder because that folder is not even a legitimate directory, that’s not part of a Windows install.
I’d want to know more about if that’s a known attacker working directory. Show me anywhere else I can find it in my network. Show me anywhere else I find an r clone executable because that’s not approved software, that’s not something we legitimately use.
We could do one quick Google search and find that, a lot of ransomware and extortion, actors are using tools like r clone right now because it’s basically this multi cloud backup tool.
R Clone will basically connect to Google Drive, Dropbox, mega, all these different cloud providers. And it’s a command line utility that syncs files to any one of those cloud destinations.
Really, really popular tool. As soon as I learn about these things folks, I’ve got a handful of pivot points I can use to go learn more about what’s going on in this environment. Figure out what my scope is, figure out how many systems are impacted and affected and the list goes on.
But let’s see, wanted to also talk about for instance, baselining persistence mechanisms and things like that because this is just another one of the necessary steps in an IR is to understand like okay, so even if we find all the threat actors malware and we find the systems that they’ve impacted, we’ve even found this, the evidence of exfil and all that kind of good stuff.
We can’t wrap up this Ir until we know for sure the threat actor is gone, eradicated, completely removed from the system. Right. And one of the ways that threat actors keep a foothold in your environment is through the use of persistence mechanisms.
And there are many of these, and we’re only going to talk about one of them, here for the sake of time. But services are a popular mechanism that they like to use because they can create a service on an endpoint that’s going to start automatically at boot.
And from that day forward, even if you reboot that system, they’re going to maintain a foothold on the machine every time that system reboots. One of the cool things we can do is we can ask the question of, show me all the services across all the systems in this environment.
Just to give you an idea of what we’re working with here. Get a total count. We’ve got 2600 services across ten systems, folks. You see how easily threat actors can hide in these things because nobody knows how many services exist on a legitimate system.
It’s well over 100, and nobody knows what they all do. So this is a perfect place to hide for an adversary to create that persistence mechanism, because there’s a lot of noise, there’s a lot of stuff in here that they’re just going to blend right in.
So you already generally know the approach that I like to take when I’ve got a large data set like this. One of the first things I’m going to do is I’m going to ask the question of, show me the rarest ones, show me the least frequently occurring ones across these ten systems.
Also, I only really care about the ones that are going to start automatically at boot because there’s a lot of services that are disabled, they’re not even in use, which would mean they couldn’t possibly be a persistence mechanism because they’re not actually going to do anything when the system reboots.
So only show me the ones that are going to start automatically. Group them where the name, the display name and the path to the binary are the same. Group them, stack them and count them.
Right. So again, you already know where we’re going with this. I’m going to get back now. The least frequently occurring, it’s that, that count of one there is really what I’m looking at.
But here’s the thing, we’ve got a lot of those, the counts of one. And if you recall what we talked about a little bit earlier, you might immediately know why. If you notice, the overwhelming majority of these svchost exe ks, and the reason they come back as unique is because of the user mode services that, we saw introduced in Windows ten, where the same service by the same name will look unique because of the presence of this random hash at the end.
And this just has to do with how Windows ten is able to run the same service multiple times, depending on, how many users are logged in. So again, in the name of data reduction, I’m going to leverage that knowledge to filter out those really noisy false positives where I’m not so much worried about the SVC hosts right now.
And let me just pause for a second and say that folks, sometimes we got to put on the tinfoil hat and say, but wait, what if an adversary took advantage of the fact that we tend to filter out SVchost, ks, et cetera, et cetera.
Could they use this against me? They absolutely could. Right? So, and this is true about anything. What I’m trying to show you here is kind of the Pareto principle of, hey, here’s the, here’s the 80% that’ll get you there, right?
But there’s always going to be that edge case of, wait a minute, sophisticated threat actors are doing sophisticated things to blend in and hide. So you just have to kind of approach with caution and be mindful of that.
But let’s not let, perfect be the enemy of good here. So doing, once again, the stacking exercise, folks, check this out. Just by filtering out the svchostash case, I’m left with three entries that have a count of one.
Notice the rest of them all have a count of ten because those services are consistently on all ten machines. So that’s not much to look through here. And furthermore, there’s another pretty surefire giveaway that this one here might be the one that we care about.
And, that path name. There is also something that we previously observed in one of the previous labs. We’ve got this svchost exe sitting in an unusual location for svchost.
See Windows services, which is not a legitimate directory on a Windows box. It’s convincing, though. Looks legitimate to the untrained eye. but noticing the service, name MS update display name Ms M update, it’s trying really well to blend in.
And I’ll be honest with you all, folks, this would blend in to a lot of, and it system administrator would jump on this box and not think, not, not even think twice about an entry like this because svchost is one of the most commonly abused binaries because it runs many, many times on an endpoint.
It’s a very noisy thing. I’m actually curious if I bring up my task manager here, just. Just kind of going through and looking for sVchost. I might have to.
Oh, it’s going to be nested. I hate that they redid that. Yeah. So you see all these service hosts, these are all SVC host processes on this machine, folks. You see how hard it is to get in there and pinpoint, like, which of these is legitimate?
Well, there’s too many of them to count. So the point is, this is a commonly abused lol bin for that reason. But how did we find it, right? This is not rocket science. And yes, somebody, asked if the hash would be different.
Yes, the hash would be different, because this is not a legitimate SVC host at all. and again, if I’m ever in doubt, I can always just ask the question if open source intelligence has this file.
Oh, well, hey, mystery solved, case closed. Like, yeah, I query the hash, and apparently 46 avengers say, this is no good, again.
Now, there are more than one ways to get persistence on the box. We do cover many more in our training, but just, showing you again how that baselining can work and how we use least frequently occurring events to find, those outliers.
We do the same thing with scheduled tasks, WMI, event consumers, you, name it. Lots, of really cool things. Drivers are even a really cool one to do that with.
But, I do want to talk about another really cool forensic artifact that, we focus on. It’s another evidence of execution thing called prefetch.
What’s really cool about prefetch is it’s a way of identifying the order that things have executed on a windows system. What we end up doing with this is we come in here and we add a query for one of the initial indicators that we know about that was shared with us from the user of this system, the victim, if you will.
When they contacted the security team, they said, hey, look, I made a mistake. I opened up this thing from the Internet, and I think it might be malware. We at least had a little bit of a thread to pull on.
We knew it was something to do with the word burp suite. Was the name of the executable. What I’m simply asking of, my zoom is crashing again. You got to be kidding me.
Let’s see if it’s going to recover. I mean, it’s still running, so I guess I’m okay. So I’m asking the question of our prefetch artifact here. Show me all of those evidence of execution artifacts.
where burp suite is found. And what it does is it brings back a specific moment in time. Here’s where you see, here’s where you see that execution of burp suite.
So we know it did in fact run. The malware. The thing that is suspected to be malware did in fact run. We know exactly the moment that it ran. Right. We could see it happened on 312 1830 611, Zulu.
So this is UTC or UCT. that’s a fantastic data point for me right now, because, yes, I know that the potential malware ran, but I know exactly when it ran.
So now as I’m diving into larger evidence sets, I have an ideal pivot point. I don’t necessarily care about anything before that point in time, because when I start diving into the evidence, I’m going to be overwhelmed very quickly with how much data there is to see.
So the fact that I now know that moment in time that the initial malware ran lets me focus in on the timeframe that matters most. And that’s exactly what we do in this next query, is we say, ok, we now know when that malware ran.
So let’s now ask the question. Show me everything that happened. Yes, immediately before that burp suite execution. But really what I care most about is what happens after that.
Show me everything that happens on the machine after the fact. Now, folks, let me just throw out a quick, sanity check. You might be thinking to yourself, isn’t this exactly what EDR is supposed to tell us and what our seam is supposed to tell us?
Yes. Yes it is. if you’ve been working IR for any amount of time, especially as a consultant, that most often you drop into an environment there likely is no top shelf EDR to use, there is no top shelf seam product for you to dive into.
So a lot of times we’re left with going through traditional forensic artifacts to answer some of these key questions. So that’s exactly what we’re doing right now. We don’t have any kind of fancy tool to use.
Well, actually we do. It’s velociraptor. Right. But we’re having to basically retroactively tell the story of what happened here because we don’t have that telemetry. In hindsight, once that burp suite pro execution runs, we see lots of things happening on the system.
Now, the tricky thing about this is I won’t exactly know immediately which of these things are adversarial. Prefetch is just simply telling me these are the binaries that ran. But I can use temporal proximity to kind of understand.
Okay. It is possible that anything that happened after this could be related to the malware, because it happened after the malware ran. Right. So just looking for things that stand out as.
Whitney Champion
Huh?
Eric Capuano
Huh. I want to know more about why that happened. Right? And folks, whoami exe, really? This is like adversary 101. Pen testers and threat actors alike all love to run.
Who am I? Because it’s the quickest way to figure out what privileges they have on the system currently, what user contacts they’re running as. Now. Again, who am I? Is not a malicious thing.
I’m just saying that it’s oftentimes a behavior of things that are up to no good, but we keep scrolling. There’s some other things, like task list is a way to enumerate running processes on a system.
And looking at the simultaneous, execution, it looks like task list was run at the same exact time as find string, which is a common behavior, you see, when find string is used to analyze the output of another command like task list.
So I’m just going to go out on a limb and say this is something that you might see when someone’s trying to look for specific processes running on a machine like EDR, antivirus, what have you.
So, I mean, folks, the list goes on. We could talk about this, for a while, but SC Exe is used to create, manipulate managed services. Could be the creation of a persistence mechanism, which we actually just saw, right?
Schedgetasks Exe used to create, manipulate, delete scheduled tasks. We might be looking for a scheduled task now, right? And then we see some svchost executions, which we already know.
There’s some relationship here to svchost. but then the list goes on, and it will start to taper off into, hey, there’s a lot of other things happening on this machine that may or may not be related, but it’s just kind of cool how a single artifact can start, to start, to reveal, where we should be looking.
But folks, what really gets interesting, one of my favorite things to talk about with the prefetch is when we have the ability to parse the internal data contained within prefetch.
Folks, this is where things get really freaking cool, because prefetch not only shows that a program executed, it also will show you all of the files and folders that that program interacted with within approximately 10 seconds of execution.
All right, you might see where I’m going with this. For instance, if I know which programs I believe to be adversarial, like any one of those interesting single letter executables we saw.
B exe, c exe, p dot exe, r clone is a big one here. I just basically drafted up a quick regular expression to say, show me all of the prefetch parsed entries for these executables.
What that’s going to show me here is every single file or folder that was accessed by these executables during that ten second period of time that they ran.
Folks, this is like mind blowing, again, using forensic data to answer the question of what happened. This happened a month ago, but I can still go back and use this evidence to say this b exe, what was it likely doing?
Folks, let me just throw it out there that this takes time and it takes a deep understanding of some of these things. But what I could start to put together as a theory here is whatever b exe is, it sure seemed to be interested in the web browser history of multiple browsers on this system.
Mozilla, Firefox, Microsoft Edge, Google Chrome. It’s going after their web history files here. And oddly enough, it also accessed a file in C Windows temp, which is the same place that the b exe was stored, by the way.
C windows temp. It accessed a file called one txt. Folks, if I was, a gambler, I’d put a couple bucks on the fact that it probably wrote whatever it did, it wrote it out to that one txt file.
I now know where I’m going next. I’m going to go look at that. But hey, I do want to show you one other really cool thing. and it’s that r clone utility, that r clone command line based tool that’s going to exfill files from our environment here.
I would love to know if the prefetch might have captured some of the files that were exfilled.
There’s some really, really cool findings in here, folks. And obviously I’m moving quickly over what we spend a lot of time on in this course we go through and do some really cool cyber chef stuff to process this data very quickly, to filter just for the things of interest, to pivot on the things that, we’ve identified, like the windows backup location.
We share with you several quick tips and ways to get to these answers in a quicker way and to really, reiterate the understanding of what the data is trying to tell us here, but SD is another weird executable we see in the course or we see in the evidence ends, up being a, well I won’t spoil it.
We’ll talk, we’ll save that for folks that want to go through it and learn a little bit about it. But man, we’re coming up on time. Time, flies. So let me, let me cover a last couple of cool items.
But one of the things we also, we talk about in the course and folks, so much of the background of this course, I know I’m doing all the talking, but so much of the background and the heavy lifting of this course was, was Whitney.
And so a lot of the things that we do early on in the labs is we, we leverage Velociraptor to deploy agents like Sysmon, right, to start shipping real time streaming events from m the system.
Like those sysmon logs, right? And we do a lot of exercises where we rack and stack those real time events and we go through, we look for threat actors, we look for adversarial behavior.
And yes, it’s there because one cool thing I’ll share with you about this VM, it’s not only been hacked and pwned historically, it’s pwned right now.
There’s an active threat actor on the VM right now. Well, when I say active, it’s scripted elsewhere, but to the student it looks as real as anything else.
There’s background activities happening and an active c two shell. And we navigate that by looking at the streaming real time events. And we see is there any of this stuff still going on right now?
And where, and furthermore, how does the threat actor still have a foothold? What process are they injected into? Are they using a particular well known c two beacon like cobalt strike?
And we answer all these questions we go through using the power velociraptor to very quickly. For instance, Yara scan for cobalt strike in memory.
Find it exactly where it is injected into one of our lol bins. And we even go so far, this one’s a really cool one, we even go so far as to extract the beacon configuration.
We extract and decode the cobalt strike beacon configuration to learn about its c two servers, the hard coded user agent all the way down to the injector, the injection, lol bins that it’s basically configured to inject into.
And we expand on all this and it’s just a really, really good time, folks. So we’re coming up on time that went a lot quicker than I thought it was going to.
if any of that seems interesting or exciting to any of you folks, really, hope you’ll keep an eye out for, our on demand course that we’re working on right now.
We are putting together an on demand version of this for anti siphon and we are also hoping, to submit and hopefully get accepted to return to wild, wild hacking fest this year where we’ll be teaching it in person.
It’s a ton of fun, folks, and we hope to see you there. But, all right, I guess I’ll go ahead and wrap up and sorry if I missed any questions.
forgive me, it’s limited screen real estate.
Zach Hill
No, I was just going to say, we did have a bunch of different questions coming in, but it looked like, whitney was tackling a lot of them. So thank you, Whitney, for being on top of that. It’s awesome.
And, thank you, both of you, for being here with us today and sharing your knowledge. There’s a lot of great feedback from everybody. So you guys did phenomenal.
Eric Capuano
Great to see one last cool thing I want to plug. And this is something that’s brand new to the course. It’s brand new to velociraptor is, recently they added the ability to run basically sigma detection rules against the logs on your endpoint.
And it’s the coolest freaking thing ever. So you’re getting basically retroactive detections on event logs using the sigma rule set and the hayabusa ruleset. and that’s a pretty, pretty brand, new feature that we’re going to be rolling into the course.
Zach Hill
Awesome. So cool, man. We got have people here rethinking coming to wild west hacking fest just to come take your course. So that’s always good to hear. Appreciate that. and for everybody who’s here, if you guys don’t have any other questions, about.
For anything, for eric and or whitney, you want to ask us questions, anti siphon training about our training or anything security related, certification, career related, you can join us in the breakout room.
So at the bottom of your zoom screen, you should see a button down there that says breakout rooms. You can join Kyle and Brian over there and I’ll be joining them shortly and they can answer any questions that you do have.
Awesome. Eric and whitney, do you all have any parting words before we leave for the day?
Eric Capuano
Wait.
Zach Hill
You gotta love it.
Whitney Champion
I’ve been heads down answering a million questions. This was fun and it’s awesome to see so much interest around this. So looking forward to our on demand and hopefully wild west hack investing.
Zach Hill
Somebody asked when the conference. So, wildlife hack and fest is, not August, October 8, 9th, 10th? I believe I could be wrong. I think it’s for sure like the.
I should know this off the top of my head, Jeff, I’ll just say.
Eric Capuano
For anybody that hasn’t been so Whitney and I have wanted to go for years and we finally got the opportunity to go last year, and I gotta say, it just made me, it just made me regret that, that I was just now getting to go.
because, getting out there to that venue, it’s such a unique experience. It’s such a beautiful location, and it’s a little tricky because it’s like an hour from the airport and stuff, but once you get out there, you’re like, I don’t want to leave.
Can we make this longer? Like, this is amazing. Like the venue, the hotel, the little town. But more importantly, the quality of the trainings. The other trainers that I met out there were clearly top notch.
And it was just so awesome to know that this training was being made available to folks at this conference. the keynote speakers, all the other speakers, like, folks, this is easily one of my favorite conferences now, hands down.
And my goal, my plan is to be there every year that I can. So if you have not been, you must attend Wildwall aging fest.
Zach Hill
I dont think I could have said that any better. I mean, its like the same stuff that I say about it. Ive went the last two years, this will be my first year going as an employee, so it’s going to be a little bit different.
But like, I, I cannot, like, I just agree with everything that you said there. It’s such a unique experience. And I asked a lot of people last year, I was like, because they had a lot of different teams there. I was like, how come you guys all come out here and they’re like, we come out for the training?
I was like, that’s really interesting to hear. it’s like here we are in the middle of nowhere. but you have all these like really intelligent, like credibly, like gifted and like caring, compassionate, like, people within cybersecurity just coming together and helping each other out.
It’s such a cool thing to see. So thank you for sharing your experience, and thank you again for, for coming here today. And I look forward to seeing you out in Deadwood this year. It’s gonna be a lot of fun, both of you.
Eric Capuano
Yeah, we’re stoked.
Zach Hill
Awesome.
Eric Capuano
Cool.
Zach Hill
Well, thank you again for joining us. We’ll be back here again, same time, same place. Next week, so please come and join us. And again, if you have any questions for us anti siphon training, please join us in the breakout room.
Otherwise, y’all can have a fantastic day. Take care, everybody.
Eric Capuano
Take care.