This webcast was originally published on March 10, 2020.
In this video, the speaker discusses the intricacies of incident response, highlighting the common pitfalls and emphasizing the importance of preparation. They critique the use of flowcharts in incident response, arguing that they are ineffective and often lead to paralysis when deviations occur. Instead, the speaker advocates for a flexible, well-practiced plan that utilizes documented procedures like Lego bricks, which can be reassembled to address the specific challenges of an incident.
- Effective incident response requires preparedness, not just pre-defined flowcharts.
- Adopting a flexible and modular approach to incident response is crucial, akin to assembling Lego bricks for various scenarios.
- Preparation and regular drilling in incident response are essential to avoid panic and mistakes during actual incidents.
Highlights
Full Video
Transcript
John Strand
All right, everybody, hello and welcome to our conference on. You are compromised. Now what also known as the conference that I literally wrote while I was dealing with basically canceling the ground based portion of Wild west hacking Fest in San Diego.
So I’m going to go right out there and say, hey, this isn’t my best work. I knew what I wanted to do. I got all the things that I wanted to cover covered in this particular webcast, but it just doesn’t have the quality of the memes and the knock knock and fart jokes that you guys have come to depend on Black Hills information security to provide for these conferences.
Now, a couple of quick things. If you’re joining and you want me to talk about Wild west hacking Fest, I will at the end of the webcast. All right, so let’s get started. So what’s going on right now is I’m kind of taking a look at Black Hills information security and what we have documented in the form of webcasts and blogs and short videos.
And I’m trying to develop a whole bunch of content that would be, introductory level content for free. If, you go and you check out our YouTube channel, there’s a bunch of five to ten minute videos on a number of different topics, from using Rita to using wireshark to doing recon to all these different things.
And there’s some topics that I can’t do into just a simple ten minute video. And we’re talking about incident response. It’s absolutely impossible to cover all the aspects of incident response in a simple ten minute video.
So I wanted to set up a longer video that we could discuss the finer points of preparation and also how you can actually start your incident response process. So first stop, first steps are incredibly tough.
And many times with incident response, we end up with massive mistakes and paralysis. And almost always, whenever you’re looking at massive mistakes and paralysis, it almost always stems from the fact that people are not prepared.
They have not thought about it, they have not drilled, they have not practiced, they have no frame of reference where they can actually start working through the incident to handle it properly. And with an incident, it’s always critical that you can keep moving, but you keep constantly moving forward and working through the incident.
And for that, many times you need to have a plan. But I’m going to have to cover some basic first steps. So the first thing I want to talk about is the wrong way of doing incident response.
This is one of those areas teaching for the Sans Institute. Over the past few years, I’ve gotten into a handful of arguments. But many organizations like to establish flowcharts for incident response, where they say, we received this call, and if it’s this particular type of call saying the system’s not performing correctly, then we’re going to do these activities.
If we received this call that said that there was dancing gerbils that popped up on the screen, we’re going to do these particular steps, and they set up these really detailed flowcharts of how to handle the incident.
This is incorrect. Do not do incident response through floor charts, ever. It’s never, ever, ever a good idea. Just don’t do it, please. And the reason why it’s a bad idea is almost universally the people that create the flowcharts for incident response are people that have never done incident response.
So they come up with very weird ideas on how things are supposed to be done. The other reason why it’s an incredibly bad idea is because whenever you give someone a flowchart, they will follow it.
And what that means is anytime that there’s any types of deviation or things that don’t match the flowchart, people once again go into a state of paralysis. The point of coming up with Flowcharts works great for developing communication plans.
It works great for developing user markup language modeling for code languages. But whenever you’re dealing with a very fluid situation, such as an incident or computer incident in your environment, it is exceedingly difficult for you to handle a flowchart or use a flowchart to handle that.
The greatest quote in this is, plans are useless, but planning is indispensable. So that came from Dwight Eisenhower. And, we’re basically looking at the whole concept of coming up with the detailed flowchart of how you deal with a confrontational situation is ineffective, because the first thing that you throw away whenever you come in contact with the enemy is your plan, but planning is indispensable.
These two statements really seem to be in conflict with each other. You have one that says, plans are useless, but planning is indispensable. Aren’t they the same thing? No, they are absolutely not the same thing.
Whenever you’re developing a plan, you’re developing a detailed series of steps that are to be followed. However, if you’re working with planning, I want you to think of Lego, bricks.
Now, the reason why I want you to think of Lego bricks, I, keep saying blix, and I don’t know why Hans Blix, sorry, Team America.
Reference. The reason why a Lego block really works for this is if you look at incident response as a series of technical capabilities and procedures that you have documented in your organization.
You’re actually thinking about it the right way. If you look at Windows live forensics, well that’s a brick. If you look at Linux Live forensics, that’s another little brick.
If you look at Unix, web server analytics, that’s going to be another brick. System isolation, that’s a brick. Firewall analysis, that’s a brick. Read up, that’s a brick.
All of these different things that you can use should have documented procedures that you can follow on how to use that very specific technical component.
Now the reason why this is essential is all of these things can exist independent of each other. I can develop procedures for workstation isolation. I can develop procedures for web server log review.
I can develop a procedure for standard server analysis on a Linux system. And they’re discrete unique chunks of functionality and training and step by step procedures that can be used in an incident.
Now the Lego analogy really also flows into the movie. If you remember in the movie there were movies, there was Elliot, the main character who was a master builder and Batman was a master builder.
And you had all of these different master builders. My favorite person was the spaceship guy that was just like spaceship. And he would go crazy and they could take these bricks and they could reassemble them to build something that would meet the challenge that they were encountering in the movie itself.
Well, that also applies for incident response. So when you’re looking at incident response as you have all these little detailed procedures on how to use Rita and how to do log analysis and splunk, and you can then tie them together to basically match your incident response procedures and what you’re going to do technically to match the attack that you are dealing with.
This means that you’re flexible, there’s planning involved, there’s training involved, and you’re able to throw it together in the last minute as you need to. So let’s talk about Legos.
So one of the main components of backdoors and breaches and the reason why we created backdoors and breaches was to really drive home the idea of these discrete capabilities that you can have for incident response.
So we created specific IR cards for procedures for endpoint m analysis, crisis management, endpoint, security protection analysis, netflow, UEBA isolation, internal segmentation and server analysis.
We developed all of these cards, cards to really start getting people thinking about them as discrete activities that can then be leveraged against very specific threat actor techniques and tactics and tools and you build your deck based on these basic bricks of incident response functionality.
So I wanted to go in a little bit more detail on some of the things that you could and should be doing whenever you get into an eventual incident that you’re going to work with. But always keep this kind of Lego brick analogy in mind and stop thinking about flowcharts.
So I want to pause for a second. Do we have any questions so far that I need to answer?
CJ Cox
Biggest thing is that everyone says the slides look really tiny. And I don’t know if that’s a screen arrangement on your part, John, or what, but it’s apparently much smaller than it has been.
Whoa. Was that better? Everybody okay? Don’t have a lance. I think that definitely fixed it on my screen. All right, I don’t have any other questions yet, John.
John Strand
I was monologuing on mute again, so that’s great. It’s that day. It’s that day, right. So the first thing to keep in mind, anytime you’re working with incident response, and this is how I always started, sans 504, is you need to remain calm and don’t panic.
And I know that that seems very, very, very difficult, but not panicking is probably the most important thing you can do, because panicking won’t actually help you with an incident, like, at all.
So don’t freak out. I mean, serious, don’t. Don’t freak out. Now, how you get to the point where you actually don’t panic is you actually start by drilling.
And I always use the analogy of, weapons training. CJ’s son just graduated as a marine, so a round of applause.
That’s not what you say. You don’t applause. Yeah, it’s Ura and semper Phi for CJ and his son. That’s really, really cool. Now, whenever you go through weapons training, they don’t take you out on a sunny day with no wind, and the birds are chirping and they got these cute little sandwiches with the crust cut off.
They don’t take you out on that day and say, okay, well, you point this that way, and you pull this little thing here it goes pop. And that’s it. Your weapon certified. That’s not how it works. They basically have you go through and do it again and again and again and again.
I had a friend recently. not recently. This was a few years ago, but, he was a sniper. And I was like, oh, man, that sounds awesome. If you’re a sniper, that’s, like, really, really cool. You get to shoot all the time.
And he got this look on his face, which is kind of a little bit distressed. And he’s like, no, it was actually really crappy. Like, what do you mean? He goes, we had to go through and shoot again and again and again and again and every single time.
We had to document the conditions, we had to document how far we were off. We had to analyze what we could have done different. And they basically did it multiple times, over and over and over again, recording again and again what had happened, because that’s how you get better.
Further, whenever we’re talking about weapons training, they tend to make it happen under very stressful circumstances with people yelling at you, possibly gunfire going on. And still to this day, whenever I talk and I teach class, if I say, how many ways are there to clear a jammed weapon?
You can sometimes see people going through the hand motions to clear a weapon. And the reason why is they drilled it again and again and again. And that same thing needs to apply whenever you’re dealing with incident response.
That same thing needs to apply with all of the tools and the procedures that you’re using whenever you get into an incident, simply because you need to drill again and again.
So it becomes second nature. It becomes something that you do not think about. Another great technical example of this, is, Jaff.
Jaff did this webcast, an amazing webcast about python and regular expressions. And in the webcast, he was literally changing regular expressions on the fly. He was changing them up, typing them out, and for many people, it looked like black magic.
But realistically, Jaff had done this so many times, working at a university and working as a consultant, that it was second nature to him. He had been drilled in this many, many, many times.
We really need to get to the point where we’re looking at memory forensics, we’re looking at deep blue Cli, which is a tool we’re going to talk about. And then instead of response scripts and logon tracer, we really need to be at the point where we can do that activity without even really thinking about it whatsoever.
Let’s get some things out of the way. There’s a whole bunch that I’m not covering. Securing the area is one of my favorites with securing the area.
A story about this. We had an incident that we were working a long time ago, and, this one guy we used to call Indiana Hamill, he wore an Indiana Jones hat and like, these white, white tennis shoes everywhere.
And, it was literally an Indiana Jones fedora. And the guy had this, like, life size cardboard cutout of a stormtrooper in his cubicle. Just a really, really.
He was fun to talk to, but he was kind of out there a little bit. This gentleman came in in the middle of an incident when we had a server pulled out, and the top was up, and he walks up to the domain controller, and I can’t remember what we were doing.
I think we were doing hard drive duplication, or we just finished hard drive duplication, and he walked up to the server and reached in, grabbed the memory sticks, yanked them out of the server, and then put them back in while it was running.
What he told us when he did this is. Yep. Sometimes these servers get a bit flaky. You just need to unseat the memory and shove it back in. And then he turns around and walks away, and there’s, like, five of us sitting around the server that are just like.
You just see that? Did that just happen? He pulled the memory out of a running computer system, plugged it back in, and was like, see ya, folks. And, like, walked off into the sunset.
Doo doo doo doo doo doo doo. And it was amazing. As part of that incident, we should have secured that area first for that type of thing to not happen, but we maybe should have told them to stay away, which we did from that point on.
But the point is, we didn’t secure the area, notifying appropriate officials. we’re not going to cover IR coms, but there are tools that you can check out, like cyber CPR.
This, is by Steve Armstrong, a fantastic sans instructor. And he has a full step by step, cyber CPR environment that you can actually use, and you can actually download the free version of it.
So it’s a really cool product, and they have a free version that I think is great for everyone to get started with. Check that out. We’re not going to get into a lot of detail about recording everything you do in a logbook.
Whenever you’re working an incident, you should be taking detailed notes, something we cover in 504. Pull the team together. pull together the right people. Don’t pull everyone. You need to have the network administrators, systems administrators, security team.
But at what point do you need legal? At what point do you need HRDH and stuff? So all of this is more than likely for another webcast. So let’s get into the technical components.
So, starting this out, where are your logs? This is one of the biggest things, we talk about whenever I’m together with Jake Williams from rendition security is when we’re working incident response and his team is working incident response as well.
It is very, very common for organizations before an incident happens, to not have a clue where any of their important logs are as well. they don’t know where to find them on a server, they don’t know how to export them.
They don’t know how to pull logs for different services and web applications and routers and switches. They’re just completely lost. So there’s lots of logs that are out there, but can you actually get them?
So I say drill, baby, drill. Because you can practice, like get your entire team together on a Friday and say, we believe we have a workstation compromise. Let’s get the workstation logs, let’s get the active directory logs so we can do offline analysis and then see how your team actually reacts to that particular incident.
This is key, right? This type of practicing, this type of drilling is something you need to do, like tomorrow on the Friday, your standard, Friday Funday, you need to do that now.
You do not want to wait to do it whenever you get into an incident as well. So what are some of the key logs? Well, the most important logs for me is active directory logs.
even though I think logging in Microsoft Windows is absolute, complete and total garbage, which it totally is, we can argue that, but if you do, you’re wrong. They are still valuable in doing identification, or helping with identification for things like lateral movement.
If we have a user account that is then accessing multiple computer systems, well, we need to be able to pull those logs to be able to do that analysis. Now if you’re looking at this, I would recommend looking at the ATT and CK tactics six and seven webcasts that we did in the ATT and CK tactics webcast.
We talked about the type of logging that would be required to identify things like password sprays, the types of logging that would be required for you to detect somebody accessing multiple different files on a file server at a very high rate of speed.
You should go back and check out that webcast because it was all there. The other thing about active directory logs is user behavioral, user and entity behavioral analytics is your friend and I want to spend a couple of minutes and I want to talk about things that kind of suck in security that you just got to get over.
When we’re looking at security, it’s very common for administrators or cisos to say, oh, well, that’s too much time involved. there’s too many false positives.
Okay, every time I hear there’s too many false positives, the only thing I hear in my head is, they’re just being lazy. If you go back. Cause I’m old, 15 to 20 years in computer security, the only thing we had was syslog and a shoestring.
And you would have everything logged to a syslog server, and then you would create these scripts to filter through them. Now, a very, very dear friend of mine who’s actually on this webcast answering questions right now, Chris Breton, basically talked about an approach for handling logs off of a server.
What he did and what he taught us back in 2000, I think it was 2000, may have been 2001 or two. but at any rate, what he taught us was, whenever you’re looking at logs back in the day, you would go through and you would sort your logs by type and count, and you would take the most common logs, and you would say, okay, I know what this log is.
I know what this log does. This log makes sense to me. And you filter it out. And then you go to the next log and you say, okay, this one. I don’t know what this is. You’d research it, you would learn what that log is, and then you would filter it out, and you would keep doing that until you had a firm understanding of all the different types of logs that were being generated in your environment.
That sucked, folks. That was not fun. That was what we call hard. But what it did? It made us good.
We m didn’t just buy a product and then expect this product to tell us what’s going on. There was a tremendous amount that you learned in the process of doing that. You learned services that shouldn’t be running. You found out that there was user accounts that should never be logging in.
You would find out that you weren’t logging something that you thought you were logging. That would be important to working through an incident, and you would have to modify your logging policies. We had to do this back in the day before we had things like splunk and elastisearch to deal with.
This is just what we did. Now, I don’t want this to be in something that’s like, well, we walked up school both ways, and it was snowing, and you should do that, too. screw it. I’m just going to say that. Yeah, you absolutely should.
You should never be looking at your sim and your logs as just something that is, a magic black box is going to handle that. And the other thing that bothers me is UeBA, alerts.
Yes, there’s a lot of them, but there’s a lot less logs than we had to sift through back in 2001. So if you have an alert that you have a system account that’s logging into hundreds of computer systems, great, filter it.
If you have a user account like an administrator that’s accessing multiple computer systems because it’s part of their job, set a clipping level and then filter it. That’s literally what our jobs are supposed to be.
And I know there’s people like, but this is hard. I clearly don’t understand. No, I do, and I completely get it, but you need to be doing this. So user and entity behavioral analytics is without question your friend, because it’s a lot easier than doing that type of log analysis by hand.
And there’s a whole bunch of tools that are available to you. I don’t care if you’re using splunk or logarithm or logdoctor or whatever product you’re using, there is a user and entity behavioral analytics module that you can plug into it.
Yes, it’s going to generate alerts, but it will catch lateral movement. So you need to do that. But in this webcast, I really wanted to focus on free utilities, different free tools that you could use to be able to do this analysis for you.
But before I do that, I want to see if CJ has a question. I see it turned on his camera. CJ, any questions or things I should answer now he’s on mute.
See, I’m not the only one. I’m not the only one that’s making that mistake.
CJ Cox
Okay, no questions, but go to presentation mode, please.
John Strand
So if I go to presentation mode, it goes like this. So here’s what I’m going to, here’s what I’m going to do. Give me a second here real quick. Okay, give me a moment.
If we have to re, we’re going to, here we go. We’re going to completely redo everything in the hopes that it all doesn’t die, very quickly on me, so on my computer around here.
Don’t unplug, don’t unplug, don’t unplug. the cables are just long enough. There we go. Now I think I can go to presentation view without too many issues and close that and present that way.
How’s that? Does that look better?
CJ Cox
It looks fantastic.
John Strand
All, right. And the camera’s still coming through?
CJ Cox
Absolutely.
John Strand
There we go. I just had to unplug my monitor and life is good. So there you go. So there’s the logs. So I wanted to give you all some free tools and utilities that you can use to do this type of analysis for incident response.
But I also want to make it very clear that this is very hard to scale. So the first tool I want to talk about is logon Tracer, which is an amazing tool that has been created by JP cert.
Now what this tool does is it ingests your event logs and then it’ll automatically start do a correlation of what systems are these user accounts currently logged into.
So if you have a user id and that user id is now logged into 23 computer systems, it’s going to show up on login tracer. Now this opening screen from login tracer that pops up can get extremely noisy in a full domain because you’re going to see all these different connections that are going on all over the place.
You can actually put in filters for specific user ids. If you’re working incident response and you have your active directory logs, you can simply find the user id for the account that you believe to be compromised and then you can filter on that account.
You can see how many systems it’s logged into. You can also search on computer name. If you have a computer that is compromised, you can basically filter and see all the systems. That particular computer system itself is logged in as well.
Very, very cool capabilities. The other thing that I love is the ranking capability in login tracer. With a ranking capability, it’ll show you your top login users and your top logged in workstations.
Because remember, a workstation can have multiple users logged into it. You’re able to actually see that activity, do that sort, do the filtering to be able to get good telemetry about what’s going on in your environment right now.
So this is free. It does not compare to a full commercial offering that has UEBA, but free is free. The other thing that free does for you that is incredibly powerful is free can become the foundation for you to get the funding that you need for the tools that you want in your environment.
So here’s an example. If you want to get user and entity behavioral analytics in your environment, but it costs a lot of money, well, you can run this free tool. It’s going to take a long time to run.
I recommend giving it lots of cpu cores, giving it lots of memory. It’s going to take a while for it to run. Now you can sit down and you can talk to your CISO and say, look, this is what we got from a free tool.
The CISO will say, yeah, that’s really, really neat. But clearly if it’s free we can just keep using it. You can say, look, it took us an hour to process our logs to get this information.
It’d be really cool if we could do that immediately on the fly. Well, that’s a powerful narrative. This helps you with all of that. So check it out. The other thing that logon tracer does is it’ll show the user activity, the ranked user activity over a 24 hours period.
During normal business log on times, you would expect to see a larger amount of activity. But what this can do for you is it can look for accounts that are logged in at strange hours.
The other thing it can do is maybe pinpoint the exact time that an attacker got really active. So if all of a sudden you see at 10:00 in the morning, the SG or SYSG admin account all of a sudden logged in or 1600 hours, it basically logged into 36 systems.
Well, that would be a good pinpoint of when that attacker started looking and moving around the network laterally. This narrows the overall window when you’re working an incident of where you’re going to look for other logs for that activity on file servers or maybe web servers.
But instead of looking for a full 24 hours period, you can now break it down to a much smaller chunk of time for doing analysis. So I want to pause. Any more questions so far?
I think the screen thing has been fixed. All it took was me disconnecting my computer. Any questions before I move on to deep blue? Cli by Eric Conrad.
CJ Cox
Yes boss.
John Strand
Let’s go.
CJ Cox
so how does incident response in.
John Strand
Office 365 error change whenever you’re looking at office 365? It really depends on what El level you are and then also what you’re paying for.
So there’s a whole bunch of different security features that you can enable in Office 365 that are like user and entity behavioral analytics that will help you identify things like password spraying or potential attacks that are coming into your environment.
So you need to look at the security features that are there. The problem with the Office 365 suite and it is getting better is traditionally it tends to take them a lot longer to detect different types of attacks in the environment and it has to do with log correlation between their load balancer and then the Office 365 instance that exists on the backend.
But yeah, that’s probably a whole webcast series that we should do. For instance, response in the cloud. We should absolutely do that.
CJ Cox
Yeah, absolutely. Every cloud brings its own little different challenge and wrinkle. What is the best way to bring local workstation logs like Powershell logs into your sim.
John Strand
So we’re going to talk about that. But there’s things like Windows event forwarding and specifically I’m going to be talking about Sysmon. Now you can turn on the event logging on all of your workstations and have them forward their logs to your sim.
However, that can become problematic with the total volume of logs. You can also specify the specific event ids for Windows command line log auditing and Powershell logging to just send those, a little bit later though, I’m going to talk about Sysmon.
And I really do like Sysmon a lot better for most of this stuff, but still, you should be turning that on. Also, this is a great segue for deep blue CLI. You see, if you don’t forward all of your workstation logs into your sim, you can use the sim and your EDR and your security appliances and network analysis products to detect the initial attack.
But if you have the Windows logs directly on the server, let’s say that you increase the logging size to, let’s say a couple of gigs where you can do all the logging for Powershell, you can do all the logging for the command line.
They may not be in the sim, but they’re going to be on the workstation. And that’s pretty valuable because if they’re on the workstation, that now brings tools like Eric Conrad’s deep blue Cli to bear.
If you can get them in the sim and get them as part of your logging for Splunk, that’s great. Do that. If you can’t because of cost or sizing restrictions, it’s still okay, you could still keep the logs on the workstation itself.
Now there’s a couple of caveats with it. First, it is possible to pause the event logging service, delete the event logs, and then basically restart the event logging service without an event log being generated.
In fact, mimicats is a tool from Benjamin that has been doing this for quite a while. We’ve also seen a handful of tools from the shadow brokers and vault breaches that also had the capability for event log editing.
But here’s the kicker on that. I’ve never really seen them used in real incidents. That doesn’t mean it hasn’t been used, it just means it’s not all that common. The other thing to keep in mind is if we basically looked at every single security idea, whether it’s firewalls or antivirus or anything, and we basically told ourselves, well, that can be bypassed, don’t do it.
Well, then we wouldn’t have anything. So what I like to recommend is, yes, an attacker absolutely could gain access to a workstation and clear out the event logs and suspend the event log service and do all of this thing.
That is absolutely possible. But if you’re confronted with a question of, do we log this on our workstations or not log it at all, I’d recommend logging it on the workstations.
So, that’s a great question. So, any other questions before I get into deep blue CLI a little bit more? No.
CJ Cox
Takes me a minute to get off the. I turn the camera on and.
John Strand
I know, I know, I know.
CJ Cox
Does logon tracer or similar require logs to be specific? Type text, CSV, etcetera?
John Strand
Just the standard export of the logs. Logon, tracer can import those. So whenever you do export logs, you can import them directly and run them.
CJ Cox
Perfecto. Gracias.
John Strand
Very cool. All. right, so let’s get into login tracer. So, there are very few people in the world that I respect more than, like, a lot of the instructors that I taught with at the Sans Institute for years, and Eric Conrad is one of the people that just rises all the way up to the tippy top.
Eric Conrad is absolutely fantastic. And more importantly, he’s constantly giving things back to the community, and he’s someone that’s been in the trenches and actually works through incidents.
And I have a tremendous amount of respect for him on that. He wrote this tool called Deep Blue Cli. Deep blue Cli ingests those event logs. Yeah. You’re going to need to have Powershell and command line logging enabled to make it work properly to pull down all the special features that it does.
But. Oh, hold on. Okay, that’s fine. Go to school. Okay. Sorry, son, just trying to get ahold of me. So what this tool does is it’ll look through the event logs for certain activity that is indicative of an attack.
Like a user that was created on a system, certain lsas injection processes that are injection techniques that can be an indicative of a mimicat style attack and all.
That’s really, really cool. But my favorite thing that deep blue Cli does is it’s starting to touch on a concept of durable, durable threat intelligence.
And what do I mean by that? This is a whole webcast that’s going to be coming up the week after Wild west hacking faster in two weeks, where we talk about durable versus a firm threat intelligence.
And an affirmative threat intelligence would be like an ip address, it would be a host name for a domain, it would be a hash. And those things can change rapidly and the attacker can actually change those out very quickly.
So there’s value, but it’s not as much as you would think. Where we get a lot of value is durable threat intelligence. And one of the things that he does is he’ll analyze PowerShell and he’ll look as to whether or not there’s types of encoding being used.
He’ll look at the alphanumeric characters that are used and the dispersion of what characters are being used to see if there’s some type of a 64 encoding that’s being employed in the attack. And that’s really cool because now it’s starting to touch on that durable threat intelligence.
Instead of looking for a static signature, it’s looking for how a command was executed, and it’s making a determination as to whether or not it’s sketchy based on encoding and all kinds of obfuscation that can actually be utilized with it.
So very, very, very cool. So it’ll do things like the user that’s been created, password spring, the type of encoding, bloodhound usage, different suspicious services with random names being created on the system.
Just really neat. But here’s the thing. He gave this presentation on threat hunting via sysmon. So recently he started adding in sysmon log analysis onto deep blue CLI as well.
And as you all know, when we’re talking about things that we absolutely love, Sysmon is one of our all time favorites because deep blue CLI also has deep white CLI.
And deep white CLI analyzes the sysmon logs on a workstation for potential malicious activity. In addition to the Powershell, the standard event logs, and the command line auditing that you can get on a Windows computer systems.
So that’s 3d text for the price of one. This is also something you can run on a live system. So this becomes really powerful for that initial step, first steps of incident response, because now you can look at the system from a live analysis, not just running through a series of different cheat sheets that exist, but now we can run a tool that’ll do that type of analysis against the logs and then bring back if there’s any suspicious activities on that computer system.
So very cool. The other thing that he’s adding into it is with Sysmon, specifically an event id, one for process execution, you can actually generate a ShA 256 hash of the executable that is ran automatically for every single executable.
And Sysmon will log that as part of this tool. It parses those sysmon logs, pulls those hashes, and then submits those hashes up to virustotal, through the virustotal API.
And then it’ll come back and tell you if that hash has been seen in its database. I’m not a huge fan of blacklist type techniques, but incorporating this, where you can check the hashes across 47 or more antivirus engines to see if there’s any activity on that system that triggers, not on just your local antivirus.
Put on all these other antivirus engines. That’s just a feather in your cap as an incident responder. As I said, an attacker, a good attacker can bypass those. But when you’re covering due diligence as an incident responder, the ability to do that on the fly automatically with something like deep white Cli, that’s just really cool.
The other thing that it gives you is the ability of feeding it in a baseline for the whitelisting. So if you believe that system a is compromised, that one of your default builds is not compromised.
You can run this script where you do get childitem within Powershell. That’s going to pull down all of the things that are in the Windows system 32 directory, including the executables, DLL’s, Sy’s, com files recursive.
Pull down the hash. You can then feed that into deep white CLi and it’ll say automatically ignore all of these different file hashes and just focus on the potentially malicious file hashes of.
That’s cool. And this shows kind of that really creative thinking that Eric Conrad has whenever he’s working through incident response when he’s teaching his classes. So check it out.
So let’s move on to the network. The networking side is where Chris and myself and Ethan and Keith and Sam and Lisa and everybody have really been focusing on.
And of course we have Rita in here. We’ll talk about Rita some more. But securityonion is fantastic. It’ll automatically set up Sarakata, it’ll automatically set up Brozik for you.
it’ll give you all the stuff that you need to get an initial head start. I’m, moving my computer right now to get it into a better position, so that’s automatically going to be handled by security onion also, once again, Rita is huge, and we’ll have some screenshots and some stuff with Rita here in a little bit.
But Rita now runs. We can install it on prozek or it’s coming out really shortly. But it also is important to get access to the raw firewall logs so you can see failed access to temps to different network segments.
Once again, this gets into that drill bear beach drill idea. Before you get into an incident, you want to test getting access to your firewall and router logs. You don’t want to do that because you took a sans class and you’ve never done it before, and you’re in an incident, and literally the first time you’re going to do it is in the middle of the incident.
That’s just a bad idea. Don’t do that. There’s also other great tools for basically keeping a protocol breakdown, communication breakdown for system. That would be tools like end top and then also coffee.
So I hear clickety clacking. That means that CJ’s gone off mute, so now’s my time to ask him. So do we have any questions, sir?
CJ Cox
Yes, you do. Several.
John Strand
Yay.
CJ Cox
Should I be using something like Winlog, beat on workstations and servers?
John Strand
Yes, absolutely. If you’re using Sysmon, you can use Sysmon with Winlogbeat, get that data ingested into an elk stack. We did a number of webcasts about that.
If you just do search Black Hills information security, implementing Sysmon, there’s a blog post by, Derek Banks, a webcast that Derek Banks did a number of years ago, and then another webcast that I did on how to set Whenlogbeat with Sysmon to send that data into a halk instance, and we did it live on an active directory, pushing it out through group policy.
And that was late 2019, so we have a whole nother webcast on that. Fantastic. Any other questions?
CJ Cox
Does chronicle change the game here? They don’t charge the log size.
John Strand
Oh, my God. Oh, geez. That breaks my heart. Chronicle last year, and I’d like to get Chris’s take on this, was one of those technologies that we saw at RSA that absolutely knocked our socks off.
Chronicle backstory was amazing, but if you do Google searches for Chronicle backstory dead, it appears like it didn’t, that it didn’t get the love, care, and feeding that it needed as a startup, and it appears to be dying on the bind.
I don’t know if that’s changed, but it was a really, really sad article talking about how Chronicle was kind of falling off to the wayside.
CJ Cox
it sounds like they put a lot of focus on the infrastructure and the speed and the searching and all the stuff that you would expect Google to be good at.
And not a lot of focus on the security side of it. John, you may remember when you and I went by their booth, we were looking at their beacon detection, and when you and I looked at it, we said, this is going to catch everything but beacons.
John Strand
Right by the frequency analysis. Yeah, that was bad.
CJ Cox
Yeah. So it just kind of seems like they spent a lot on the back end, not enough on the front end, and didnt really have something that could produce value.
John Strand
Preston, the other thing was, the people that Chris and I knew, who were former students of ours, were awesome to talk to. But I also believed that there was this too cool for school attitude in the company where there was a number of people that were very large customers, that were very interested, and they put in a request, they put in a call, and their sales team would never get back to them at all.
And that’s not a good way to run a company.
CJ Cox
All right, would you recommend piping sysmon into existing security onion server? I think that was covered in that same webcast.
John Strand
Yep. Yep, you can. We actually talked about doing that. there’s other distributions like helc, there’s security onion. We have a beaker that’s coming out. So I might hand it over to Chris to do a demo.
And, Chris, you can get it ready to go, but that integration between the sysmon events and your other security tools is going to become more important moving forward. But, yes, you can absolutely do that in a number of these different elk distributions.
CJ Cox
I’m not sure I understand this question. Would running a hash through VT essentially notify the hacker that you’re onto them? And should that matter?
John Strand
I don’t think so. the reason? Well, it can. So, within virustotal, if a hash hits it, and it doesn’t hit virustotal, then shares that with all the vendors, and they could see if they have any hits, and then they can write signatures for it.
That actually takes a little bit longer than many people would think. So, yes, there is a possibility of that being submitted to virustotal, tripping the attacker, but it’s very unlikely that it actually would.
But really. Good question.
CJ Cox
Nice. Yeah, I’ll let you get on with it a bit.
John Strand
All right, sounds good. So let’s move on. Here we go. Somebody was just mentioning security onion. Securityonion is free, and it kicks most commercial tools to the curb.
I spent a lot of time walking around and looking at network analytics tools, and they’d go through. It’s like, wow. We take, brozeq data, we can show you all the protocol breakdowns.
We can show you the workstations and what’s communicating with what security does that for free. And the dashboards that Doug Birx has created with his team are m pretty solid. I mean, it’s an impressive product.
Also, they offer training, they offer support. So you can use it for free if you want, but if you want more support, you can actually work with them. So it has that zeek, it has Sarakata so much more.
And I don’t know, Chris, I know it’s in our dev branch, but we now have it. So it works with Rita. It did work with Rita before, but Rita had to be changed to, support JSON for Brozik export.
And Reto is either a supporting Brozik now or like, really, really soon the main branch of Rita is going to support.
CJ Cox
so it was actually the other way by default. Security onions pushing, JSON and Rita would not, excuse me. It was having Zeek generate JSON files.
Mhm. Rita would not ingest those. So you had to change it to tab comma, tad separated. Yeah, you still need to do that. You won’t after the release.
That comes out in about two weeks.
John Strand
Check it out. Now. You’ve got beaconing and you’ve got security onion, and there’s other tools like endpoint security analysis tools that feed into it as well. It’s amazing, folks.
So check it out. Now, let’s talk a little bit about Reta. You knew I was going to go here, but Reta’s specific goal in life is frequency analysis and beaconing detection. It also does blacklist analysis, it does long connection analysis, and it’s all using median average distribution of the mean to look for these cluster patterns.
If you look at this slide, the closer that you get to the center, the closer it is to a perfect beacon or a perfect attribute of data size or connection time. And that’s what Rita does.
Rita’s looking for those clustered consistencies in your connection patterns, and it will pull out things like long connections. See all the different connections that are being made of, and it’ll tell you who are the longest connections that are out there.
one of the changes for long connections that’s coming out is instead of looking at just long established connections, if we have a system that’s communicating very rapidly with lots and lots of short or random connections, if you add all those connection times together, it’s still going to be a long connection.
So that’s very cool. So check that out as well.
CJ Cox
And then also that being built into it. You can do that yourself with data mesh. actually the webcast where that slide came from.
John Strand
I stole a whole bunch of slides from you, Chris.
CJ Cox
Awesome. but I talked through, how do you calculate this stuff? Because that’s one of the things with security onion is you can see longest connect time, but not if it’s a kinect going off every 30 minutes.
For 30 minutes, that’s still going to fall to the bottom. This allows you to go through, just pump it through data mesh. That’ll add it up. You get to see cumulative communication time, which makes life a whole lot easier.
John Strand
So the analogy that I like to use is whenever you’re thinking about a total connection time for a system communicating to another system, think of a stick of spaghetti. Imagine that stick of spaghetti is 24 hours, and then we break it.
If I break that stick of spaghetti, it’s going to be inconsistent sizes. You’re going to have tiny pieces, you’re going to have longer pieces, medium sized pieces. But if we reassemble them, it’s still 24 hours of connections.
That’s where using data mesh can actually look for those long connections as well. All right, moving quickly on to memory forensics. this is another webcast that I’ve done in the past.
If you want to see it, I’ve got the link down there. Blackholes infosec webcast, Windows memory forensics. The two main tools that you would use for memory forensics are volatility and recall you absolutely.
As part of one of your first steps for incident response. If you think you’re compromised, dump the memory with a tool like winpmem M and also, Ftk imager has the ability to dump memory.
There’s a ton of memory dumping utilities that are out there, but you want to dump that memory quickly off of that computer system and then you can do offline analysis of that memory dump.
So here’s just a dump from, sans 504. I popped it in here and you can see that we’re doing ps list and we’re pulling out all of the processes that are running on this computer system. And we can see the parentage hierarchy on those processes as well.
I didn’t want to get into too much detail on cheat sheets. I started getting into the cheat sheets and there were just so many cheat sheets that were out there. For most of these cheat sheets, you’re going to need to have command line and Powershell logging enabled.
I provided a couple of links for references for you to do this. If you decide to utilize something like deep blue CLi. The Deep Blue CLI website actually has step by step instructions for how to enable the proper logging for deep blue CLI.
A lot of these cheat sheets actually require that type of logging to be enabled. Lenny Zelster has tons of cheat sheets. The digital forensic sans cheat sheets for windows, for Linux, for splunk are all available cheat sheets for malware analysis.
There’s tons of them out there, have them available. And this also ties a little bit to something I talked about at the top of this webcast where I was talking about record your actions.
Whenever you’re creating a logbook and you’re recording your actions, you don’t want to have to write down every single command that you ran. However, if you can write in your logbook, I ran the sans Windows cheat sheet, well, that’s a whole bunch of commands that you can now record in one shot in your logbook.
I ran Deep blue Cli. You don’t have to record everything that Deep blue CLi does, but now you’ve got a recording that you at least ran that toolset. So using these cheat sheets kind of become shortcut for kind of become shortcuts for sort of a shorthand recording of what you did as part of that incident.
So yes, there’s plenty of cheat sheets that are available for you for looking at the command line on Windows and Linux. Now this entire webcast, just so has been very heavily focused on Windows.
Sure that a lot of people have gotten that. People will also say, well, why don’t we talk more about Linux and Unix systems? Two reasons. One, I needed to focus on the biggest bang for the buck for people on this webcast.
And two, al has kind of agreed to start doing regular webcasts with Black Hills information security. And one of the webcasts that he wants to do is a live Linux forensics webcast for detecting attacks on Linux computer systems.
So rather than me talk about it, I’m going to have Hal talk about it because he’s going to be much better about it than I am. So yes, that is in fact coming up. All of this really ties down to a concept.
When you start working with an initial incident comes down to a concept of overlap. I work a lot with ions and there’s a ton of questions about we have too many security tools.
We want to get rid of the security tools that are doing the same thing. I don’t want to do network forensics or network analysis on the inside of the network because I have EDR products and they tell me everything I need to know about my endpoints.
Do they? I mean, probably not. So when you’re looking at that endpoint, it’s about bringing the focus of at least four things, all to the endpoint that is in question, and kind of marshaling all of those logs in the upper right hand corner quadrant, the live forensics that you can do with things like deep blue CLI and the cheat sheets, the memory analysis, and then also that network analysis and aligning them all on that workstation and pulling out all the logs that are applicable to that workstation or user id, pulling out all of the different data that’s relevant for the memory on that workstation and all the network traffic that’s relevant to that specific workstation as well.
This allows you to focus. I really like to think in this way, not necessarily always in Venn diagrams, because that’s just annoying. But I like to think in this way of whenever I work an incident or I do something, what are the things that I need to align to be able to make me effective at what I am doing and then getting those pieces in place.
But this ultimately requires you to get all of those pieces in place before the incident occurs. Can you gain access to your active directory logs? Can you run through live forensics?
Do you even know how to do live forensics through the cheat sheets? Can you dump memory? And do how to analyze memory? And then can you also do analysis on the network?
Further, all of these are existing webcasts. If you look at logs, we did a webcast on ATT and CK tactics. I think it was like seven or eight that was talking about the logs that you need to be able to detect things like password spray, lateral movement, and SMB relay style attacks.
We did a live Windows forensics webcast. We did a memory forensics webcast. And yes, we’ve done a lot of stuff. Whenever it comes to the network, there’s a lot of resources that are available to you that you can now bring to bear whenever you’re doing this type of initial forensics analysis.
But it does require a little bit of legwork to get there in the first place. As I mentioned, this is brought to you by Wild west hacking Fest. We’re not doing the live event.
I’ll talk more about that here in a little bit. But before I talk about it, I would very much like to handle some questions, from the audience. So, CJ, what do we have?
CJ Cox
Oh, my gosh, so many good questions. So someone’s talking about, would you mention if some of these tools are windows agnostic. You addressed that opinions on Logarithm, Netmon.
John Strand
So Logarithm and Netmon. logarithm is a very, very, very solid log analysis tool. And the Netmon product is very similar to like NTop.
So if you’re looking at what’s going on on a system, you can then look at the network traffic associated with that system as well. So that’s really powerful. And yes, you can do that for free with things like security onion.
You can absolutely do that for free and kind of correlating that. But you do get some features whenever you actually get a commercial product that actually helps you with it.
CJ Cox
Yeah, somebody was pointing out that they were doing security onion spent two weeks playing with it. And those things are labor intensive. That’s what you get with the commercial tools.
John Strand
Yeah. But on the flip side, you sure learn a lot about the security in your organization. It’s a great point.
CJ Cox
Yep, very good stuff. Distinct advantage of Suracata over snort.
John Strand
So this is weird. It seems like a lot of love for snort went away when they were bought by Cisco. They weren’t as updated as much as I think the community would have liked them to have been updated.
And it really just seems like the community has kind of started moving on. I wouldn’t say, hey, completely ignore snort and you can run snort, you can install it with security onion. So you can get that second opinion of Sarakata ids and Snort ids.
But it really does seem like the open source community didn’t feel a lot of love from the snort community, when they were bought. And it seems like people just moved on.
Yeah.
CJ Cox
there’s a way to filter out logs on the front end so you can manage resources in an elk stack.
John Strand
Yes. So whenever you’re dealing with the sysmon config, in particular, swift on security’s configuration is very much geared toward doing that type of filtering.
We could argue that it’s too much filtering, too little filtering. I think we can go back and forth and there’s tons of arguments both ways, but going through and filtering out standard windows processes, that may be valuable.
But then again, if I make a share mount from one system to another, that’s done by the system process. In event id four. If we ignore that, we’re going to be missing a lot of what’s actually going on.
And I think the community really hasn’t coalesced, really as strongly around a standardization as we need to. There are some projects out there like sysmon modular.
This is covered in the purple teaming class that, Kent, and Jordan are working on. That gives you the capability to do some more fine grain analysis and log filtering from sysmon before it’s ever sent.
So be on the lookout for those projects. But yes, you can, but be careful, you might actually filter too much in.
CJ Cox
Order to use all these tools. How many copies of logs do you need? It seems like you need a different copy for each of the tools.
John Strand
That’s possible. If we go back to the slide and we look at it, the live forensics, that’s totally going to be something you’re going to have to do on your own live. The memory forensics, that’s totally something you’re going to have to do on your own live.
The log analysis will exist on its own and the network analysis in almost every situation is something that’s separate and distinct. Even if somebody is doing things like Netflow and they’re, doing Netflow analysis, many, many, many organizations will still end up doing a full pcap or getting a full brozik pull on that system.
So yeah, your whole point as an incident response or is to correlate and fuse all this information together.
CJ Cox
Recommendation for network tools is to not be in line, is that correct?
John Strand
you can, I would not recommend it. And that’s basically like an old school kind of way of looking at the universe. If your tools are running in line, that creates that single point of failure for the network going down.
So the best thing that you can do is sidechain it off with a span or a tap port. Awesome. favorite tools for memory forensics, volatility and recall. They’re literally the only two tools and I also recommend running both of them.
And the reason why is different memory dumps and the way the memory mapping is done. Certain tools like volatility work better than recall for some tools, and recall will work better than some tools for some other things.
And that really is dependent on the operating system version and build. So it’s really a good idea to get two dumps. I recommend one from something like FTK imager, the other one from when PMem goes to recall.
The FTK imager can get sent to volatility.
CJ Cox
CIS benchmarks suggest not having command line process auditing. Would you advise going against that?
John Strand
Yeah, I disagree. And the reason why I disagree is right now, the amount that we’re logging for event logs is garbage. And people, it bothers me because we’ve been logging trash for 20 years.
I don’t care about c four mores and spin locks and, all this weird stuff that windows normally logs. Now, all of a sudden we get command line logging and Powershell logging and sysmon logging, which is exactly the information that we want.
But because we’ve been monitoring this huge mountain of garbage, now all of a sudden everyone’s like, eh, there’s no room for the stuff that actually matters. So, yes, I would recommend reevaluating your logging structure.
And usually the question I ask people is one, how much are you logging? Okay, people can answer that question fairly quickly. Two of everything that you’re logging, what percentage of your logs actually have alerts or signatures or reports written for them?
The answer to that question is usually less than 10%. 10% is high. That means 90% of the logs that you’re logging are trash. But the vendors will tell you, well, you need to log all of that because you don’t know what you’re going to need until you get there.
Well, that’s helpful for them because their entire business model is the amount of logs that you throw at them. So it’s convenient. So I think we need to step back and start asking questions of what are we logging?
Does it actually provide value to detect attacks? And really starting to work from there, really. The, fine folks at JP Cert, they did a, tools analysis, cheat sheet, and they went through this whole entire wonderful cheat sheet of a lot of different ATT and CK tools that are used in the actual techniques that can be used under the hood.
And for each one of these, they actually broke down. What are the event logs that are generated? What is the assistmon event that’s generated? So now we can be a little bit more intelligent on what we’re logging. That was a long answer, but I hope that answered the question that’s pretty good.
CJ Cox
Concerning the Venn diagram, what’s the difference between live forensics analysis and memory analysis?
John Strand
Excellent question. So, memory analysis is where you’re actually dumping the memory. And a lot of the tools that you can run like ps list and DlL list, and Netstat and, like con, those different commands that you can run in the various tools very cleanly mapped to what you can run locally with, like Netstat, but they’re different.
So when you’re talking about live analysis, you’re talking about querying that system directly. You’re talking about using things like analyzing the, the event logs with something like deep blue CLI.
You’re talking about running Netstat NAoB and looking at the process ids and the network connections that are being made right now. Talking about taskless space, forward slash SVC to dump all the services.
So you’re looking at the same thing, but you’re really looking at it from two different perspectives. One perspective is just the memory dump and the other one is the live analysis. And what you’ll find is sometimes with memory analysis, you may not get all of the data that you would get with a live analysis like net use.
And looking at the shares that were mounted at the time that the attack came, you may not get the level of fidelity that you’re looking for with just looking at the memory on that computer system.
Great question.
CJ Cox
We’re almost up. here someone asked about opinions on using the NIST framework for IR. The analyze, contain, eradicate, and recover. I’ll talk to that just a little so you can rest your voice.
John Strand
Go for it. Go for it.
CJ Cox
look, that’s a great mental model. It’s a very, very broad overall sort of philosophical thing. Those steps certainly make sense. Right? But the devil is in the details.
We had a lot of questions about flow charting, your playbooks for incident response. I would say that the idea that no von molque, no plan, survives contact with the enemy.
however, you have, dwight Eisenhower saying plans are useless, but planning is essential. The more you do things like just think through these things logically, make some plans, have some ideas.
Whatever actually occurs will be radically different, but your planning will mean you’ve already gone through some of the thought process. You have already laid the groundwork and done the thinking that’s going to help you react to whatever chaotic, situation you step in.
John Strand
Exactly.
CJ Cox
All right, we’ve got 1 minute left. John, people are one. They love us for sucking at capitalism, and we say, thank you for that. Another one is based on the WHF has sold out.
John Strand
we’re going to be opening that up. Hey, it, looks like Marita shut off the ticket sales. I don’t think Marita knows that we canceled. We were going to shut down ticket sales today and then allow people to do walk ups.
Can you tell her to turn it back on real quick? Because there’s people that are virtual that want to come to the virtual conference now. So we’ll get that turned back on, folks. the tickets will be back here, in just a little bit, and.
CJ Cox
We are at the zero hour. So take us out, Johnny.
John Strand
You bet. Thank you so much for coming to these, y’all. one of the things, with this conference and everything that’s going on. I would like to say the last 15 years of doing webcast, if there’s one thing it’s prepared us to do, it’s virtual events.
So we’re going to do this. If you want to attend Wild west hacking fest, give us about 15 minutes and we’ll turn on the portal so people can buy tickets if you want to come. If you do buy a ticket for wild west hacking fest for this one, it gets you access to the virtual event, the Metacf cyber range, and it will automatically get you a ticket for next year.
I know that’s two conferences for the price of one, but I know people had to change their flights and it’s impacted their lives and their schedules and their travel. And the best thing we can do is give them a free conference as a way of saying sorry.
So also, this is brought to you by backdoors and breaches. There was about 150 decks left before out on Amazon. I’m willing to bet that that’s down significantly, since we started this webcast.
And it’s also brought to you by, AI hunter and Black Hills information security, bhis. we do pen tests. We love this stuff. we love sharing.
If you need a pen test, a threat hunt, if you need any of that, don’t hesitate to look us up. That’s what we’re here for. And also by AI hunter. before I talk too much about AI hunter, I want to give everyone kind of the opportunity.
I should have done it before I started talking about the advertising at the end. I should say, folks, we are now entering the advertising section of this webcast.
You can leave right now. if you want to get out of here, I like to give everyone the opportunity, to bail, as quickly as they can because I hate it whenever you’re in the middle of webcast and it turns into an advertisement.
So I apologize, but please bear with me. It’s been a hell of a day, folks.
Chris
did I hear free CTF access for folks that sign up for the conference next week?
John Strand
Yup.
Chris
You’re kidding me.
John Strand
Wow. No. Yeah. Meta CTF has a capitalist.
Chris
You do suck at capitalism.
CJ Cox
He is so bad. You really.
Chris
It’s beyond suck because I have seen what’s been set up for that CTF environment. You’re going to give it away to free to folks. Oh, my God, that is awesome.
John Strand
Yep, yep.
CJ Cox
No, it’s not. Shut up, Chris. He doesn’t need any encouragement.
John Strand
Yeah, this is, Yeah, like normally I just blow off what you guys say. But, boy, I gotta tell you today, it stings.