This webcast was originally published February 13, 2017.
In this video, John and Alyssa discuss advanced memory analysis techniques. They delve into the use of tools like Volatility and Rekall for dissecting memory dumps, and demonstrate how to extract critical data from a Windows operating system. The session also covers practical tips for identifying and analyzing malicious activities and artifacts within memory, enhancing the viewer’s skills in forensic investigations.
- The webinar provides detailed insights into memory analysis, focusing on the use of multiple tools for effective forensic investigations.
- The session emphasizes the importance of understanding and using various memory dumping tools and frameworks, such as Winpmem and Volatility, to handle different system scenarios.
- The demonstration highlights the critical role of image information in identifying the correct memory structure for analysis and the potential issues with tools not being updated for newer operating systems like Windows 10.
Highlights
Full Video
Transcript
John Strand
All right, all right, all right. So let’s get this started. Okay. So let me do this quickly. Quickly. Let’s change the subject. All right. so let’s go ahead and get started.
We have the recording that’s been kicked off. Yes. This is, in fact, being recording. And, in all seriousness, the conference is Wild west hack invest. If you do wild westhackinfest.com, it’ll take you right to the website.
It’s a conference that we’re doing here in South Dakota. I’m in, like, my small closet of an office right now because we had people talking. So you got, like, wires up there, networking equipment over here.
Because this is my life, really. I did play with, like, nice little screens and raspberry PI Dropbox hotspot.
Moderator
Hey, Alyssa.
John Strand
So, oh, Alyssa made it. So this is where I spend most of my time, kind of screwing around, working with things. so if it looks a little bit sparse, it’s because I’m hiding.
And we’ll wait for Alyssa’s audio to come through. Alyssa’s just going to interject as well. This is great. We got Alyssa and Joff on, so we’ll wait for Alyssa’s audio to come through.
Alyssa
Testing.
John Strand
There she is. I am so happy that we have Alyssa here, because we actually have an expert in memory forensics. And that makes me feel so much better. about what’s going on.
she’s actually teaching a class. Where are you teaching? It looks like it’s an onsite.
Alyssa
no one’s in the. In the room. No one’s in the picture. So don’t worry, everyone. I can’t even tell you where I’m teaching this week. There is a secret of a location. I can’t even tell you.
John Strand
There’s a creepy dude that every once in a while peers around the corner of the door. Looks like it’s stocking. Every once in a while.
Alyssa
I’ve been doing that all weekend. Doing that all week. Out of here?
John Strand
Yeah, no, that’s fine.
Alyssa
I thought you were fine. We teach all kinds. All kinds.
John Strand
It’s just the way it works. so. All right, here we go. Let’s go ahead and let’s get started. So these slides are predominantly from sans 500 four’s memory section. we actually have a full class dedicated to memory analysis.
And Alyssa, what is the number for that class?
Alyssa
that is forensics 526.
John Strand
So check it out. All right, so we’re going to talk about memory analysis. And we’ll talk first about memory dumps. And I’m going to go through and do a memory dump in two different ways. And we’ll talk about two different frameworks, talk about volatility volatility framework and Google recall.
And we’re going to talk about the big thing about today’s webcast is the importance of having multiple different tools. And that applies for taking the memory dump and also doing the analysis as well.
So the two main tools that I like to use are winpmem and I also really like to use the ftk imager, which I’ll show you guys here in a little bit. But I also have a bunch of old tools up here as well.
And every once in a while I have students that are like well that tool is old, you shouldn’t use it. And I disagree with that. And the reason why I disagree with that is you never know whenever you’re doing memory analysis or you’re doing incident response, what type of system you’re going to have to pull the memory off of if it’s a Windows 2000 box or a Windows Server 2003.
So it’s good just to keep them all in your toolbox. I mean you can just have a directory on a USB stick. That’s just memory dumping tools. And in some of my assessments, on assessments I’ve gone through and I’ve dumped the memory with three different tools at the same time and opened them up in different places.
Now Alyssa, you talked a little bit about different formats, and some emails that I was shooting back and forth with you, when you’re talking about win PMEM and compression and things like that, do you want to share a little bit about how the formats can in fact change between these different tools?
Alyssa
Sure. we know the volatility offers an incredible amount of support for different formats. I mean we’re looking at support for the e zero one, so expert witness format and definitely support for the compressed hibernation file.
So fantastic. But largely people are approaching the analysis of volatility with the raw memory image. So when you’re using Winpmem, which is recall’s acquisition tool, I think that’s the one you’ve chosen to use.
Well dawn, I like your choice there, John. it by default is going to output to an aff four, which is compressed, right? I mean that’s a, compressed. We were talking about it in class.
AFf four has a lot of data streams. It’s fantastic. I don’t mean to steal the show with AFF four conversation. But it’s fantastic because you can throw in the page file into the AFF four, you can throw in all the drivers, which is the version we’re using in class, is like capturing the drivers on the system that is dumping memory for it.
So everything is an information turtle, everything is indexed and you have these data runs. So there is some compatibility that exists. I mean you have Af four export that you can use to pull out the raw memory data runs and throw it into say an IMG or zero, zero, one file.
And then volatility is like right on, we got this thing and volatility can parse it then. That’s what we were doing yesterday in fact.
John Strand
Yep, very, very cool. All right, so let’s jump in. I’m going to be showing both of these tools and I’ll talk mainly about recall because that’s what we use in sans 504. and really I don’t know, we’re probably going to end up changing it up by using multiple different tools.
But the big modules, I would say if we’re going to talk for an hour of what you absolutely need to know, if you’re going to sit down and use these different tools, the big modules that you absolutely need to know and be able to use are here on the slide and I’m going to focus in on quite a few of them, as I actually go through and do some demonstrations.
image info is incredibly, incredibly important. It may not seem like it’s that important, but it actually is. And the reason why is whenever you’re taking a memory dump, when you’re dealing with Windows versions like Windows seven, Windows eight, Windows 8.1, Windows ten, and even between 32 bit, 64 bit, or you’re looking at specific releases, service packs, where the memory structures reside, changes.
And sometimes, especially when you’re looking at volatility, you have to actually feed that image information in so it knows where to look for information such as network connections or processes, dynamic link libraries, some tools like recall does a relatively good job of automatically identifying what that memory image is.
Now you get into some problems whenever you try to do analysis offline, because recall likes to reach back and actually pull from the database and then pull information direct.
And that keeps it up to date. you can create an offline database just by doing a git pull and basically changing a variable within recall to point locally. And that will work just fine. And we use that in 504 because it’s kind of a contrived exercise of a memory dump, and I’m going to go through that exercise and kind of walk through what we actually capture.
So, image info, very, very, very key. when doing analysis, and I’m going to talk more about some problems with memory dumps and some of the utilities and what they pull and what they don’t pull here a little bit later.
And these are some of the new problems that we experience on windows ten. And Alyssa knows a lot more of the inside politics and baseball on that, and we’ll deal with that here in a bit. Netstat is pretty cool.
Netstat lists all the open network connections where you have your process id port and protocol. Very, very simple to use very, very fast. Pstree is what you would expect. It’s going to list out all of the processes in the hierarchy.
what is the process that begat which other process? You can see the child processes, and you see a lot of this information in modern tools like, carbon black, for example.
they will graph things like this differently. They’ll actually say this particular process, like Internet Explorer, was invoked by outlook, and then Internet Explorer invoked another executable through Explorer Exe, and you can see how all of these different processes tie together.
And being able to tie that process all the way back can help you better understand exactly how things were executed on the system and why they were executed in a certain way.
dllist lists all the dynamic link libraries loaded by a process, and one of my favorite features is the command line invocation of how that process was actually kicked out.
How did it actually start? And that’s really cool, because if you see something like svchost with weird parameters, it’ll allow you to say, that’s not normally the way that Svchost actually runs, under any normal circumstances.
now I got a couple of questions that just popped in. Somebody said, is the slides already been posted? They have been posted. The slides have been posted here, and I’ll send it to everybody and you guys can pull them down.
They’re in a PDF format, so you guys can see that. People can see me. That’s great. All right, so getting those dynamic link libraries is important because you can actually look at the command line invocation and see how things were run.
Once again, it can differentiate what a real program is and what something might be. Malware. Now, Netscan is different. whenever it runs, it tends to take a little bit longer.
It tends to be a little bit more exhaustive. So, Alyssa, when we’re looking at Netstat, which seems to be very very quick. Netscan tends to take a little bit more time. In what situations would you use Netscan over Netstat with something like recall?
I think she’s on mute.
Alyssa
Perhaps you can hear me now.
John Strand
Yes, we can. Absolutely.
Alyssa
Normally I just default to Netscan. I guess we all have our preferences, but, I always kick back to Netscan because it does offer that comprehensive, sometimes I’ll get historical view of network connections or in the past maybe, they’re in a closed wait state.
So this, that’s, that’s normally what I would run. Rarely do I use net stat, but I can see the use case. It’s just a real fast enumeration of current connections and currently bound sockets.
John Strand
M those previously open connections are also really important, especially whenever you’re dealing with malware that’s beaconing, where it may make a connection and then it drops it. And then it makes a connection 10 seconds later and then it drops it.
So I like that too, quite a bit because we do a lot with beaconing. I like Netstat, and this is kind of a goofy thing. I like Netstat because it nicely fits on a slide for, sans 504 for looking at network connections, which I think is pretty cool.
file scan lists the files that each process is open, which is helpful for obvious reasons. PE dump dumps code associated with running processes and modules. lists the different modules associated. One of the cool things that I really didn’t use very much, Alyssa, was live forensics capability, in recall, and that was something I learned from you last week.
It’s like recall can do analysis on dead images, but you can also run it in live as well. Am I getting that right, Elissa?
Alyssa
Absolutely. we got, we actually will run it right from our USB for live analysis. And I think that’s more what you’re referring to as the live analysis capability of recall.
Live analysis, like Michael Cohen says, it’s always in beta. Regardless of what tool you’re going to throw at your running machine, you’re going to get varying degrees of success. And we go forth knowing that.
John Strand
Yeah, don’t we know that very well what we’re talking about, these tools? All right, so these are the instructions, some kind of partially cut down instructions of what we do in sans 504.
And we use recall in a memory image, and then we work through that memory image. And I’m going to walk through some of this. Here’s the netstat once again. The reason why I like Netstat is because it’s very nice and tight. It kind of shows me exactly what’s going on, and it fits on a slide.
Netscan also takes a little bit longer, and I’ll show you what that looks like a little bit later too. now, one of the things I like to do is tie together what we talked about in the last webcast a couple of weeks ago. With today, we’re talking about, live analysis.
If you have a system that you think is compromised, it can be very, very, very helpful to be able to run these tools, of course, with the memory image, but also running it live. So if we do Netstat, it’s very similar to Netstat.
Nao, not quite the b, possibly showing all the dynamic link libraries, but piping it through find and then looking for established connections. Now, as Alyssa mentioned, that’s great for established connections, but what about beaconing malware?
What about malware that’s trying to use UDP and making outbound connections that isn’t part of an established state? So that’s where Netscan would have a lot more value to show you some of the command or, some of the connections that are actually dead.
So this gets to kind of the heart of computer security, especially forensics. I learn more and more and more whenever I work with people like Rob Lee and Phil. And, Alyssa is, it’s not just about having one way to do things.
It’s about having multiple different ways of achieving the exact same goal, because inevitably you’re going to run into roadblocks, and you’re going to run into a situation where thing one works but thing two does not.
And there are some points about this. Let me just show you guys real quick. so whenever I was setting up this webcast, I thought I was an idiot. and Alyssa confirmed that I was a moron on some things, but on other things, not so much.
Whenever you’re looking at some of the modules, I really like to work with network connections and, working with network connections and then kind of working my way back whenever I’m looking for malware on a computer system.
So I can start with Netscan or netstat with a memory image and then see what executables are associated with that. Then I can look at the parentage associated with those executables and see how those executables were invoked.
Were they invoked from explorer Exe, iexplorer Exe? Were they invoked from Flash? How exactly did they kick off? And I can start to pull that chain together.
I started gearing up for this, and all of my stuff for lab and memory analysis was Windows seven, windows eight, and I have brand new windows ten systems. And the network connections weren’t working very, very well at all, with volatility or recall.
And I actually opened up some issue tickets, with these memory images. Now why do we talk about this? I think that this is important because whenever you’re dealing with these tools, and this also goes for penetration testing tools in any tool that you use in it, you need to understand that there’s going to be situations where these tools don’t work properly.
And when you’re dealing with 64 bit windows ten profiles, they may not have updated the code to adequately represent or actually identify where in memory network connections are, or maybe processes for specific versions or dynamic link libraries.
And when you’re doing that, it’s not necessarily a horrible bad thing, as long as it doesn’t surprise you. And what I mean by that is if you get in front of a judge and a jury, let’s say you’re working through a forensics investigation and your tool isn’t working properly, it’s incredibly important for you to document that it’s not working properly and then also replicate that situation on multiple other systems.
So if you’re doing a memory analysis, let’s say for Windows 8.1 image and something isn’t working in volatility and isn’t working on recall, document it, take pictures, screenshot it, and then try to recreate it in a lab environment that’s isolated from the investigation that you’re working.
And then say this is a known issue. Issue tickets have been created or even go to like volatility’s git website or go through and go to recall’s website and see the different issues associated with it.
So like my problem is we have Windows ten with network connections and volatility is showing the network connections, with the established. That means outbound connections have a process id of negative one.
And there’s a bunch of people that are kind of working through this together, which I think is great, right? So with this there’s people that are working through the issue and you’d want to document this in your forensics investigation as well.
Basically letting people know, because if something doesn’t work that’s not the end of the world. But if you’re caught off guard, like you get in front of a judge and jury and then across examining attorney says what are those negative ones?
And you have no idea, you look like a complete idiot. And it really helps to have really smart people that you can talk to like Alyssa, help me out now. Alyssa, what did change? Like, I know that whenever they were talking about the process id, not process ids, but network connections with Netstat and Netscan and recall.
And in volatility it was working. But then in windows ten it kind of stopped working, as well. What kind of happened? Kind of behind the scenes. I know that we had some emails going back and forth with the people of recall and there was some, there was a little bit more information about what’s going on here.
Alyssa
Well, a little bit more information, John. largely there are unpublished things about the windows operating system, undocumented, not publicly available.
And tcpip sys is one of these things. so a lot of reverse engineering has to come into play when you’re dealing with understanding the pool allocations, that are allocated by that driver, TCP IP sys, not as easy as other things that we do know or have documented about the windows operating system.
So in talking with one of the core developers, recall, he did state there were changes as of windows eight and beyond to these structures, making it harder to parse or even as you see in recall we’re not able to run a net scan and properly enumerate those structures that are created by TCP IP.
So I guess you could say that the underpinnings of the very driver changed and are not publicly available. So they really require a whole lot of reverse engineering research and analysis.
So constant changes in.
John Strand
And I would say that that’s one of those things where if you want to contribute to the community, this would be huge, right? to be able to contribute. Like I know that he was talking about, he had tried to have an intern that would reverse engineer this and it was kind of a daunting task and they couldn’t quite do that.
But if you love memory analysis, if you love memory images and you want to dig in, this is a great place for people to contribute and actually have a good impact on the community as a whole. So. All right, so let’s move on to some other ones.
ps list giving us the processes. this is nice because it gives us the process id. It also gives us the parent process id as well. And that lineage becomes really important. I’ll show you what that looks like here in a little bit as we kind of walk through a sample investigation that we do in sans 504 to try to tie through what’s going on, on the system.
Because a lot of people, unfortunately, whenever they’re looking at, memory images, they’re looking at it specifically from the perspective of where’s the malware and where’s the malware talking to. But there’s more to that story.
You could have malware that’s communicating out to Estonia or Tennessee or someplace weird, or South Dakota, and you could just focus on that malware.
But if you’re not looking at what that computer system is doing, insofar as making connections to other computer systems, you’re only really getting a partial partner of the overall image of what that computer is doing.
So being able to see the lineage of how programs executed is very important. And PS list can give that. And it’s also very similar from an output perspective to running WMIC process. Get name, parent, process id, and process id to dump that information out to you.
Now, the DLL list is, like I said, it’s one of my favorites, right? So DlL list, you give it a process id and it will dump all the dynamic link libraries associated with an executable.
And it will also give you the command line and vocation of that process as well. And that’s great, because now we can see if it’s something like sVchost or SCCM, or some type of weirdly named process like SCSI host, you can actually look into it and say, well, what does the normal process with that name look like?
And how is it invoked? And how is my malware or my possible malware invoked? Because there will be a difference in those command line invocations in some situations as well.
so very, very, very cool stuff when you’re dealing with memory analysis and pulling down the command line. And don’t worry, I’m going to show you guys what this stuff looks like more or less live.
Now I’m going to go through and kind of show you guys a couple of vms. and we’ll do a memory dump as well. So I’m using standalone versions of these tools. so we’re using recall and we’re also using volatility.
so this is volatility and volatility. You can give it a file where there’s a memory dump. And I created this memory dump with FTK imager and I can run the module Netscan and it’ll go through and pull down all of the different network connections associated with the system and the process ids as well.
So I wish I could make this a little bit bigger. Maybe we can. Let’s go through and make our font bigger. There we go. That was absolutely one of the dumbest things I think I could have possibly done.
Let’s, see how big I can make it where it’s still useful. There we go. There we go. All right, so with this kind of pre canned some of the commands that we have here. So basically using volatility 2.6, Windows 64 standalone.
And I do my, and I give it a memory dump file at the end. I’ll talk about creating a memory dump file here in just a little bit. And then we ran Netscan and we give it profile and we said Windows 1064.
Then we have a build number here, ten, five, eight, six. And it shows us the local connections and the ports that are listening. Some of the stuff you would absolutely expect to be running on a Windows computer system.
But then we also have some remote connections. So if we look at these foreign connections, I really like to focus in on foreign connections. We can see that we have, this particular process is starting on a source port 20 317.
And it’s making a remote connection to 1312, 533-4233 on port 443. Now the big problem with this is we’re missing some very, very critical information.
And this is part of that issue ticket that I opened with volatility and some other people that opened with volatility. And recall also has a very similar issue as well. Recalls is a little bit more acute.
We don’t have the process id, we don’t have the process name, we don’t have the date associated with, with it. Now that doesn’t mean that that data is not there. It just means it’s not where it’s specifically looking for, or it’s looking for that data.
But we at least have those connections in some fashion and then we can do some research kind of tying with what we’re going to do a little bit later. What is this IP address? Look up who is information, IP info on those particular IP addresses and see what it is.
It could be something as simple as like enterprise security weekly website or Black Hills information security website, or it could be some really horrible malware site that has bad, bad, bad things going on.
So it’s kind of a hole, right? That’s kind of a hole insofar as, what we have, with these memory dumps that we can do now, if we look over here, I can kind of show you recall as well.
Here we go. So this is recall, Oh yeah, go ahead.
Alyssa
1 second. I know. we need to get back to analyzing our page files now, John. So we want to thank you for inviting us on.
John Strand
Thank you so much for coming.
Alyssa
now it looks like you got everything well in hand, man. Enjoy that recall analysis. So, see you guys.
John Strand
Sounds good. And, maybe we can talk about it a little bit later. But page file analysis is really important. and the reason why it’s important is a lot of times you can look at processes that have been paged out of memory.
So if you have malware that is quote unquote dormant, like it’s only running and only doing things at a certain amount of times, it may get paged out of memory. And there’s also some research that was done by, Jamie, Butler back in 2008.
it was called shadow walking and it was malware that would wait for sequential memory reads on a computer system and then it would page itself out. So the memory read would go past and that would page itself in.
It’s kind of like rope a dope. so the memory analysis would always be a little bit off. So that’s why the page files are analysis, which is very cool, but I was happy Elisa could make it. All right, so we’ve got network connections.
If we look at P’s list, we have all the processes that are currently running and we can see those. And recall works just fine. With the current version of dumping out all the processes, we can do Pstree, and Pstree is going to list which processes begat other processes.
We can look at that overall hierarchy. So if we do have a process that is interesting to us, we can actually go through and pull down that specific process.
Let’s say redline was interesting. So I could do dll list. And let me see if it’s going to do it. The, other thing that I like about recall is it’s got tab autocomplete.
Now, anytime you’re trying to run something and you don’t know exactly what that process or what that module was. In recall, you can just like type D, or I can type p and hit tab, or I can hit s and hit tab.
And it’s going to show you all the different commands and the different things that you could run for this particular context. So I could do services and then just tap autocomplete it and see if it works. And there’s all the services that are currently running and you can literally just spend hours doing this, just kind of going through the Alphabet.
You can also go through the module list, at the website for recall, you can see what all the different modules do, and this thing just keeps going and going and going. But these are all the services that we’re running on this specific memory dump.
So in a little bit later I’m going to talk about kind of the order in which the modules, I like to run through the modules here. so that’s very very very cool as well. Let’s talk a little bit about dumping, memory and taking a look at a memory dump of a system.
So on this particular system, I’ve got some malware running on it. I created a backdoor and I ran the backdoor and it’s connecting back into one of my ip addresses, which I’m probably going to have to burn, after this webcast because I can’t trust you people at all.
but it’s a digitalocean instance 1042-361-2492 and oh, I lost my connection. All right, so we can just go back to my image.
I was hoping that that wouldn’t happen, but it does. So I’m just kind of cheating and I’m running the malware directly. Just take this. I’ll just run it as administrator and reestablish the connection.
There we go. Let’s see if it works. Digitalocean, does get a little bit, does get a little bit funky, from time to time.
Moderator
John, we had a couple questions. do you have to know the version of Windows to build the number to use the tool?
John Strand
Yeah, you do. But remember that image info is important because image info actually gives you the capability to basically kick out what the version is and volatility will say, my best guess is this specific version and then you can go through and use it and it will give you its 1st, second and third guesses, when it’s, when it’s pulling that information.
Sorry, I got sidetracked. So never mind. The man behind the curtain, other.
Moderator
Than ps list, does recall have a module, aka like p scan that does not trust the linked list of the process, rather heuristically look for e process.
John Strand
Structure and that would be the closest thing. That’s why, Netscan, is so much better, because Netscan tries to exhaustively go through and search for all the network connections, but still it needs to know what the right place is to go look for, for and how to parse those as well.
so yes and no. I mean, it’s not as cool as just scanning through the entire thing everywhere and exhaustively looking for it. It still has to get some type of a roadmap, to be able to do that.
So, hold on, my vm. There we go. Interpreter reverse TcP. And then set l host this.
Come mhm on. There we go.
Okay, let’s see if we get a connection. There we go. All right, so now we got a session. so on this particular system, we’ve got it compromised.
And this, is just a windows ten vm that I actually. And then I can dump the memory, from it. Now, I’m going to dump it in two different ways. I’m going to dump it with access data, FTK imager, which works really, really well with volatility.
And then I’m also going to dump it with winpmem, which works really, really well with recall. So two different ways. Now, as Alyssa mentioned, when you’re dealing with, AfF compression, you get a whole bunch of things with, winpmem that maybe volatility doesn’t care about, but you can export it the proper way.
But I like just using multiple different tools. And these just so people can run multiple tools to dump memory. So I’m going to do capture memory. We’re going to call it memdump two, just because I’m incredibly uncreative.
so we’ve got that, and we’ll do enclosure. Let’s go through like this. I want it to go to this PC tools.
There we go. So I’m dumping it to my tools directory. I’m calling it memdump two mem. And I do capture memory, and now it’s dumping the memory from the computer system. I only have two gigs of memory, for this particular computer, so it goes pretty fast.
Of course, the more memory you have, the more time it’s going to take to actually dump the memory out of the system. So here it goes. Ran out of disk space. Oh, my gosh.
All right, so that’s funny. let me go through and, let’s try to clean up a little bit here. So here we go.
Moderator
Is FTK imager free?
John Strand
Yes, it is. You just go to the website and you can, download it for free. It’s perfect marketing, though, right? I mean, if you have this free tool that everyone is using, it, works really well.
All right, so I’m going to blow away my memory dumps here with my previous ones to make more room. There we go. Don’t you always hate it when you run out of disk space?
Oh, my gosh. Yeah, this is like Dallas says, someone.
Moderator
Didn’T pray to the demo gods today.
John Strand
I didn’t pray to the demo gods. It was, well, no. So. Okay, recall is done. And, let’s close it. There we go.
When in doubt, shoot it through the head. All right, so delete. There we go. All right, I’m not running it right now.
Nothing should be using that memory image at all. All right, there we go. Now it’s gone. Now I can do it. So we’ll go through and create memory image, give it the path to the tools directory, call it memdump.
There we go. Capture memory, and away it goes. And hopefully it doesn’t puke. I should probably empty the recycling bin.
Moderator
Tim said I took 504 last June and passed the GCI, in October. Is there a way to get the updated training material or do you have to take 504 again?
John Strand
I don’t think you need it quite yet. That was pretty recent. but these webcasts help. It’s kind of like I’ve been telling everyone, anytime I update anything or add new stuff, just keep coming to these webcasts and you’re going to get it.
No, problems. All right, so there we’ve got a memory dump. Hooray. Of our system with, FTK, and I’m also going to do one with winpmem. So let me cd to tools.
Cd to tools. I’m all flustered. So we just run winpmem and, you can just give it an output file and, just call it win pmem m out dump.
And then we go like that. Oh, I’m not running this administrator common error. So let’s go ahead and fix that one. So we’re going to right click run as administrator CD into tools.
Then we run WinpMem without Unix syntax. We’ll give it an output file dump and away it goes.
However, I want to show you here at the end, so you can see it’s doing aff four. Now you’re going to see that the size of the winPMm dump is going to be a lot smaller than what was done because it’s automatically going to compress it down and then recall can parse that data.
No problems at all. So it’s going through and we’re just about done. We should still have our malware session. We do.
Awesome. Still have a meterpreter session, so it’s dumping the memory off of the computer system. Now, I named the executable something very, very obvious. but you’re not going to always see malware.
That’s like 4444 msf exe. It just doesn’t happen all that often at all. So we got that going. I’m going to open up another command prompt and we’ll run it as administrator.
I’m going to show you that image info. There we go. So we got volatility and we’ll give it a file.
God, swear to God. Where did I hit a dot? See the problem is I was going through and prepping and I’m jumping over to a Linux image during memory analysis there here in a couple of seconds.
All right, so now it’s going to try to identify what the image is for this particular memory dump. And this is really really important because somebody asked this question already. They were talking about what’s the importance of running image info.
And the importance of running image info is you don’t always know what the image itself actually is at all. So while that’s running, we can go ahead and we can jump into recall and we can open up our other memory image.
Now once again, I have one memory image that we’re doing analysis with volatility and we used FTK imager for that. And we did another one that we dumped with recall with the Wayne Pmem tool from the recall project.
Now as Alyssa mentioned, it’s very very cool because it dumps the memory. You can see where it’s done dumping the memory here, but then it starts dumping all of the drivers and the page file associated with this operating system as well.
Once again, that’s really really important because you might want to do some additional analysis if you do actually have a driver that is of interest because a lot of malware likes to pretend it’s drivers and so on. Hold on a second.
Oh God, it still hurts to cough. they say that my ribs should be pretty well healed, but they still hurt a lot. Alright, so what I’m going to do is I’m going to cd into program files and I’ll go into the recall directory and we’ll just run recall.
And then we’re going to run it and we’re going to open it up against a file directly. So we’ll run it against tools. Come on. There we go. And we’ll run it against the win pmem memory dump that I just did.
And then recall is really really good at automatically identifying what the specific image is. So here in a little bit you’re going to see a recall prompt that jumps down. Volatility is currently taking its sweet time and the reason why is it’s actually going online and it’s going through and doing a full search through that memory dump to try to identify exactly what type of windows operating system it actually is, as is volatility, right now.
So there we go. So volatility won in that little race. But remember, volatility at this point doesn’t really have very good connection information. So if I try to do Netstat, volatility is just going to puke all over itself.
Of course, being the fact that the demo gods are currently failing me, it’s probably going to work, but I doubt that varies. But that’s because they haven’t actually put in the, linkages as far as memory analysis exists.
So it’s going through, and I’ll show you how Netscan actually looks different, and I’ll actually go through and dump that information real time. So while those two are churning and burning, let’s go back to our slides because we don’t want to stop things, and I want to talk about the overall process of doing memory analysis and how you should start it.
Now, whenever you’re working with a memory dump, I really like to start by trying to identify the remote connections first. And this is where Netscan comes in and is very, very important because it’s not just going to show you active, established connections.
It’ll actually show you some connections that are dead as well, which is very, very, very cool as well. Now, we did have a question Dale asked is, does, FTK imager need to be installed on the system or can it be run from a USB?
It actually can be run, from a USB stick. So you can actually have that as part of your forensics analysis toolkit that you carry with you. Somebody asked. The rip thing was real.
Holy crap. I thought it was part of a country song or something. No, it was very real. broke two ribs, one of them in two places. They actually went floating. I sneezed three days after breaking them, and then I punctured a lung and spent two days, in the hospital, and that sucked.
That was not pleasant at all, and it’s just still a little bit sore. All right, so I like to start with network connections. Now, please understand, there are malware specimens that you won’t really see established connections.
Once again, Netscan, when available, is important for looking at those different systems. So you need to have that capability, but you have to start somewhere.
So they might be using DNS, they might be doing some kind of weird ICMP protocol, but you have to start someplace. And this allows you to start by basically mapping out what are the remote ip addresses the system is talking to?
What are the remote ports the system is talking to? What’s the process name, its parent process id, and its current process id, you can fill that out. Also you can populate another table.
Once you have the information for the network connections. You can then fill out the process id, the parent process id, the name of the process, and then the command line invocation. So what we’re really getting is two separate things.
We’re getting the remote connections and we’re taking that table, the information that we have from this table, and then we’re populating another table which allows us to see the command line invocation of the different systems, or excuse me, the different applications that are currently running.
And the reason why this is so critical, and I talk about this in sans 504 quite a bit, is because whenever you’re trying to find something like you’re doing forensics analysis, it’s not an issue like it is on Hollywood where people are just sitting there and typing really, really, really fast.
Clickety, clickety, clickety, clickety clickety, and the answer comes up and bites you in the ass. A lot of times it requires us to kind of create these tables and be more analytical and try to treat it more like an accountant, more than, just simply trying to be a television hacker.
I’ll talk about why that’s important a little bit later and we’ll talk about the importance of drawing things out. So here we got the processes by connection. So I can go through and I can run this over here on my Linux computer.
So we got that. It’s completely froze up, which is awesome. That’s great. That’s okay. We don’t need it. I have another system, not Netstat.
Netstat. There we go. So here’s a memory image. And you can see that the memory image actually gives us out the established connections.
Now once again, we’re trying to make these established connections really, really, really easy to identify. just because it is a nice little webcast and here I made the font a little bit bigger for you.
And you can see of course we got Netcat running and it’s impossible for me to do anything without having Netcat involved. But we got Netcat. We also have a program called Hotpix Exe, we have metaserver Exe as well.
I’ll talk about how those are all different here in just a little bit. But it’s helpful. You can see the local address, you can see the remote address, you can see the state. And you can see that it’s actually established and the process.
So if we’re going through and trying to fill that out, the information that we have whenever we go through and dump it is all right here. We can populate most of that table. We have the process id and we have the name, we have the remote IP address and we have the remote port.
We can go back and run PS list and we can see the process hierarchy for the image as well. So we can see that we have the process ids.
Let me go up here. There you go. We have the process ids of interest. We have 400 817 40, 417 48, 3614 28. And if you see any of those that are interesting to you, you can then drill down.
You can say, well, the metaservice was started by the process id 496, which is services Exe. And I’ll talk about what that is here in just a little bit and how that works. And if you look at 3600, 3600 was started by 20, five, one, six, which is explorer exe.
And I’ll talk about why that’s important as well. but yeah, there’s just tons of different things. And the pstree is helpful because we can actually see that hierarchy.
what process started, which other process. So here you can see hotpix exe with a process id of 3600 started and spawned off 400 and 818 40 417 44, 21 24.
And we’ll talk about what all of those are doing and that ties back to the network connections as well. So a number of ways that you can actually track what the processes are doing. Get the process hierarchy determining command line invocations.
so if we have a process id in this situation of 1744, so we can do dll list pid equals 1744, hit enter.
It dumps all the dynamic link libraries associated with that specific process and it actually gives you the command line invocation. Once again, incredibly useful stuff when you’re doing base forensics analysis on systems.
So now if we take this, like I said, kind of having that sheet and drawing that stuff out is incredibly important. So you can actually pull that. We got the netstat, we fill out the table with the network connections and the parent process ids and the process ids.
And then we pull the dynamic link libraries associated with each of the processes. And we actually pull the command line invocations and we fill out the table the rest of the way. And we can see if we fill this out, you can see which processes begat which other processes.
Now we don’t actually have the entire picture yet. Some people can see exactly what’s kind of going on here with pivots and relays. But it really helps visually to start drawing this out because you can see that we have systems that we have like 145, we have 137 that are being talked to through this particular computer system.
And we can draw that map out. So we have the victim’s computer system, we can see that we’re shoveling a shell CMD Exe. We have a port forwarder going through the victim as well. Now, it’s not always this clean, and I need to make that very, very, very clear.
But the reason why this is key is because this more important is the process, right? We can dump the memory in multiple different ways. We can identify the images, what is the, version of the Microsoft Windows or even Android or Mac OS X or Linux operating system that you’re doing analysis on.
And then you can start kind of piecing all of the processes and the network connections together. So you can get to this point where you can basically draw out what is the system communicating with remotely.
So you can see that this victim is talking to 145 and it’s also talking to 137. So if you’re doing a forensics investigation, it becomes very important to start now looking at these other two computer systems.
What processes were they running where, they actually compromised as well. Because if you were just looking for Hotpix Exe, which isn’t necessarily a red herring, but it’s not the only thing that matters when you’re looking at the malware on this computer.
And so many forensics investigators would stop at just looking at Hotpix Exe and say, that’s the malware, delete the malware, reboot the computer system and go back, but they’d still be completely inevitably hosed.
In this situation, you found the easy thing. But there’s some things that make this a little bit harder. So Hotpix Exe, right, we can see, appears to be the root of all evil, but we can see that it was invoked by Explorer Exe.
And the reason why this is important is on some versions of Windows, if you double click on something, it doesn’t say that John Strand was the process that double clicked on that executable or ran that executable.
It actually is invoked by Explorer, which is in fact your Windows GUI. So in this situation, very clearly, Bob went through and double clicked and ran on an executable called Hotpiks Exe.
Full story of what actually happened is it was a PDF that dropped an executable and then automatically tried to execute the executable. so we have that particular executable, but there’s more to it if we dig in some additional processes.
If you remember our connection table, if we go back here, with our connection table, we have metaserve Exe as well. Now once again, these names can be changed, and most likely they will be.
But metaserve Exe is interesting because it’s running metaserve DL, it’s the meterpreter. Now the reason why this is important is because newer versions of metasploit, whenever you exploit a system, it doesn’t actually drop a dynamic link library on the system.
what it’s actually doing is it actually injects into an existing process escape JT memory injection. However, this is a persistence mechanism. Whenever you run a metasploit shell, as Joff will tell you, one of the first things you do is you run persistence to get another shell back out of that computer system.
And on some versions of Metasploit will actually drop a dynamic link library on the computer system so that it can load the data that it needs. Now, I think it does, a VBA file does something a little bit different on newer versions of metasploit, but it’s going to leave an artifact behind.
So what we have here is two separate shells that are basically connecting back to the bad guy. We have the meta service, which is a dynamic link library as part of persistence. And we also have the initial attack and exploitation.
I’ll show you what that looks like as well. And we can actually drill in. So you can see Dllspid equals 920. We can pull that as well. So I can go through DLS 920.
There we go. And you can see that I’m doing a little ping sweep from the command line to find additional systems, to be compromised. But yeah, you can go through and pull all that and then fill out your tables, which is just cool.
Now the final thing I kind of want to leave you with is this is kind of the process map of everything that happened in this memory dump that we just ran through on my Linux computer Explorer Exe invokes Hotpix Exe and then Hotpix Exe kind of sets up that pivot.
It connects to the additional computer systems, shovel shells and sets up the port forwarding. It also started up the metaservice Exe by invoking services ExE and then services Exe started up the meterpreter service.
And then you have the server running on the computer system. So we really have two separate things going on. We have the initial attack and pivot and we also have persistence as well. So let’s go back to my vms.
as I said, they’re a little bit under power for these things and it looks like they’re both completely frozen. Actually recall is frozen, but I want you to look at what volatility, did.
this goes back to that image info being important with image info. I gave it the memory dump that we just created. And image info says that this is a Windows ten. It could be 1058, or it could be a 14393, could be just Windows 1064.
So it’s giving us a number of options. So now whenever we try to run something with this file, we can go like let’s say ps list like that.
And then we could do equals and we can actually put in one of these profiles. And now volatility has a good idea of where it actually needs to go to actually pull that information.
so that’s why that image info is so key, because now it knows where to go. So there’s a tool that automatically tries to identify what the specific image itself actually is.
And then you would use that to specify and fill out the profile, to be able to run the different commands. So if we go through.
Alyssa
Let’S see.
John Strand
If Netscan is going to work with the memory dump that I just created. Take it, run with it, go. Volatility, go. So just kind of trying to tie things up.
Let’s go through the beginning, all the way through the end. The first thing is you have to acquire the memory. Now in this example I used two separate utilities to do that. We used winpmem. And one of the nice things about winpmem is it actually dumped all of the drivers and the page file from the computer system.
And we also used FDK imager. One of the nice things about FDK imager is it’s very compatible with volatility. Once you have those memory images, you would then use a tool like image info which would go through and identify specifically what is the version of Windows that the memory dump came from.
And you can use that in a number of different memory dumps as well. So there we go. We’re actually getting it. Awesome. And then once you have the image information you would actually start going through and doing analysis.
Now the analysis. like I said, I like to start with network connections. Look at the different network connections that are being made for remote computer systems. Here we go. And then based on the network connections that are being made, or if you’re looking at Netscan, the historical connections that were made, you would then start filling out, what are these executables?
How are these executables actually invoked, and what possibly was the root cause? How did these executables get ran? So that’s kind of a quick overview of memory analysis.
I’m going to go ahead and I’m going to open up the panel, and we got a bunch of questions that are in here, which are very, very, very cool.
Moderator
I think most of them got answered. Joff was cool enough to try and help me answer them. So if you still have questions that haven’t been answered, go ahead and ask them. it looks like Keith says, will slingshot become publicly available or just a stands conferences?
John Strand
Yes, we’re trying desperately to get it to be something that is publicly available, like the, sift workstation. And we actually have a meeting not next week, but the week after.
We’re getting all of the GPEM instructors together, and we’re going to talk about what slingshot should be. And I love the guys, at Colley. I love what they’re actually doing.
just really, really a lot of respect, but we kind of need to have an image that’s dedicated to our classes. while Kali is absolutely awesome, we don’t have control over it. And whenever we write labs, things may not always be consistent.
So we’re hoping to release slingshots, not necessarily directly to compete against Kali, although, let’s be honest, yeah, that would be kind of cool, but there’s no way. I mean, it’s just an absolute juggernaut, but it’s more in line whenever we do these webcasts.
Then you guys can play around and follow me. so we have a consistent image that we can all work from. So hopefully that’ll be coming here soon. Do you use Moonsol’s dumpit?
no. Usually I just kind of stick with ftk, imgur, and winpmen. Probably more heavily on the winpmem side, than even ftk imagery as well. A, lot of the volatility stuff can be learned using the art of memory forensics and brought up, and that’s an outstanding book.
Absolutely. Check that out. Also, we have the sans class of memory forensics. I actually recommend people do, like, before you come to a sans class, it’s important to, like, see these webcasts or go and find a book by the author of the class to kind of prep yourself because, Sarah was giving me a little bit of a hard time about how fast we go.
And this speed at which I talk and Alyssa talks fast is how we go for six days and people’s heads explode. quite a bit. And it really helps kind of pre the gears as well.
sorry if you already answered this. Have you had any luck decompressing windows ten high fall and using volatility on it? Nope, I have not yet. and that’s kind of getting to the problem of windows ten as a whole.
Microsoft is keeping a lot of the stuff associated with Windows ten very close to the best. So we’ll see if they open it up.
Moderator
I like that Nathan said learn faster because, yeah, you do talk so fast. We are trying to observe it as fast as you talk, but that will be maybe in the future a little bit.
John Strand
how to teach John to slow down.
Alyssa
No.
John Strand
James brought up a very question. Mandiant redline. Do everything we did today? Yes and no. where you have recall and volatility have some issues with memory structures on windows ten, redline seems to be even worse.
so for example, with the memory image that I just did of them that I created, if you actually try to open that up in Redline, it tends to puke all over itself and blue screen my computer. and that’s because Mandian has their own commercial tools that they’re keeping up.
They just haven’t really kept up redline as much as I’d like them to. If you’re working on like a windows seven image or a windows eight image. Yep. That’s absolutely awesome. it’ll work. It should work just fine with those as well.
Doesn’t, a lot of what I discussed today require two people typing on the keyboard at the same time? Thankfully, no, it does not. so let’s see, any other answers?
Wow.
Moderator
Jeff was says Redline isn’t being updated anymore or developed, so.
John Strand
Well, that explains a lot then why it’s not working. So there you go. Frank basically confirmed what I said and Frank’s much better at the stuff than I am.
can you remember a good tool for Linux? Linux OSX volatility. And those tools actually do memory analysis on OS x and Linux. Of course, specific versions are kind of hit and miss and it can be somewhat difficult, difficult as well.
How did I break my ribs? Now we get to painful topics. I was walking across a parking lot at a ski area. I have my daughter’s ski boots. One, was on the front and one was on the back, and I was carrying a bunch of stuff, and I slipped and I fell three Stooges style, where I slipped out, and then I came down really hard on the, on the, ski boots, and, I broke them, and that sucked.
It was kind of fun. My daughter has her learners pyramid. And I got into the car, and I’m like, take me home. And, she’s like, no, I’m taking you to the hospital. Because I was screaming and cussing. I bet you it was really, really funny to see me in the parking lot because I immediately get up and I’m, like, ready to punch anything, and I start screaming at the ground.
I’m like. And I get into the car and, I go into shock. And, pretty much she took me straight to the hospital after that, which I’ve never been in shock before.
I’m like, I can’t stop shaking. Apparently, that shock, which I thought actually made the pain lessen. It doesn’t. So, will you be giving out CPE credits?
Will you be giving CPE certificates? I don’t hand out CPE certificates, but go ahead and put down this webcast. and I’m pretty sure that people with ise squared don’t mess with these webcasts. If you provide the link to the webcast, that’s usually enough.
And also, if you get into an issue with IaC square, just shoot me an email and I’ll vouch for you as well. that’s not a problem. The people at ISC squared are pretty cool, which I thought I’d never say, like, ten years ago.
but the people at ISC squared are pretty awesome, to work with. They’re very, very, nice to people. at least you confirmed your rib incident didn’t start with, hey, Paul, hold my beer.
Well, yeah, hey, Paul, hold my beer. I’m going to carry my daughter’s stuff across the plane. Parking lot would be really, really lame. can you do these with shadow copies? No. however, one of the things, that is pretty cool is with a virtual machine, you can snapshot the virtual machine, and then you can feed the snapshot image through, many forensics tools.
And that is just awesome, because then malware doesn’t have a chance to see that it’s actually being analyzed at all because it’s being pulled. The, memory is being pulled at a much lower level, which is very, very cool. Good question.
Very cool. All right, guys, that’s all I got. thank you very much. And guess what’s coming up? Our next webcast is Rita. And for the Rita webcast, you guys are going to, be installing Rita, and we’re going to be doing analysis on brologs, and we’re going to be capturing beacons and analyzing beacons within an hour.
and for that webcast, go out and download ubuntu 1604. And I also have it running on Linux Mint. but get your 1604 image built, don’t do anything to it, just build your image.
And I’m going to go through, I’m going to give you guys all the instructions step by step on, how to build out Rita in like one command. And then we’re going to import logs and we’re going to start doing awesome things.
It’s going to be really, really cool, super excited about it. But you guys can literally have Rita, in your environment ran. And if you don’t know, Rita is real intelligence, threat analytics, and it is the, tool that we have for beaconing analysis, for going through bro logs and looking for beacons.
so that’s our next webcast. I’m stupid excited about it. Hardware recommendation, just. It’ll work inside of a vm, just make sure that you give it to two cores. And whenever I shoot out the email through mailchimp talking about what we’re going to do in that webcast, I’ll tell you guys, to have a Vm with at least two cores, because, Golang takes advantage of those two separate cores.
60, four bit is the one that I used, Jacob as well, so. And it’s. Yeah, and Dale, it’s the same install script. it’s the exact same, install script that I gave a couple of weeks ago.
So we have some really cool stuff. Get an Ubuntu system set up, set up, dual core, 64 bit, give it about two gigs of memory, and you’re going to be ready. and so it’s going to be, it’s going to be very, very, very cool.
All right, guys, I’m excited. I’ll see you guys in the next webcast. Thank you so much for coming and hanging out. I’ve got another meeting to go to. This has been recorded. We will share the recording. The slides are at tinyurl, um.com.
and this is if we missed something.
Moderator
Ask us on Twitter.
John Strand
Ask us on Twitter. so thank you so much. Gotta go. Take care, everybody. Bye.