Instructor: Hayden Covington
This Anti-Cast originally aired on March 5, 2025.
Want to discover what life in a SOC is really like – beyond the marketing and myths? Join us for a free one-hour training session with Antisyphon instructor, Hayden Covington, and other members of the Black Hills Information Security (BHIS) SOC team. They’ll pull back the curtain on the realities of life in a Security Operations Center.
They’ll share authentic experiences and insights about what working in a SOC truly means and what you should know if you’re considering this career path. You’ll learn:
- The truth about day-to-day SOC operations
- Real challenges and rewards of the SOC analyst role
- Career growth opportunities in security operations
Highlights
Full Video
Transcript
Hayden Covington
All right, well, thank you everybody for coming to this webcast today. I alluded in the Discord that things might be a little bit different than usual, and that’s because I have a panel. I chatted with Zach some and we bounced around this idea.
But the goal today, which I’ll dig into a little bit, is just to talk about SOC and to get rid of a lot of the, illusions or, misrepresentations or misunderstandings about what SOC work really is, what it looks like, what it feels like, how to get into it, all that stuff.
I’ve been collecting questions from Discord, from LinkedIn, from, just people asking me over the last few months. And my goal today is to kind of posit a lot of these questions to the folks that are on this webcast with me.
And they don’t know what these questions are ahead of time. a few of them have asked specifically for them and I told them no, because what I want is raw responses from these guys. I want them to just give me, true raw responses.
So before I get any further ahead of myself, I want to introduce all of them. My name is Hayden. I’m a lot of things in the SoC. I also work on the D4 side of things. but let’s, let’s jump to Will.
Will, do you want to introduce yourself briefly?
Will Corbin
I’m Will. I work in detection engineering and help out with tickets and triage.
Hayden Covington
Awesome. How about you, Paul?
Paul Lammers
Paul and I basically do the same thing as Will. I a lot of detection engineering and then also do tickets, triage, on the side.
Hayden Covington
Awesome. And then Tom.
Tom DeJong
I’m Tom DeYoung. I work in the soc primarily, but, I’ve been helping out on some, incident response engagements recently. So, we’re trying to learn as much as we can while we’re here. But yeah, hop around between those two, mainly.
Hayden Covington
Awesome. Okay, and so that’s our panel for today. And I already talked a little bit about. We want to cover the realities of things. We want, to take your questions. We’ll take some questions live near the end of this webcast. We have some pre center pre prepared ones.
But also part of that is my promise is that I’ve mentioned to these guys, I want you to be entirely open and honest about your responses because someone mentioned it might have been Tom earlier that Red team is the sexy side of security in a lot of ways and SUC is unsexy.
And that is and isn’t true. There are definitely parts of it that can be pretty tough. and if you don’t realize that coming into SOC work, it will be a rude awakening. And so I want to cover some of that stuff today.
Cover a lot of those, nuances and things before somebody kind of suffers from not understanding that. So, again, we have some prepared questions. Let’s start it off simple. I’ll.
I’ll give my input and my feedback, and then I’ll toss it out to, how it views on my webcam, which would be Tom, wheel, Paul, and I’ll just kind of ask you guys if you have anything you want to add, and we’ll see how this format works.
So the first one that someone asked is they wanted to know what kind of skills or experience do you need to work in a soc? So, from my perspective, what this really means to me is that you could, you could, you could get into a SOC with really any skills or experience you can.
We’ve had a lot of people come into the Black Hill SOC with very minimal technical experience. but a lot of the pivot that I see comes from originally from help desk. So going from help desk understanding, ticketing, understanding, like backend computer inner workings is a very easy transition to SOC work.
And I’ve told a number of soc analysts this before is I could train you how to do soc work in a day or two or a week max, and it’s just the hard part that you can’t really train is the, the back background, abilities to research, and the desire to find the answers to things.
Tom, what. What sort of skills do you feel like you need to be a SOC analyst?
Tom DeJong
I think, I think those are all very important skills. I think that soft skills are also very important. You, got to be able to work as a team, and, you have to. I feel like this job will humble you. so I think, I think, the more you can take it in stride when you have make a mistake.
I think that’s going to be something that’s going to be valuable to you. just know all of us make mistakes all the time. And that it’s okay to ask questions. I know that we’ve had people come through the SOC that felt uncomfortable asking questions, and it happens to me all the time.
Tom DeJong
I’ll be sitting there, I’ll have something typed up, and I’ll be like, is this stupid? And I’ll, And I’ll be like, no, you Just gotta send it. And I’m sure it’s helped other people in the past. So I think, I think being able to admit when you’re wrong and have some soft skills and teamwork is very valuable in the society that.
Hayden Covington
That is huge, man. Soft skills. We, that is one thing that is extremely difficult to teach. that’s. Yeah, we, we expect people to have those when they come into our interviews. As we, we can’t teach you how to be responsible with your time and to, self manage your workload.
We can’t, we don’t have time to teach people how to do that. Will, what about you? What, what skills or experience do you feel like kind of helped you as you, worked your way through SOC work?
Will Corbin
I think you guys touched on it pretty well with basically you need to be able to react in the moment and research stuff right then and there. I think it’s a big misconception, especially with people starting out in the SOC.
I know I had this issue a year ago when I started is you feel like you have to know everything up front and that’s just an impossibility when it comes to technical things. Things are changing so much constantly.
The way things work change constantly. So you’re never going to be able to know everything. So you really need to be comfortable and I call it Google fu. and you need to be able to research topics as you go and just know that you’re never going to learn everything.
But definitely don’t stop trying to learn.
Hayden Covington
Yeah, that, that imposter syndrome is real. And it came like it. I have it too many times and I’ve done this for a while now. I don’t know how long I’ve done soc work, like seven years or something at this point.
It’s been a while. And it really didn’t connect with me or click for me until I was on a detection engineering contract with a very big company and one of the guys that I had like read about in school was on this contract with me and I asked him a question about something and he was like, man, I don’t know.
he like had no idea either. And so this really like helped connect in my mind. The ones that are the best in this industry and that are, the smartest are just the ones that know how to find the answers.
Like they know how to go fix things and they know how to go figure things out. And again, we can’t teach someone how to go figure something out. They have to want to figure it out. yeah, that’s, that’s a great point.
Paul, what about you, man?
Paul Lammers
So before I really got into soc work, I did a lot of classes. And every time you’re in one of these classes, everything is a threat. Absolutely everything.
one thing you need is patience because in real life, 90, 9% of it is going to be false positives. Like it, you’re just closing one false positive out. Another because, most of the traffic is actually good.
Some of them’s weird. Some, some programs decide to do something strange, so you have to look into it, but it’s, it’s going to be good and that, but you have to still be ready and not just close out everything because every once in a while you’ll find that, DNSC2 or, actual cobalt strike going off.
So it’s there, but patience.
Hayden Covington
Excellent point. That, we, I was teaching someone how to do SOC work today, from our red team side, and I think they mentioned, like, ghosts in the machine, like seeing ghosts. Every single analyst that we bring in that I’ve ever worked with does that computers are, I’ve said it a lot.
Computers are weird and they do weird things. And most of the time that weird stuff is just weird. It is not evil. And one thing that one of our IR guys says frequently is almost all of the time evil is very clear.
it is not frequent that it is well hidden. There is always going to be something that sticks out very clearly. yeah, so that. Great points, all of you. I’m already, I’m very excited about this format.
I think we’re hitting, hitting things very well already. but speaking of sort of hitting things very well, interviewing someone specifically asked how to prepare for SOC interviews, and we’ve just changed our whole interview process on the Black Hill Society.
so I, I, I’ve built out some technical questions and we’ve gone through this whole, whole process. But how I think you guys probably would have interviewed for the soc, a couple of you. Not all that long ago. So, so what do you do to prepare for a SOC interview, Tom?
First, let’s hit Tom.
Tom DeJong
I will say it. I don’t think I had an interview for this job. You told me to come to the first Wild West Hacking Fest and help out.
Hayden Covington
You were an intern too, though, weren’t you?
Tom DeJong
Yeah, I was.
Hayden Covington
Okay.
Tom DeJong
but while I was in school, I did, we did do a lot of studying on how to be, successful in Interviews and stuff. So whenever I have an interview, I try and do, as much research as I can about the company.
I will go through and read on like, just online about what common interview questions are. I’ll try and come up with, like, preset answers. And, a lot of times they don’t ask it, but still, it’s getting these thoughts in your head about who you are, what you bring to the table, what you’re good at.
So I feel like just practicing going through interview, example questions online can be really helpful. And, I’ll circle back to it. But when you’re in the interview, it’s okay to be wrong about something. Or if they ask you a technical question you don’t have the answer to, it’s okay to say that you don’t know.
I feel like that’s going to be good for both parties. just. Just, being honest about where, what, what level you’re at, so that it will work for both of you guys. you may not be a good fit there. And if you try and, lie about your true experience, that could end up being bad for both.
Both of you guys. So. So, I think, I think. I think, practicing example questions is, Is the way I prepare them also.
Hayden Covington
Yeah, great points. I’ve done a lot of interviews for soc folks or interns or stuff lately, and one thing that I always really appreciate is just honesty, is because if we can bring you in and know where you’re going to need extra help, we know whether or not we can actually give you that help.
and kind of like you just said, Tom, if you come in and you’re lacking in an area that we don’t have time or availability or people to help you in, and you misrepresent that, it’s going to be really hard for the company and for you.
but folks coming in and saying, I’m a little weak in Active Directory, we then know, okay, well, we have Kirsten in the Sock, who’s a rock star with Active Directory, and he could help teach this person that, and that’s something that we could do.
yeah, no, that’s a great point.
Daniel Lowrie
Will.
Hayden Covington
How about you, man? Did you have an official interview?
Will Corbin
I did. You were at it.
Hayden Covington
That’s right.
Daniel Lowrie
I was so appealing.
Hayden Covington
It was so long ago.
Will Corbin
It was only crap. It was December, not, last year, the year before. But so, yeah, I did have an interview. it was for me to get ready for it. Basically I kind of just since I was introduced through John’s classes and things like that, I just made sure I touched up on those classes again.
And I actually kind of did a little digging into some of the people I knew were going to be at my interview. did a little osin on them and had some questions that I had asked about them personally during the interview.
my big recommendation for anybody out there looking for work, especially in such a big competitive environment right now, is social skills Network, Network, network.
And I don’t mean just like go meet people, introduce yourself and exchange info, like actually talk to people. Make sure you’re building relationships up with people and then you never know what happens when they’re looking to hire somebody.
You might get a random phone call. that’s kind of my biggest advice on the hiring process is just don’t go in too stressed, do a little research on the job you’re applying for, the company you’re applying for and then, just try to build relationships with those people because you’re gonna have to work with them if you get hired.
Hayden Covington
Absolutely, yeah. And the another thing that I find I’ve been asking lately is like the tail end of interview questions is I ask people what their home network looks like because to me that kind of gives me an indicator of, how interested you are in this work.
And I know, this is a job, right? No one’s going to be interested in this all of the time. That is not an expectation. Even at Black Hills, like a company like this, it is not an expectation that you’re excited and bushy eyed, to go in and work stock tickets every single day.
That is not the expectation. But we would much prefer people that have a genuine interest in, in the field and if I ask somebody what their home network is and it’s a router and that’s it, I’m kind of like, well that’s not a disqualifier.
But if you ask someone and they’re talking like I talk with Ethan a lot about our networks and I have a ticketing system for some godforsaken reason and I have, all these different management servers and failovers and all this other crazy stuff that is unnecessary, but sort of what that question gives insight to is whether the person finds this work enjoyable.
that’s, that’s one way that you’re going to connect with like minded people. Especially at a company like this. Is everybody actually enjoying the work that they’re doing at least at a base level. And that makes it easier to get through the hard days.
how about you, Paul?
Paul Lammers
I’d, say research. Well, first I got a question. Do you make your cat, like, submit a ticket whenever it has problems? Anyways, now research the companies. like, you got to know what company you’re going for because honestly, the way you prepare for the interview depends on the company.
I definitely. So I may m not be the greatest person to ask because I’ve only done one interview in the last 20 years and it was both Black Hills. so. But I went ahead and researched.
Like, I went over my coursework, my books back there, and just went over a lot of technical stuff. Even though they did, I don’t think they asked me a. One technical question that they were, they more looking at whether or not culturally.
But look at the culture of where you want to go. One, see if you actually want that job. And two, learn what they want. That way you can present that way.
Hayden Covington
Yeah. And that’s a luxury that you won’t always have, being able to know if you would want to work in that place’s culture. But as much as it’s possible, it’s important because I remember before I came to Black Hills, I interviewed at a number of other places and one of them sounded like cool work, but I wanted to get a vibe to the culture because if, I was going to hate working there, I wouldn’t last very long.
And, I asked if I could, chat with their team lead to get a vibe on the culture. And this director said, sure. but any questions I asked, the director was also on the call and answered all the questions for this team lead. And I was like, ooh, I don’t want to work here anymore.
Because the, it’s, it’s things like that. But no preparing for the interview. You guys have all pretty much harped on research and preparation, and it sounds a little cheesy, but if you prepare for an interview, even at a basic level, like understanding the job requirements, understanding some basics of the job, you’ll.
You’ll do so much better. You’ll be less stressed. it’s. That is those little things that add up. this one is one of my favorite questions is what does a day in a soc analyst life really look like?
Like that. That changes day to day for a lot of us. But the question really is like, what does that look like? What sort of challenges are you facing? What sort of Rewards are you facing, Tom? I’m going to see if you want to go first on this one.
Paul Lammers
Yeah.
Tom DeJong
so a typical day. I guess it depends on the day, depending, on how busy it is. There’s m. A ton of variables, but I would say a lot of my days spent working tickets, which, the tickets can vary in length of time too.
sometimes you might see a duplicate ticket. Okay, you can close that out in five minutes, link some other resources where it’s previously been investigated. But I mean especially when we have new rules being implemented or new customers onboarding, these things are going to add a lot of time.
So if it’s a brand new alert that’s, or a brand new rule that’s firing, it could take some, it could take some time. So then you’ll move on to doing your research about what this is. so I, I guess, I guess there’s days where I’m just working tickets all day and those days kind of suck.
But hopefully, hopefully there’s other days where you can do some side project or some training. or. Yeah, I don’t, I don’t know. We seem to have always side projects going on or for me personally there might be a couple days where I’m working an incident response engagement.
so, so I think it’s important to have variety in your work because I think if you’re only working tickets, you are going to get burnt out after a while. So I, I think if you can get away from doing tickets all the time, I think that’s very useful for things.
Useful for your, your well being. so, so I like to try and do some different things, learn some different things. But, but yeah, totally depends on the day.
Hayden Covington
Yeah. And, and that’s another question we ask in interviews too. Kind of circling back is like what sort of hobbies do you have? Because if you’re your only hobby is work and like the same work that you’re going to be doing, you’re pretty, pretty high chance of burning out.
because like Tom said, sometimes the whole day for a SOC analyst is working alerts and those are the days that sometimes they’re okay, sometimes they’re exciting, sometimes they kind of suck. that’s just a reality of pretty similar to help desk is sometimes it’s just, you’re, you’re doing the chores basically of knocking these things out.
Tom DeJong
well, I will say I think coming to I forgot to even touch on the rewards. But I love, coming to the end of like a three hour investigation feeling like very certain that this is something that’s not malicious or malicious.
I, I just really like the investigation a lot. So I feel like that’s one of the main, main rewards. And I also love too, when we get some analysts together as a team, and you can solve these tickets together and come to these conclusions together.
I feel like there’s like, I’ve always thought camaraderie between teammates is really cool. So, I feel like that’s another really rewarding aspect.
Hayden Covington
Absolutely. Well, how about you, man? What sort of challenges and rewards would you say? Kind of follow the life of a SOC analyst. And I know you’re, I know you’re more DE now, but that also applies, right, because we’re all one soc.
Will Corbin
Yeah, it absolutely does. So over in de, basically the general day is spending the day doing like rule change requests, which are filters for tickets that analysts have found are, benign behavior.
a lot of times that also involves us having to go back over those tickets as well, just to see the logic of the investigation, see if the client themselves confirmed the activity or if it was even necessary to go.
and then overall we have to maintain the rules that, search out and find this activity to begin with. So in that vein, it’s actually, I’m going to touch back on how people are talking about purple teaming and stuff. It’s actually really good for anybody interested in doing DE to learn red side as well.
So, what, what you’re looking for to help write those scripts to trigger alarms and things like that. And then of course as it gets busy, I will also step in and triage tickets.
Hayden Covington
And yeah, the most, the best detections that I’ve ever written are, times that I’ve gone and actually performed the attacks because that’s the best way you’re going to get the most realistic data and.
Will Corbin
Right.
Hayden Covington
So that’s not always possible, but the closest you can get.
Will Corbin
well, we just, DE kind of currently just set up a playground for us to do that specifically.
Hayden Covington
Nice.
Will Corbin
So I am actually looking forward to that.
Hayden Covington
We’ve had a couple detection engineering customers in the past that were like, yeah, we want you to run the malware. We’re like, are you sure?
Will Corbin
Right? We’re like, you sure?
Hayden Covington
We will if you want, as long as you’re sure.
Will Corbin
But, now we actually have a safe space we can detonate that and.
Hayden Covington
Having a detection lab is good.
Will Corbin
No longer do that. And like, the rewards to me is basically knowing that we’re helping protect people.
Hayden Covington
Yeah, absolutely.
Will Corbin
Too many businesses get hit, too many people’s information gets leaked out because of it. It’s, it’s, it’s a nightmare out there. Trying to protect your information these days is crazy. So to me, the, the benefit and the reward is knowing I’m helping out in that some way.
Hayden Covington
Awesome. And how about you, Paul? You come from a different world than, than most of us coming from like the army. So how does that really compare and like the rewards and challenges that you’re going to face?
Paul Lammers
so. Well, back in the army, it’s monitor most of the time and the networks that are on were all air gapped, so there was very little unexpected traffic unless the red team was actually after us.
And then it stood out like a sore thumb. Now once I came to the SoC and was working, day to day tickets, like, like Tom and Will said, it is mostly just doing tickets.
But find your. Find time in between to do some training, learn something new.
Hayden Covington
Yeah.
Paul Lammers
Now that I’m in detection engineering, I always start out my day with rule change requests. Try to find those filters that we know are good. The goal, I’ve said it many times of the goal is the lowest number of false positives with zero false negatives.
I mean, I’d rather go investigate 10 things that are benign than have one malicious thing slip through. if it gets up to 100, then you’re probably going to ignore that one anyways.
Hayden Covington
You are. Hey, did you take my class by chance? No. Even in the workshop on Friday, I have a whole couple of slides on risk acceptance and risk strategies for filtering and detection engineering. that talk exactly about what you’re talking about is you can overload your analysts to such an extent that they will just not see the true positives anymore because they’re so tired and so overwhelmed with just raw numbers.
Paul Lammers
I basically audited your class. I kind of watched it, but didn’t, do the labs and stuff.
Hayden Covington
Yeah, Paul was one of the original test people for my Foundations of soc class. That’s true. He was. Back when you were not even full time with us.
Paul Lammers
I was still interned or Skillbridge or whatever you want to call it.
Hayden Covington
Yeah. okay, let’s hit this one a little bit on the quicker side. Shiny tools, tools and technologies that you get to play with in a SOC. The last place that I worked at had splunk fireeye, all the fancy, very expensive tools.
and we were definitely very spoiled. Black Hills, we have Elastic and a lot of our customers have a bunch of fancy tools. So even then you get to play with cool stuff that you probably wouldn’t be able to afford on your own in your own home lab.
Hayden Covington
so I’ll ask you guys, name me like your favorite tool you’ve got to work with or experience as part of the SoC. I’m going to make you limit it to one. how about you, Tom?
Tom DeJong
I just answered a question. questions and answers. But I want to say for like 90 to 95% of my job, I can usually do it within Elastic. Okay. The SIM that we use, I, I, I think that it’s just, the sims are just crazy.
They’re, they’re really cool pieces of technology and you can do so much within them. so I want to say that’s my main tool I use, but I always enjoy when we have clients that have other EDR tools I can use. it’s always fun trying to learn something new.
so I’m all about, I’m all about doing that. But yeah, I want to say that’s my main tool I’m using.
Hayden Covington
Yeah, that’s pretty much what I would expect is the SIM is our Swiss army knife. But sort of like the MSP perspective is we get to play with so many tools that our customers are using and sending us data for. And that’s very, very cool to get to dive into some enterprise EDR platform that is little heard of or too expensive for you to buy on your own and you get to go play around with all that cool stuff.
all right, Will, name your favorite tool.
Will Corbin
so outside of using the SIM for investigations, I’d honestly, this might sound like a cop out answer. My favorite tool to use for investigations is Google.
Hayden Covington
Thank. you. Okay.
Will Corbin
Between Google and Spur, I would say that is where I spend 75 to 80% of investigations looking up all the crazy crap Microsoft likes to do in its background that makes you think somebody’s hacking you.
Will Corbin
And yeah, Spur is great for Registry.
Hayden Covington
So like, yeah, spurs to get information on IPs. It’s a tool, that you can search online with. Would highly recommend.
Will Corbin
And I’m going to disappear for two seconds while I try to fix this Mic, you’re good.
Hayden Covington
All right, Paul, hit me with your tool, man. What’s your favorite tool?
Paul Lammers
At the moment it’s going to be flare.
Hayden Covington
Oh, Flare is So good.
Paul Lammers
I love digging into it for, it just, it’s research. It, it digs through the dark web so you don’t have to necessarily go to those shady spots and gets you the intel that you need.
I’ve been digging into that some, trying to get some new, detections going on.
Hayden Covington
That’s awesome. Yeah. Flare, for those that don’t know, is like an OSINT tool for credential breaches and things like that, and it basically collects all of them into a centralized pane. and actually the Black Hill SOC is building out our attack surface monitoring platform and project for our SOC customers, which will alert on specific instances of, credential breaches.
Some of them are just, Yeah, your email and hash password got breached in Adobe 18 times, 30 years ago. No one at this point really, that’s not actionable at this point. But there are, info stealer alerts in Flare, which info stealers are going to crop up when there’s malware actively on a host stealing credentials.
and those are the ones that we want to send to customers because that is confirmed that there is malicious activity on that host. So Flare is a great one, great one. and sort of to that extent, we’re talking a little bit about info stealers.
What sort of common attacks or threats are you seeing regularly? I know what Tom’s going to say. I’m going to steal his thunder a little bit. Business email compromises are a huge one. somebody using an invisible proxy and stealing your credentials and your tokens and cookies to Azure and just logging in as you.
We, have a lot of detections on that because of how prevalent it is. Am I stealing your thunder, Tom?
Tom DeJong
Yeah, you’re all good.
Hayden Covington
Hit me with it then. What are you seeing the most?
Tom DeJong
Well, when we first started doing the incident response, Hayden and I were kind of training up together. I mean, we were seeing Gootloader. I, I remember hearing the IR or one of the Patterson basically talking about Goader all the time.
So, I feel like that’s one we got to actually work a case on. yeah, yeah, I’m gonna, I’m gonna submit that for my final answer. I’ll think I’ll try and see if I can think of any other ones and I’ll chime back in at the end.
Hayden Covington
Well, I think you’re right. Is, is Patterson gave a workshop at, Wild West Hack Infest Denver. And in that workshop he talked about a lot of Common malware strains. And he had a comparison between, I don’t know, five to 10 years ago and today.
And largely that list did not change. because the malware that is prevalent then, a lot of it is still prevalent now. It just kind of evolves and changes ever so slightly because in a way, it is a business. It’s the same way that, I would go out and pay for notion because I want to take notes.
Some threat actor decides, yeah, I’m just going to use this because it’s available to me. And some cases like ransomware, it becomes monetized where you’re almost buying a license. but in a lot of cases, this just malware that just works.
how about you, Paul?
Paul Lammers
I think that the one I hear about more, I haven’t worked on one is it’s bec adjacent of, the apps. An app is added with, incorrect permissions, and so you have more permissions than you should.
other than that, I think the only thing that I’ve really worked on is, sorry, I just got distracted by discord. Powershell scripts.
Hayden Covington
Okay.
Paul Lammers
I saw an interesting powershell script recently with, unique, obfuscation.
Hayden Covington
Yeah, Wall bins, man. Living off the land and using just native binaries on a machine. That is a lot of a shirt. That is a lot easier just to use the tools that are on every single Windows machine forever until all time than trying to bring down your own tooling.
Not just because you might not be able to, but also because everybody uses PowerShell a lot. And so being able to blend in with normal traffic, you’re going to risk throwing detections less and just if it works, it works, man.
PowerShell is built for that kind of stuff. Will, are you back?
Will Corbin
Maybe.
Hayden Covington
Sounds fine so far, man. What sort of common attacks and stuff are you seeing the most lately?
Will Corbin
I would have to say the ones that are popular. Cracking again.
Hayden Covington
That’s fine. Go for it.
Will Corbin
I would say the most common I can think of are like business email compromises and social engineering attacks.
Tom DeJong
Awesome.
Hayden Covington
BECS again. Yeah, BECS is a big one. A lot of those are, you can detect them with forwarding rules being added. We’ll see attackers create a forwarding rule to try and intercept, business email or business transactions and invoices and insert themselves into an email chain and basically be like, yep, here’s the, the address that you should pay us at.
And if they have done it correctly, the person who is supposed to be receiving that email will never see it. and, there’s. There’s plenty of cases out in the news about that kind of stuff happening where companies lose millions of dollars and don’t realize it for a while.
and those attackers definitely get a pretty decent payday off of that.
Will Corbin
And then, of course, we always see the ever present. Just password sprays.
Hayden Covington
Password sprays, always. Yes. That is a staple. we talked about this a little bit earlier. Is making mistakes. that is something that everybody’s going to do. That’s just a part of life. All new SOC analysts and all experienced SOC analysts make a mistake.
we’re all a team and it’s. There’s nothing wrong with, calling each other out when we’re doing things wrong. I think that that’s important, that we can do that without judgment. Do that with, a mindset of improvement rather than a mindset of trying to correct one another.
but, Tom, how about you give me, the number one mistake that you would think, that either you, as exact analyst experienced or that you see newer analysts experience. We’ll limit it to one so we can get through the rest of these questions.
Tom DeJong
Oh, boy. okay.
Hayden Covington
No pressure.
Tom DeJong
I feel like knowing when to ask for help and admitting when you’re wrong. I’m just going to sound like I’m beating a dead horse with this answer, but, I feel like sometimes people will, just not admit when they’re wrong or, admit when they need.
Need help with something. and, that can, that can have. That can have a lot of impacts. Like, maybe. Maybe somebody else is using your ticket as a reference later on and it actually has some wrong answer and they didn’t have time to do a full investigation.
So. So, I, I just really. I cannot stress how important I think asking for help is and how important it is to have a good, good relationship with your team or teammates.
Hayden Covington
Absolutely. And we. I’ve had that conversation with an analyst, a couple months ago, is they would put things in a ticket, an authoritative sense when they weren’t entirely sure. And I had to explain to them, there’s nothing wrong with saying I think or, based on the data I have, this is what I believe.
and so that way there’s understood to be, some sort of, not a 100% answer. I would rather somebody say, I think this is what’s going on than not really know. But Say, I know this is what’s going on, because, you’re going to take that at face value and knowing when to, admit that you’re not sure, admit that you need help or ask for help.
that’s the, the whole reason we’re not running one man SOCs. Right. We’re a team, so we can bounce ideas off of each other and ask for help.
Tom DeJong
I’m gonna piggyback on that answer really quick. But I think it’s so important to provide context in your tickets. you can just put all the data you see, but, it’s so nice to have somebody’s opinion about what they’re seeing.
and, in those, I’ll say something like, I’m not. I’m 80% sure this is what this is or whatnot, but I think it might be doing this. And, it helps. It helps people that are reviewing the tickets know where your thought process is and the things that you’re seeing and what you’re thinking.
Tom DeJong
So I just think that’s so important, as well.
Hayden Covington
Absolutely. Yeah. Yeah. Because when it goes to customers, it needs to be more formulated, needs to be more professional. When it’s internal comments among your analysts, I would much rather have somebody say, I don’t know what this is, but I think it’s this.
So that way I have a common understanding of what we’re working with and what they do and don’t know. how about you, Will? You are muted, sir, but at least your mic isn’t crunchy when you’re muted.
Will Corbin
I’m so sorry.
Hayden Covington
That’s okay.
Will Corbin
Finally dying on me.
Hayden Covington
That’s okay.
Will Corbin
Yeah, it’s. You need to keep an open mind in this job. So like I said earlier, like, when you go to research things, you’re gonna find out that something you thought was somebody hacking was probably just a normal activity.
Well, the vice versa can be true. Also. You can see something that’s just so well disguised that you think it’s a normal activity and it truly ends up being something that’s malicious. I feel for the job.
It’s definitely. You need to keep an open mind and you need to be able to go, okay, yep, I did make a mistake. And, you just need to learn from that mistake and move on. Don’t, don’t hinder on it. Don’t harp on it.
Yes, you made a mistake.
Hayden Covington
Yep. Everybody makes them.
Will Corbin
Yeah. I don’t want to say it’s no big deal because obviously, we miss something, but it’s not a big enough deal to beat yourself up about. and that’s not productive. Right. You need to learn from it.
You need to take whatever criticism’s coming at you and just add it into the knowledge bank and go. In the future, I need to remember this 100%, man.
Hayden Covington
The best analysts that we have are ones that take feedback and they write it down, and then they don’t make those mistakes again. Or if they do make them, it’s a lot less frequent. They take that feedback and they work with it and build on it instead of taking it hard.
It’s not personal.
Will Corbin
I can 100% say my tickets are now completely different than they used to be. If you look at my investigations because of that.
Hayden Covington
Absolutely. All right, Paul, how about you, man?
Paul Lammers
so I had one ticket, a couple moons ago, and I came to the complete wrong conclusion. Like I said, nope, this is good. This is fine. It’s something weird going on with our sensor.
And actually, Tom came back and said, you might want to look at this again. It’s one reason why QA is very important. And, it ended up being a red team, so it is my worst fear of a false negative.
Hayden Covington
Yeah, it’s the whole team dynamic, man, is we all. With soc, there’s so much volume. You can’t spend all day on one alert. You got to get through several a day just to stay on top of things. And so having analysts that are doing qa, and we have a lot of automated QA things that will go and pull stuff for review automatically based on a number of different variables.
Hayden Covington
But being able to have analysts that you trust and that you understand and you, you work with, well, go and review your work. And being able to, like, with Paul. Paul could have just said, all right, Tom, why are you stepping on my turf, man?
Hayden Covington
Like, get out of here? and that would have not have been productive for anybody, but being able to say, okay, and then working with them. And from Tom’s point, I’m sure Tom offered help and advice on how to investigate that further.
And that team collaboration is how you actually succeed and catch, the bad guys, or in this case, the bad guys that are on our side after the bad time ends. yeah, so we’ll take.
Tom DeJong
We hopped on a call together, and I thought that was something that, like, help increase our friendship, too. So I feel like calling and talking to your other analysts and teammates. I feel like it’s just always good.
Hayden Covington
All around, so because it’s easy to get your head buried in the sand and do alerts for 10 hours a day and then, try and recover over the weekend. That’s easy. So, that’s something that every team needs to constantly pay attention to and work on.
let’s hit on one more of these that I feel is pretty important and then we’ll, we’ll open it up to some audience questions. after one other, one other question here. So this one is very important to me is work, life balance.
it is a constant struggle for me personally, as I’m sure many other people. Sock, work is hard. Sometimes the hours are long and sometimes the on call shifts suck. There have been on calls where I slept fairly little or, I remember one, I called our elastic Engineer at like 3 in the morning and I said, please, please turn this thing off.
I don’t have permissions to turn off this sensor that’s alerting. Please turn it off. I know it’s not bad. Here’s my investigations. Please just turn it off so I can sleep. there will always be times like that and there’s a story of, a SOC, incident that Tom and I both worked on that lasted like we were up 24 hours straight or something working on that.
And that is going to be a story that I will get to tell people, for a long time into the future. those, those rough moments you can treat as, reasons you don’t want to work in a SOC, or you can let them get you down, or you can spin them the other way and use them as learning experiences.
They make really, really good stories most of the time. but even the times that they’re not being able to at the end of the day have, you can look it up. It’s called like a shutdown ritual. Something you do to trigger to your mind that your day is over with work.
Being able to have something like that to where you turn off, you recover and then you are able to spend time on your hobbies, your passions, that is super important to being able to do this long term. because if you don’t, you’ll, you will suffer.
That is just a fact. so to turn this around to you guys, Tom, how do you, how do you keep on top of this? How do you keep on top of, not burying yourself in work, all hours of the day and all days of the week?
Tom DeJong
Well, usually I try and go on a walk during lunch, just make sure I get outside for the day. So, so having dogs is very, very helpful in that regard. but even if there’s no dogs, wherever I’m at, I’ll just try and go walk around the neighborhood for my, for my lunch break at least 20 minutes.
and other than that, I try and have my office in a room that I’m not going to be in, like a bedroom. And, that’s not going to be possible for everybody. But, I think having the separation of your workspace and your relaxation, space is very important.
but, I think, I think part of it’s just keeping up with social, social people or like your friends and stuff. Make sure you’re getting out and having fun with people. sometimes after work I’m just so tired, but I try and at least reserve one day of the week that I just go and hang out with people on the weekend or something.
So, I think, I think that you can work these things in here, but I, I really think getting outside is a powerful thing. and socializing with people. So those are, those are the two main reason ways that I, I stay sane, I suppose, get.
Hayden Covington
Getting out, socializing, exercising, having a different space that you’re not, in all the time. Like, if you have a. Your gaming computer next to your work computer, that becomes harder. and that can cause some additional stress.
somebody in the Discord asked, do you get lunch breaks? Yeah, sometimes. sometimes we don’t, but other times, Tom will be like, hey, I’m going to go eat lunch. And so people know on the team, Tom’s eating lunch. I’ll keep an eye on things to make sure that, everything’s going okay.
And, I’m sure if we messaged Tom and said, oh, Tom, we need you, buddy, he would, hustle up his lunch or whatever to come help. But it’s all about communication with your team to, to make sure that you are not overworking yourself either.
Because you taking a lunch break might not be a lot of time, but, and it’s not going to be helpful for your team. Then, how about you, Will? How are you? How do you stay on top of work? Life balance, it’s, struggle for a lot of people.
Will Corbin
Yeah. Sorry if this sounds a little crappier. Switch mics. But it shouldn’t.
Hayden Covington
Believe it or not, I was going.
Will Corbin
To say, at least it shouldn’t crackle now. so pretty Much. My thing is, once I hit the end of the day, I try to make it the same time every day, but, life happens. You can’t always get out of work at the exact moment you want to.
But once I’m done work. This is one of the reasons I really love that I have a separate laptop for work. Specifically, that laptop gets closed, turned off, and I taken off of my desk completely.
I don’t have enough space to have the separate home computer, work computer thing, but it gets hidden out of my sight and my personal computer gets brought out. And I pretty much from that point on, my personal computer does not have any of the credentials for my work stuff on it.
not like if I wanted to do work at that point, I would have to physically go get my other computer again and log back in and everything.
Hayden Covington
And yeah, those physical dividers, man, psychologically are huge.
Will Corbin
Right. And then, yeah, I just, I focus on family and video game time at that point.
Hayden Covington
Nice. How about you, Paul? How do you. How do you balance it?
Paul Lammers
I mean, coming from the army, I have literally had 120 hour day weeks.
Hayden Covington
Yeah.
Paul Lammers
Like, so more than 16 hours a day for seven days. honestly, I have, my blood pressure has gone down by like 20 points since working at Black Hills.
So, it is a lot easier for me. I still have a lot of work anxiety of, like, I still haven’t done this. I haven’t made this new rule that’s supposed to be out there and I have to pull myself away. Like, I get anxious because something’s not done.
and it’s just taken time for me to, back away from my desk at lunch or.
Hayden Covington
we’d be more to do. There will always be more to do.
Will Corbin
I do message him randomly.
Hayden Covington
Yeah. The times that you make those connections about how to fix things, at least for me, are usually times where I’m not working or I have, stepped away to go, eat lunch. And I’ll go, oh, my gosh, I realized what I was doing wrong.
Will Corbin
I will message Paul at random when I see he’s over there stressing out and be like, hey, Paul, you’re not in the army anymore. There’s nobody breathing down your neck. It’s okay.
Hayden Covington
Okay, so let’s, let’s hit this slide and we’ll take some, audience questions. Before we get into the audience questions. This is sort of a precursor and a little bit of a talk about a workshop I’m doing on Friday.
If you’re Interested Detection engineering crash course. And then I. I also have a SOC course which teaches you how to be a SOC analyst. You build your own SoC and you start from what is a SOC, and by the end, you’re investigating threat detections that you’ve wrote and that you are, test firing and evaluating it in your sim and, all that cool stuff.
Hayden Covington
so check those things out. Anything you guys wanted to add, Tom, Paul will touch that. I don’t know if you can see the slides.
Tom DeJong
I’ll add that, have come through and taken Hayden’s course. they’ve taught me things, particularly Dashboards and Elastic. But I. I think it seems great. I haven’t got to fully take it, but once I get some time, I’m definitely planning on going in there because, I think it was actually Paul that when he tested Hayden’s course, he came back with all these dashboards, and I’m like, how’d you do that, man?
No, I have no clue. So I. And, the parts I did listen into, I thought were great. and I’ve just heard great things about it.
Hayden Covington
I. I appreciate it. Yeah. That I. I spent too long photoshopping that teleprompter in there, so I had to use it. What’s up, Daniel? Welcome back.
Daniel Lowrie
Hey, Hayden. It’s, Man, this was a lively chat. Very informative and interesting. I mean, I was. I normally would go take lunch, but I actually sat here and was kind of like, this is a. They got something going on here.
That’s. This. This was a good. This is good information. So I. I really enjoyed you guys, your banter and everything, all the information that came out of all the heads on the screen today. And I think I feel like chat feels the same way, because Discord has been crazy.
And you’ve got 19 questions in the zoom Q and A. Obviously, we will. We will get through every single one of them. No, that’s probably a bit of a. Of a tough haul far. But there are some really interesting questions out there.
I would like to start with one that I saw that I think a lot of people are very interested, and that was.
Hayden Covington
Bring them up.
Daniel Lowrie
AI.
Hayden Covington
okay, let me put this slide up, too, which has the Discord link. And, my Twitter just hit me up with questions if you have them. I’m down.
Daniel Lowrie
Sure. Let me. Let me.
Hayden Covington
Whenever.
Daniel Lowrie
Let me grab that bad boy. Oh, it’s. Now it’s kind of scrolling out of my way. Where did you go?
Hayden Covington
It’s up there? Yeah. I mean, I’ll also post a screenshot in there. in the discord. Ask questions, whatever you want. I’m down for any of them. I don’t know how I put confetti on the screen just then.
If anybody saw that.
Daniel Lowrie
You are super good at this. That’s all. You’re just that good.
Hayden Covington
I’m sorry to cut you off. AI questions.
Daniel Lowrie
Yes, the AI question is a SOC manager. So it’s a SOC manager asking. I’m interested in knowing how you’re also taking advantage of AI and automation in your various SOCs. And I have. I’ve been hearing this a lot.
Daniel Lowrie
A lot of people asking this question. How can we integrate AI? Is it secure? Can we do that securely? Will it be seamless? There’s, like, a lot of questions around utilizing AI in the new security space.
What are you guys doing about that and how’s that working?
Hayden Covington
That is a webcast and workshop all on its own. yeah. How we’re doing it here is we have very strict rules about how analysts are allowed to use it. And then we have very strict informal rules. Like, one of those informal rules is don’t ask it a question that you don’t know how to fact check.
Because if you don’t know the answer, you need to ask somebody for help. Because we don’t want you relying on AI to be your, your authoritative answer on a subject. You can use it to research, things that are factual, and it most of the time now gets them right enough that you can tell and you can see, whether or not it’s correct.
but even as you go and as you progress in your, your soc maturity, automating a lot of the triage is going to become a fact of life. and you have to do it correctly because customers are not going to trust you to just dump all their alerts that you’re getting for them into AI and have AI do all the work.
it’s. It has to be. There has to be human touch. There has to be QA on those AI alerts. There’s. There’s a lot of facets to it. I guess I’ll come up with a webcast for this or something. And bring on, Derek Banks at Black Hills is working on a lot of AI stuff on the background that we’re doing, and I think maybe he and I could also have a lively chat about AI in soccer.
it’s a hard question, and some AI are, you look at Deep Seq, you might want to be careful what you put in there. if you look at local models, you could probably put a whole bunch of stuff in there and never have any negatives besides maybe some, high power bills and maybe some inaccuracies.
But yeah, that’s only like a half answer.
Daniel Lowrie
But that’s a pretty good answer. Like honestly, that’s probably the best answer I’ve heard come out of anybody thus far. Just because so many people have such limited experience with doing that very thing and what that might look like.
So it’s nice to hear from someone that has actually, messed around with these things.
Will Corbin
I’d like to add too, we use automation a lot in our job as well. So anybody out there? Python Rust, Python, good languages to learn.
you can automate so many simple things. You can automate, test for your rules to make sure they’re running correctly. just day to day tasks that seem to build up. You can automate very easily using the tools at your disposal most times just by learning Python.
And it is a pretty easy language. Yep.
Hayden Covington
Yeah, you just got to fact check it. You got to make sure that it’s being correct because that’s not at the point where it can go do its own thing.
Will Corbin
Yep. And if you are going to mess with automation, I always recommend that you set up a test environment to do it in first. So if you have, if you, if you have the resources to do it, basically clone your SIEM environment and have a separate space that you can crash and destroy and light on fire all you want without making your clients mad.
Hayden Covington
Yeah. The best application I’ve seen of like AI and Automation in a SOC is the last job I worked at. We had a soar that we had built out, automations for. So when we had a SOC alert come in, the soar would go and search in splunk based on that alert type and it would pull a bunch of logs back.
So a lot of alerts we would get to as the soc, we would show up to this alert and our SOC alert would already have a ton of logs in it and we could potentially make a decision immediately about whether to escalate or close that ticket. that is one of the best applications I’ve seen of automation.
and that was mostly without AI. That was just basically a lot of Python code. so as you work in AI, Elastic has a hook in for OpenAI API keys. You could have an AI automation built to say, yeah, for these alert types, here’s Some queries that I want you to go run, go collect logs that seem pertinent to this alert.
And so that’s a way where an analyst is still involved. They are just getting a lot of the heavy lifting done by the AI. And you’re not just automating the personable touch. That makes us suck, Useful.
Paul Lammers
Interesting stuff. Another thing with AI that I want to throw out there is never put anything into a public, AI. Algorithm that you don’t want to be Googleable because, you never know what’s going to slip out.
Hayden Covington
One of the strict rules that we have is if you don’t know what this data is, you better not be putting it into AI if you don’t know for a fact there’s a command string. Do what that command string is? No. Then you’re not putting in an AI man.
Don’t do that.
Daniel Lowrie
Hayden, somebody asked about your courses, and I actually saw it a couple of times. Which one should they take first, is the question.
Hayden Covington
So they’re both designed to be taken by themselves. Realistically, you could take either one. You could have no SIM experience and go take the workshop on Friday, because I have a couple slides that build up elastic knowledge.
So that one is primarily detection engineering, and how it feeds into the course is. A lot of that content is pulled from the course for the workshop, and the course just expands on it and gives you a ton of extra detail. So where we would spend, in the workshop, we’re going to be making KQL detections, which is like a language, one of the languages within Elastic that is more simple, still powerful detections.
but it’s more simple in the course. We deal with sequence detections, and we talk about elastics, more powerful languages that they have. So the course is a really good. Or the workshop has a really good crash course. And so from that point on, I would definitely recommend that if you want to learn how to be a SOC analyst, take the foundations of SOC course that I have.
Honestly, that’s my. My goal with that is it was internal training from Black Hills is why I was making that course. So when we had a new SOC analyst, I could hand them a course and say, all right, awesome, welcome to the soc.
These are your next two days. and then from that point on, it would be a lot easier for us to train these people on the SoC, because they would understand all the fundamentals. And then I was making that course, and somebody said, why aren’t.
Why isn’t this on Anti siphon, because other people also would find use in that. so yeah, that’s how I would recommend it is. You could do either one, the workshops Friday. So I would say do the workshop first just because that’s going to happen a lot more, a lot sooner, and you’ll get more practical use out of it.
But to build on more of that, man, the SOC course is, you build a SOC, you could transfer the billing over from Elastic Trial to your own credit card and you would have a fully functional SOC at that point.
Daniel Lowrie
That’s awesome. That is excellent, and great information. Good to know because that could be a little bit confusing. You got a couple of courses out there. Which one is it built on top. So now that we, we have that information, somebody asked and I think this is a really interesting question because someone who, like as yourself, we do training and people are always looking for a good resource.
And this name pops up a lot when it comes to this type of conversation. and content waza. What do you think about it? Have you used it? Have anybody here played around with this thing? And what’s your take on the thing?
Hayden Covington
Yeah, we actually ingest WAZA logs. I am partial to Zabbix on my home lab for management and monitoring, but I do know that the Black Hill Systems team either has or still does use WSA for some monitoring and logging.
And yeah, we, we, like I said, we do receive wizard logs from some of our customers. I think the hardest thing about being a SOC analyst, especially for an msp, is there is a billion solutions out there for everything.
and so there will always be a new waza. there will always be a new, Wiz, there will always be a new SIM that’s been spun up. It’s like AI models, man. There’s a new one every week and they’re all basically the same thing. So being able to understand those technologies at a base level and understand the functionality and how to navigate ui, the UI navigation is just, that might be the hardest part.
But knowing how they function at a base level is a lot more important because from there it’ll be an easy switch to go from a crowdstrike EDR platform, looking at that dashboard, switching to Sentinel 1, because the point of it, generally how you should use it.
And so it’s just a matter of translating that into how they look slightly different and are colored differently.
Daniel Lowrie
That’s a great answer. I’m glad that you, were able to give that because I hear that question all the time. I don’t really mess with it too much. So I’m like, go play with it, turn it on, spin it up, see what it does. Poke it with a stick.
Hayden Covington
home labs, they’re so good. Take an old desktop. One of my home lab servers that I use for a lot of stuff is an old Dell optiplex with like 8 gig of RAM and a 250 gig hard drive.
That thing is a piece of garbage. but it’s running a lot of really good things. I’m, running Zabbix and a, Discord bot for a server that I’m using with friends to, convert our Instagram reels that we share into embeddable videos.
Like, you. You can take an old computer and make it do some cool stuff. yeah, you don’t have to be. You don’t have to be loaded anymore to have a home lab. You don’t need a server rack. Mine are sitting on the floor. I have a monitor balanced on top of one of them.
Like you. You can do some cool stuff. And a lot of these tools are open source, at least at a very base level of their functionality, and that’s all you need to learn them.
Daniel Lowrie
That is awesome. And. And apparently we have sparked a debate on how to pronounce.
Hayden Covington
Oh, have we? I pronounce.
Daniel Lowrie
It was, I’ve always heard of waza.
Hayden Covington
Yeah, yeah, yeah. I was like, class. Yeah, yeah.
Daniel Lowrie
Anyway, that’s fun. We’re gonna start a religious war here in about two seconds. So that said, thank you, everyone at Discord, for joining us today, but it is time for us to kind of go into that backstage Zoom area of the Anteciphon AMA where we can continue the questions and the conversation.
So if you want to be a part of that, you got to go get that Zoom link and get in there. I don’t know if you can still register for it or if not, maybe you have it. You’re just in, the Discord right now, but we’re going to jump over to, Zoom and continue that conversation.
Thank you so much, Discord. We love you guys. The memes and the gifts are so worth having you here because you bring so much joy to our lives. But we will see you next time, and we’ll bounce over there and we’ll see everybody else in the other room.
Stay up to date on our upcoming live Anti-Casts and more at https://poweredbybhis.com
Don’t forget to check out our Course Catalog for our upcoming free and affordable cybersecurity training!