This Anti-Cast was originally published on November 6, 2024.
In this video, Derek Banks and Joff Thyer discuss the implications and applications of artificial intelligence in information security. They delve into the challenges and opportunities AI presents in the field, highlighting its potential for task acceleration and cost reduction. Additionally, they explore the historical development of AI technologies, their current capabilities, and the future trajectory of AI as it becomes increasingly integrated into security practices.
- Artificial intelligence (AI) is crucial for information security as it addresses the industry’s training and skills deficit, offering task acceleration and potential cost reduction.
- The evolution of AI includes the development of large language models, which are now capable of performing tasks comparable to a high schooler, but future advancements could lead to AI with PhD-level reasoning.
- The rapid growth in AI and large language models poses significant security risks, including data leakage and adversarial attacks, emphasizing the need for robust AI policies and security measures.
Highlights
Full Video
Transcript
Joff Thyer
All right, we’re going to start out here with part of a presentation actually that Derek and I did at Wild West Hackin’ Fest in Deadwood this year. And we thought it would be a great intro to this discussion today.
After we do this sequence of slides, about the technology itself, we’re then going to allow Derek to take the helm and do some demonstration work for us, so that you can see some real life examples of how you might apply some artificial intelligence in your day to day life.
Right? So first of all, artificial intelligence, why is it important to information security? this is a little bit of stating the obvious, right?
We have enormous demand for our skills, in this industry, right? We have a training deficit for sure, we have a skills deficit. We actually have an entire human resource deficit in information security.
The demands keep coming, things keep scaling up. There’s going to be always a need for our skills in this industry, but there’s going to be a need for us to be able to scale ourselves up in such a way that we can be more effective at our jobs.
So the point of this need to accelerate task completion, is probably the biggest take home point that artificial intelligence offers to us, in the information security realm.
And then of course for your more executives in the house, potential for some cost reduction by applying artificial intelligence technology.
We’re still in the hype cycle, right? Everybody out there with various products would like you to think that they have cracked the nut, solved the problem, that all of their AI and ML machine learning in their products is the most wonderful thing since sliced bread and they are your ultimate solution to everything.
No, we’re in the hype cycle. We’re totally in the hype cycle. And it was completely emphasized and blown up to an intense scale when we saw these generative AI models emerge in the industry with ChatGPT, and all of its other friends, right?
Frankly, there are multiple opportunities for us to apply data science, artificial intelligence, machine learning to information security problems.
Let’s move away a little bit from the hype cycle and focus on some real problems. Time series analysis, is a classic all the time what we’re doing in our industry.
I don’t care if you’re on the offensive side, defensive side, forensic side, whatever it is, you are looking at logged data somewhere, sometime, and you are looking at timestamps.
And so there is opportunity for AI technologies to accelerate your tasks, maybe even to build task specific models to help you in those kinds of tasks.
Text classification, no doubt that’s a natural language problem. Very, very helpful, to apply that in the information security paradigm. Anomaly detection, something we talk about with task acceleration is an analyst, sixth sense, right?
This idea that, you’re maybe a defensive engineer or a forensicator and you need some way of validating something is true.
Like you’re, like, I think this is right. But let me have my friend here, this large language model, this generative AI, give me a little context so that I can interpret that and use it as my aid, as my friend, as my buddy, if you like.
Right? And then of course we have the offensive side against AI technology itself. We can attack AI technology, especially generative models.
We can attack the fact that they represent a data input parameter, boundary that we can use against, the technology to make it do interesting things.
But wait, why don’t we get some meat on the bones first? Why don’t we define a little bit about what artificial intelligence actually is. The actual term was coined a long time ago in 1955 by John McCarthy.
It’s the science and engineering of, making intelligent machines. Concepts such as autonomous systems that can independently plan and execute a sequence of steps in pursuit of a specific goal.
Right. Machine learning is a part of artificial intelligence. Studying how our computers improve their perception, knowledge and thinking by leveraging data.
Deep learning. Another concept, the use of large multilayer neural networks organized in a human brain like configuration, something you’ll hear me say a lot when I’m teaching and talking about artificial intelligence, is that all of these layers that we are going to talk about still ultimately add up to this concept of simulated reasoning.
A large language model, a generative artificial intelligence system, is simulating thinking from a large data source and spitting out tokens to you as it simulates that thinking, responding to your prompt and your context.
Artificial intelligence is not brand new. I hate to tell you. I know you might think that artificial intelligence sudden popped up about three years ago.
It’s not true. It actually grew out of neural network research starting back in the 1950s. There was a real life problem that had to be solved back in those days.
And that real life problem, believe it or not, was echo cancellation on telephones. It was a telephony issue. And Bernard Widrow and Marcian Hoff at Stanford at the time developed some AI models, to do this.
Developed some essentially at that time were neural networks, called Adeline and Madeline, which solved this problem of real world phone echo.
In fact, the neural network model’s job was to actually predict bit values. Can you actually compute and predict what that next bit value is going to be?
And if it’s not what it is supposed to be, then maybe that’s an echo and I need to cancel it. Right. So this is multiple adaptive linear element was the name of the model.
Now unfortunately, or maybe fortunately, I don’t know, it depends how you look at it. There’s always opportunity everywhere. As we grew through the 1960s and 70s, the early phases of compute development, the von Neumann computer architecture actually was the dominant paradigm.
And the von Neumann computer architecture is characterized by this serial architecture that is a computer processing unit that goes out and fetches an instruction from memory.
It executes that instruction and then it puts the result back into memory and it does that in series. And unfortunately when artificial intelligence and neural network research was going on in parallel with the dominant von Neumann architecture that was emerging in early computers, there was a realization very quickly that neural networks didn’t really operate well in a serial fashion.
In fact the research was saying that neural networks need to perform multiple operations in parallel. And the von Neumann architecture just really didn’t suit that.
And so the research around neural networks kind of stagnated because of the emergence of the compute paradigm, the dominant compute paradigm at the time.
However in the 80s, some of this research got reinvigorated by a gentleman named John Hartfield at Caltech. And I find this interesting to compare and contrast against Moore’s Law, right.
We all know the intel, gentleman, who came up with this law of compute transistor density, accelerating over time, in, in a in a sort of exponential kind of fashion.
This scale is deliberately not linear. and it led me to think how does the von Neumann architecture, which is a serial based architecture, and then the emergence of GPU parallel compute architecture, how does this impact the paradigm that Moore’s Law, mandates?
Right. Well, it doesn’t really mandate or speaks to I’ll say that. And what it really is that TPUs, tensor processing units and graphical processing units, GPUs, they represent a simplification of the von Neumann architecture with performance improvements that don’t necessarily rely on smaller transistors.
GPUs and TPU is a, computational specialization. They’re specialization on vector processing. They are specialization from a speed and algorithmic efficiency perspective.
My opinion is tensor and GPU processing don’t really break Moore’s Law. They really extend the paradigm of increasing Computational power.
these are the things I think about when I’m sitting on my deck in the evening with a, with a nice cocktail in my hand. Derek’s laughing at me. thanks Derek.
Love you too, man.
Derek Banks
I’m laughing because it’s true.
Joff Thyer
It is true. It is absolutely true. I think part of me, and this is why I teach it at anti siphon training, I just want to take a segue right now and say I love to be an instructor.
It is something I think I was ultimately born to do at this time in my career and in my life. And so there’s a little bit of a computer science professor in me, which is why I’m going through some of this sequence.
Right. Let’s look at the historical AI tech timeline. In 1966, the year of my birth, Eliza was developed at MIT by Joseph Weisenbaum, 72.
We had the statistically trained natural language processor. So very early days, we’re already seeing natural language processing emerge. The things in red that I’ve highlighted here are monumental developments that have changed the paradigm from the perspective of the large language model evolution.
So in particular, there was a seminal paper in 1997 that spoke to the long term, short term model that was developed by Hochwriter and Schmidhuber.
It’s called lstm. And LSTM is the core neural network layer that makes large language models actually be able to remember context.
It actually allows them to be in a way a little less stochastic. They can sort of go back and forward in time instead of being in the moment.
And it’s a very, very interesting paradigm and algorithmic development. Coincidentally or maybe not so coincidentally, in 1999 Nvidia first introduced their GPU.
These were the prerequisites, these were the seminal events that led us to the idea of neural network machine learning, natural language models starting to be released and, and develop more rapidly.
So in the period 2000 to about 2016, we have a lot of experimentation going on with multiple new AI models being released and IBM and Facebook and Meta and Google, all involved.
Right. OpenAI was funded in 2015. That’s when that startup, was spun off.
Right. 2017 we have the introduction of the transformer model at Google. And the transformer model set up this idea of being able to use neural networks and tie it in with this concept of transforming and generating from the neural network processing.
I’m trying to be not, I don’t want to go too deep because then we get into class but all right, 2018, Google made TPUs actually available. And then OpenAI publishes a paper on GPT.
Now we have what is the beginning of the root of the tree on, natural language processing becoming large language model and generative at this particular junction, in time.
In 2019, OpenAI releases GPT2 and suddenly we have kind of an arms race of technology on our hands. Now coincidentally, right there at 2019, something awful happened on planet Earth.
And that would be the COVID pandemic. And so maybe this was meant to happen because between that time and today, with a lot of scientists ensconced in, well, their home offices, there was this massive research period.
And then all of a sudden an explosive growth of AI large language model deployments. Okay, by the way, some of these images, are generated from Dall E3, which is a visual AI model that you can prompt it.
I used a simple Python script and say, draw me a picture of explosive growth. And this is, this is one of the pictures that came out, which was a tree with, I think it’s fruit and then some numbers exploding out of it.
kind of cool. anyway, another picture, from the LLM Practical Guide, which is out there on a GitHub, shows kind of the root of this tree in 2018 and then this growth of the different branches.
And it also shows for you the incredible popularity of the decoder style transformer model as we otherwise know it, the generative large language model that really started to push out, in this explosive growth.
So we’re in an interesting point in history, right? We have these very, very large compute clusters. We finally had the compute power available to us, we finally had the data storage capacity available to us, to build these very, very large neural networks, and connect them with this transformer model paranoia.
all right, so pretty quickly thereafter, right, we came, various people said, well, how do we, how do we know that these models, which are simulated reasoning and generative, how do we know what they can do?
And so they started testing them, right? And this, if you set human performance as a baseline at the zero mark here, which is about three quarters up the slide on the screen, and then you use different domains of knowledge against the various AI generative models that we have.
Somewhere right around 2018 to 2020 we suddenly discovered that these simulated reasoning generative models actually we’re meeting human baseline for reading comprehension, image recognition, language understanding, handwriting recognition, speech recognition, predictive reasoning.
That’s interesting. Do you notice that predictive reasoning is a little bit lower. And later in time on this particular graph and we talk about that a little bit and we start moving into more and more complex domain knowledge, right?
Code, generation more complex reasoning. Now look at complex reasoning 2023-2024, about where we are now.
Complex reasoning is still subpar. It is below the human baseline. The question you got to ask yourself is, for how long?
Another way to cut this apple was we started looking at standardized testing. And standardized testing in the United States is, a long and glorious tradition. I’m an immigrant, by the way.
I grew up in Australia, and I entered this world of standardized testing much later in life. But I’m aware of many of these things. So GPT4, which released, was released in 2023, we tested, against these various standardized tests and you give it the Unified Bar Exam and it came in the 90th percentile.
LSAT 88th, SAT, 97th. I mean, GPT4 did better on the SAT than I did. and I guess I’ve outed myself now.
gre, US Biology, AP calculus, et cetera, et cetera. Right, Joff?
Derek Banks
it did better than most people.
Joff Thyer
Most of us, right? As compared to GPT 3.5 back in 2022, which was not that long ago, by the way. The older you get and you start, you start looking at 10 years ago and going, oh, I think I have T shirts older than that.
so not that long ago we were seeing numbers in the 10th percentile, and suddenly GPT4 is in the 80th and 90th percentile.
So what that means is somewhere in about mid 2023, we developed for ourselves a simulated reasoning model that was as capable as an intelligent high schooler.
What does that actually mean? Socially, ethically, morally? I mean, we got some interesting things going on here and some interesting things to think about.
Okay? Now the other thing we learned very quickly was to train these models takes a, ton of compute resources. This is a graph of training flops from situational awareness AI, which speaks to the evolution of time, of, the amount of floating point operations per second.
We need to actually train these models. And boy, when you get up to Gemini and GPT4 in 2024, you’re in 10 to the 22nd, 23rd, 24th power of floating point operations per second to train these models, okay, that means an awful lot of electricity, which I’m going to mention in a minute.
But these things are not small, okay? In fact, where where are we? Right? Current frontier models like Llama 3 are trained on the Internet.
I love that. I like, I like this sentence. And the Internet is mostly crap, right? So we’ve trained a very smart high schooler on a diet of well, McDonald’s for lack of a better word.
Right, sorry if you’re a real big McDonald’s lover. I’m not. Mostly junk data, right? So if the vast majority of their training compute is spent on mostly junk, what would happen if you could spend the same compute cycles training a model on extremely high quality data?
I think what would happen, I would just make a little prediction here that we get much better than a smart high schooler, that we start getting into the space of a pretty capable simulated reasoning entity.
So and again a lot of this is from the situational awareness papers published by Leopold. and you can go look some of this stuff up. Leopold in his papers proposes an idea for where we’re going in this whole journey that we’re on.
So the concepts here are as we accelerate into the future orders of magnitude in pursuit of acceleration are pretty much our best guesses as to upcoming changes between, let’s say we’ll look at this between say now more or less all the way up to about 2027 from a compute perspective.
We’re not going to see that slow down. We all know Moore’s law is there. We all know that Tesla, Google, Nvidia, are working very hard on processing power research and scaling.
So we’re going to see predictably a two to three order of magnitude improvement in processing power. we’ll also see continuous improvement in algorithmic design probably with a one to two order of magnitude improvement.
And then where it gets really interesting is when we start unhobbling, the training of these models.
And by that I mean when we start dropping the guardrails and using a pre trained intelligent high schooler model to start training the next model, we used reinforcement learning techniques.
We start feeding the output of LLMs, that we know is high quality back into the input side for training. We start accelerating up and unhobbling, right?
Derek Banks
And by hard to quantify I think we mean anywhere from like 1 to.
Joff Thyer
Infinity, 1 to infinity order of magnitude. So if we look at the projection, and I think this is more or less coming true as this was written last year, we’re starting to see artificial intelligence becoming a knowledge worker partner to what we’re doing.
We’re moving to instruct model to chatbot to agent level. we’re really getting to a point where things are becoming incredibly useful.
Okay, so if Today we have LLMs that are smart high schools, they, they’re able to ace the act, solve college level math, accelerate tests, et cetera, et cetera.
If we add together these predicted order of magnitude improvements that Leopold proposes, then I think our predicted intelligence may well transition over time.
And I don’t disagree with this statement where we go from what is currently a pretty smart high schooler to a simulated reasoning level from a generative AI model that becomes something akin to a knowledge driven PhD level researcher.
Huh, huh. And it gets worse or better. I don’t know. Turns out you think about it because once you have that first even smart high schooler or that first knowledge driven researcher, you can start hooking these things together and you can automate the AI research.
So if we automate and we scale up the AI research using existing artificial intelligence, we’re starting to take the guardrails off and we’re starting to scale very, very rapidly.
And what Leopold Aschenbrenner predicts is, and again, this is crystal ball stuff. So just take that, bear that in mind that we may well be in a position of having 100 million automated researchers working at 100 times human speed not too long after we automate that AI research.
And so the prediction is that it’s, it’s, it’s plausible we go from what’s called artificial general intelligence to something of the order of superintelligence.
And if we do that, what use of the humans anymore?
So here’s another graph from that particular set of papers and I do strongly encourage you to go read them. I think they’re very interesting. So we’re sitting here today and based on the predictions of these papers and everything that we’re seeing transpiring, we’re actually looking at the beginning of an exponential acceleration curve in this technology.
And we are seeing all of the technology companies, all the technology companies positioning themselves to begin accelerating up that curve as fast as they can.
All right, there’s another researcher named Roger Gross, who’s a computer scientist at the University of Toronto.
he’s a founding member of the Vector Institute at the Advanced Canadian AI Research.
sorry, let me say that again. He’s a founding member of the Vector Institute to advanced Canadian artificial intelligence research. Roger sees the artificial general intelligence trajectory over the next decade that looks a little bit like this.
And his biggest concerns are those that have already been expressed by OpenAI and others in the community are Essentially where are the alignment risks and challenges?
And a lot of the world in the past year or two has been focusing the alignment, risk and challenges kind of to the left of this curve, reinforcement, learning, selecting outputs that humans approve of, using chains of technology.
that’s kind of where we are right now with these base LLMs as we train, but we’re starting to move up the curve. So long range coherence makes the LLMs useful for generating proposals for brute force searches, right?
Policy gradient more effective, et cetera, et cetera. But as we go over the top, and we’ll go over sort of a peak where we suddenly have really big alignment risks.
And in fact there was a, governance and alignment entity that OpenAI had running for some time. And I think a lot of those folks just quit.
They didn’t know how to solve the problem. And the sentence here to the right of the model is very prescient.
As the model becomes more capable of solving problems through search and reasoning, why should it imitate humans anymore?
I, mean, are we even useful at that point? Did you watch the Matrix? Because we are kind of in this position of being somewhat in the Matrix in this world.
If you ask data scientists that are training these really large clusters, they cannot give you a precise answer of why the LSTM neural network layers in their models are actually operating in the way that they are.
And so they’re concerned that the word simulated in simulated reasoning, Tom is starting to break down. And maybe it’s not so simulated anymore.
Now again, a little bit of hyperbole, a little bit of excitement, but it is a concern. Now let’s talk about electricity. And this is where, I draw a little bit of comfort for this slide, but not the next one.
The United States is in a big problem right now, and that is we are fast approaching the threshold whereby we cannot generate enough electricity to run these very large compute clusters that are training these models that we’re all aware of, in fact we will very quickly approach the intersection of maximum electricity capacity to train these models.
So as you’re looking at your media, you are seeing suddenly a great deal of interest by very large technology companies in locating themselves near abundant sources of electricity or even further acquiring entire power generation ability so that they can solve this problem.
Derek mentioned earlier as we were starting up, that Microsoft is currently in the process of considering starting Three Mile island nuclear reactor back up.
That scares the living daylights out of me because we’re talking about 1970s technology. So good Good luck modernizing that before you start it. But anyway, now it turns out just if you’re not worried enough, China doesn’t have this problem.
China has happily been burning as much coal and producing as much renewable. In fact, China is just producing as much electricity as they possibly can produce, through all means, whether it’s coal, gas, renewables, wind, you name it.
And to make things more interesting in the artificial intelligence community with large language models and other neural network models, we are openly publishing these models on public facing websites.
So the geopolitical risks as we continue down this path are going to amplify immensely.
In fact, you will see, I’ll just throw a prediction out there in the next 12 months you will see artificial intelligence as a military concern become front and center in the United States and elsewhere.
Derek Banks
I had a little bit of a Internet lag, but that actually might be happening now. I just like read an article, article here recently where I guess it was Llama two, the meta, model, that was released.
I was 2022 or 2023.
Joff Thyer
Yeah.
Derek Banks
M. So yeah, an outdated model, by I guess current standards. But I guess the Chinese were able to take that and essentially make a military AI out of it.
At least that’s what researchers are claiming. And so I mean, I think the, it’s the cats out of the proverbial bag kind of speak kind of thing. The Pandora’s box has been open.
Whatever cliche works there. Right.
Joff Thyer
So I do that. That is actually the introduction sequence to our class in fact. And we’re setting a context here. Right.
To do a little bit of scare the crap out of you, but also to motivate you. You need to understand this technology not just because you’re a technological practitioner in information security.
Artificial intelligence is going to dominate our lives from, from a multiple paradigm perspective for a long time into the future.
Derek Banks
Yeah. I think even if it’s a technology where the bubble bursts to a degree. Right. Where let’s say that those orders of magnitude hit some kind of limitation and we get basically Everybody gets a PhD, level researcher as your assistant.
Which would be kind of cool. I would like to have that kind of thing like an analyst, sixth sense, kind of thing. I think that it’s kind of like as new technologies have come out in the past, when virtualization came out, security practitioners had to learn, be conversant in VMs and virtual machines.
Same thing with Docker and containerization or cloud technologies. Right. There’s all different things. I mean at the very least it’s going to be a technology that you’re probably going to be running into in the future, even if you’re not now.
I mean we have clients now that are having us test large language models, custom implementations in their organizations. And so that that’s going to be a thing that’s coming if it’s not already, I guess probably already here.
And this is we’re actually not going to have time to go through all the slides even in this hour presentation.
Joff Thyer
And this is a great slide though. Derek, I want you to walk these folks through this slide. This is a good one.
Derek Banks
Yeah, I was going to say this is a good segue. And then we’re going to have you jump to like side 36 I think is probably where we want to go. So this is like I said before, large language models are just a component of like the overall AI and machine learning ecosystem.
and so you’ve probably heard some of these terms, said before, even if you’re not in, in the AI, so to speak speak, where all of these different components kind of like build upon each other.
Like you can’t have an expert system, an AI expert system these days without leaning on concepts that came from machine learning and neural networks and deep learning.
Because large language models are essentially like, basically bolting a lot of this stuff together and in a lot of ways. And so I think if you, a lot of the stuff that’s in this slide here, we’re actually going to cover in more in depth in our, in our class, like what is linear and logistic regression?
What does that do? How do you use it? And then we have hands on hands on jupyter notebooks that are going to be walking through. How do you use these? What can you apply it to?
Kind of thing like how can you take away something? And actually we’re going to talk a little bit more about large language models and I’m going to show you a quick demo of something and then I’m sure we’ll be at the top of the hour by then.
So. Yeah, slide.
Joff Thyer
Yeah, so we’re going to go forward a little bit.
Derek Banks
This was a good one.
Joff Thyer
Why don’t you talk to this one real quick, Doug?
Derek Banks
So I guess this is kind of like I got over the pandemic, like kind of what I did for summer, break over the pandemic, I went and got a Master’s degree in data science.
And people often ask me like, why did you as an information security practitioner get in interested in data science? And I think that’s basically.
It goes all the way back to like when Sneakers came out in 1993. Is that I think 92 or 93? back then, Cosmo, I think was the character who said that the world is data.
Right? And like data, I’m sure you’ve heard data is the new oil. Well, a lot of what we do is as infosec and hacking, type, in this profession revolves around making sense out of the data.
And I feel like there’s a lot of overlap between hacking stills and math and stats and machine learning and just overall data science. And one of the things I think that security practitioners would benefit from is just kind of approaching looking at the world that way.
It’s all text and data and we need to manipulate it and make sense out of it. Especially like when it comes to log files and things like. Right, 100%.
Joff Thyer
So let’s jump to slide 36 now.
Derek Banks
Yeah. So, what is a large language model? so, basically now when people say AI, this is what they’re, what they’re talking about.
Right. And so, sorry, I guess I’m getting a telemarketer call. at least it’s not an election call, I guess.
and so large, large language models, I mean they’re fascinating technology. And so essentially like Joff is saying, they take all of like this vast amount of data and it’s essentially, the way I like to look at it is that it’s kind of like a superposition of the knowledge that you fed into it that then you can get output back out.
And so like when, if you. One of the things that you should look at and kind of like a novel concept for large language models in the way that one, of the ways I look at it is, it’s essentially a really large scaled up next word predictor.
So if you’ve ever been typing on your phone and it says like, hey, what’s the next word? Basically that’s an algorithm on the back end is predicting what the most likely next word will be.
But then it’s scaled up to this thing that’s essentially a bunch of neurons running in a GPU that’s as big or bigger than a human brain. And it’s kind of interesting that I heard an interview with Sam Altman on, I think the Lex Friedman podcast, where he basically admitted that OpenAI really doesn’t know what’s going on inside of ChatGPT, like what’s actually happening in there and which is also kind of terrifying, right?
I actually shouldn’t probably terrify you anymore. yeah, I think I did a.
Joff Thyer
Pretty good job of that.
Derek Banks
And we’ll go into more in depth in the class overall, how do these things actually work? But m. It’s really practically like, how do you use them? And so one of the mistakes that I made early on using large language models is I was treating it just like it was Google.
And I mean, I guess I won’t say that’s wrong, right? Like, you can go and just like type in a phrase that you, like, you couldn’t Google and it’ll give you something back. Or you can, type in a question and it’ll give you something back.
But it turns out very interestingly, the better you ask the question, the better results you get. Back again, kind of like a smart high schooler, right?
You can give them a simple instruction and you might get, something that you didn’t, expect, as the output. But if you give them like, much more detailed instructions, then, you’ll probably get a better result.
And so that’s what prompt engineering really is. So this is where we’re going to take, like a better type of prompt or question to the large language model and get a better result out of it.
All Right, next slide. So a little bit of terminology. when we talk about writing a prompt and interacting with a large language model or a couple of concepts, the first one’s a token.
And essentially what a token is, is essentially a word. It can be a word, stem, character, or punctuation or something like that. Really, most of the time a token is going to be word.
And why is that important? Because that’s kind of like the measure. Well, it’s important for getting billed, right? Because I think they build on token usage, but also the context window. And what that means is essentially like how big of a question can I ask?
I have this window of data that I can send into the large language model and get data back out of. And that’s usually like a window of size. And at the moment I want to say that the popular big models that are out there are 128,000 tokens.
Joff Thyer
Yeah, 128,128k token, which is huge. The context window.
Derek Banks
Huge. Right. And then, I know, I’ve heard. I, I Heard earlier this year and I know we’re getting close to the end of the year that by the end of the year open AI was talking about a million token context window which is dude that’s, that’s a lot of data and so why is that important?
And there’s a question earlier I was answering some of the the questions in the teams chat. I, I’m sorry, I, I didn’t get to the discord chat because it was updating the background and it had an error.
computers. but when you train one of these big models right Then once it’s trained it’s pretty much that’s like, I’m looking at it as kind of like a snapshot of that like training right.
And it’s really expensive to do and once that model is in use the context window is kind of how you catch it up to where you are, right? You give it more information so that it can give you a better result, a better output.
And then an Asian is something that is essentially like an autonomous piece of software that’s going to interact with the large language model and perform some kind of function.
Remember if you were here during the pre show banter I mentioned that there’s a company now called Drop Zone AI that claims they have a built in or a drop in SOC knowledge worker where you basically have a sock analyst and it’ll work tickets for your sock.
Now does it work, work? I have no idea. I, I don’t believe in it enough to give it the money at the moment. But this kind of stuff is coming and so why is this important?
Because next slide somebody. I see some, the little like shocked faces are floating up my screen.
Joff Thyer
That’s yeah, I love those.
Derek Banks
so inner fabric. Now this is something that Daniel Meler put out a while back ago, those prompts that I was talking about. Essentially what this is what fabric is a collection of curated and engineered prompts like ready for you to use.
And as all awesome hacker tools are, it’s command line. And so essentially you’ll take text and feed it into fabric and on the back end fabric is going to go and query your text to a large language model.
And remember I said earlier and I’m going to reiterate this in case you’re shocked out of remembering this is. I’m sure that AI is going to take jobs in the future.
100% I do think that’s going to happen. But will it take your job? Well I mean the more you learn about AI and the more you can do with AI and if you can use it and increase your own productivity, you’re going to be better than that AI in a lot of ways.
So I like to think of it more of, AI is meant for us to flourish, not for evil corporations to replace us with.
Joff Thyer
Yeah, let me drive a fine point on that just for a minute, Derek. There’s two concepts here. One is the quality of the prompt that you’re pushing into the AI determines the quality of the output.
And so you, the human, have to engineer that prompt. Okay? The second point is your experience in properly interpreting the output of the generative AI as to its accuracy, its quality and usefulness is an extremely important point as well.
If you just blindly trust a generative AI, you are going to make egregious errors.
Derek Banks
I mean, so to that point, like, I hear people all the time say, well, the AI could be wrong, right? And 100% right. And somebody asked question about hallucination earlier too.
Now, I hope your coworkers aren’t hallucinating on the job, but have you ever had a co worker be wrong? Do you believe everything your co workers tell you as to be like fact that you, you’ll blindly like, believe?
Oh, well, Bob told me this, so it’s got to be true, right? Maybe you have some people like that, right? But I mean, I guess what I’m saying is, we’re hackers. You got to question everything. It doesn’t make sense to you.
Yeah, and so, yeah, good point. And well, now it’s demo time.
Joff Thyer
Oh, by the way, this image was generated also by Dall E3. And, I think what we did, Derek, and correct me if I’m wrong, is we said, hey, man, what if we make MC Hammer into a data scientist?
And if we ask Dall E3 to depict MC Hammer as a data scientist in his lab, and I can’t remember exactly how we phrased something like,
Derek Banks
data scientist MC Hammer telling us it’s demo time, right?
Joff Thyer
And this is what we got. And I think it’s hysterical, but most.
Derek Banks
Impressed by it spelling its demo time correctly. Like, that’s actually what impressed me the most.
Joff Thyer
I think I freaking loved it. Anyway, I’m going to let Derek share and do a little bit of demo with.
Derek Banks
I mean, that’s actually, hey, this worked out time and was pretty well, I think. Okay, so I’m assuming that everyone can see my window.
Please, say something if you cannot but we’re talking about fabric and so.
Joff Thyer
Can’t see it.
Derek Banks
Can you not?
Joff Thyer
I cannot see it. Try again. Love you, man, but I’m glad. There we go. Now we go.
Derek Banks
There we go. Second time worked. So we’re talking about fabric. So fabric has this concept of, patterns, and you can stitch together patterns and you can actually make an API server that’s a loom.
So there’s kind of like, know, a theme going on there. We’re really just going to talk about the patterns. And, if you look at the patterns that they have. Actually, let me do it this way instead.
so we’ll do fabric, dash, dash, list. It’s just list. Or maybe it’s list patterns.
Oh, maybe. I don’t know.
Joff Thyer
there’s nothing like live demos to get the juices flowing.
Derek Banks
Python. So they updated it to go, and I’m pretty sure they changed that flag, but we’re in the directory, so we’ll just look at the patterns. And so there’s a ton of them in here.
And the one that Daniel Meer likes to demo, or I’ve seen him demo on YouTube is, let’s see which one is the one that he used.
Analyze. Analyze. Presentation, I think may. No. Extract wisdom.
Joff Thyer
Yeah.
Derek Banks
So let’s look back. Wisdom.
And so again with this idea of a, smart high schooler and us giving them very specific instructions. Because as the parent of a high schooler, I know the more specific I am in my instructions to my teenager, the better results I will get.
Hey, can you clean your room? Sure. Oh, wait, you left all the dishes in there. Can you clean your room? Take all the dishes downstairs, please? Oh, yeah. Thanks. Yeah. So kind of the same thing here, right?
So your identity and purpose is you extract surprising, insightful and interesting information from text content. You’re interested in insights related to the purpose of meaning of life.
My favorite part, take a step back and think step by step, how to achieve the best possible results following the steps below. Then we give them, some steps and we’re going to want things.
Basically, the idea of this prompt, I’m not going to read the whole thing to you. Go get fast fabric and read it yourself. Is to. To extract, in the best possible way, wisdom out of the text.
We feed it. Now let’s feed it some interesting text. And what I’m going to do is the same thing that Daniel Meisler does. And let’s say that you didn’t make it to Wild West Hackfest 23 and you wanted to see Joe Gray’s talk on the Truth is Out There, but you really don’t have a time to see it today because you came to our webcast instead.
So thank you, but.
Joff Thyer
And you were very wise to do that.
Derek Banks
And so what you can do now is you could So if you don’t know, YouTube has a, transcript for all the videos that are out there that with a API key you can pull down.
YT is a Go application again that Daniel Mesler wrote.
Joff Thyer
And yeah, just hit enter on that. Show the transcript.
Derek Banks
Oh yeah, good point. And so basically this is going to be the transcript for Joe Gray’s talk. And so basically it’s a wall of text. And I know that, online me scrolling up is not the best possible scenario, but basically all of the text that came from that video when YouTube so graciously ran it through a natural language processing model and gives us the text so we don’t have to do it ourselves.
And so what we’re going to do is we’re going to feed that into fabric and do sp. So we’re going to stream the results and we’re going to use the pattern called Extract Wiz Dome and it’s going to go off on the back end.
And the model that I’m using is chat GPT4 I believe. Good lord, it’s fast. Using this on the local llama model is not the same experience, because I don’t have a million dollars for GPUs.
And so anyway, the the idea is here is now I can take, replace one wall of text with another wall of text. Right. So let’s go back up here to the top.
Joff Thyer
But the difference is this is well structured text.
Derek Banks
Yeah. So we get a summary. The speaker discusses techniques, challenges and tools and open source intelligence, OSN and related fields emphasizing critical thinking, private privacy biases involving digital landscapes.
That sounds really interesting. I should probably read that or watch that talk goes into the overall ideas and then gives me some insights quotes, which is actually kind of interesting.
a broken clock is right twice a day. There’s always a shred of truth in every conspiracy. I like that quote. So anyway, you get the idea. We’re using AI to go off and take what would have taken an hour to watch the video and essentially giving us a condensed version of the most key take takeaways, which I think is fascinating.
Right. And so we have three minutes left and you might be thinking, well that’s really great, but that’s Kind of like a novelty. Right? Well, how about. How about this one?
Let’s say. Actually, let me.
Joff Thyer
This is the pot, Derek, where you say, here, hold my beer. Okay, I’ve got your beer, man.
Derek Banks
Right. So Joff and I, created, one like, let’s pretend that you’re a sock worker and you ran across a command and, you’re not really sure what that command does.
Not you. Yes, you can Google it and use the man pages for sure. But, I’m going to do history rep.
Ssh. Exe. Yeah, because I’ve done this one about a billion times. And, it’s funny, when I did the local llama model as an anecdote, I, ran this command twice and, the llama told me I’ve seen that command before because that’s still in the context window.
Joff Thyer
Derek, Derek, clear your screen. Put it at the very top.
Derek Banks
Yeah, actually, let me do this first, before I do that. So let’s look at that pattern. So, CD analyze command.
Zach Hill
Oops.
Derek Banks
M. I should have just seeded into it. And then.
Joff Thyer
Yeah, so while he’s doing that, this is a prompt that Derek and I constructed. All right.
Derek Banks
Yeah. So we give it an identity and purpose. You’re an expert in operating system commands used in Windows, macOS and Linux. we also still want you to take a deep, deep breath, and think about it.
We’ll give you the output sections and my favorite part. Again, with this, I could just Google it, right? actually it’s not in this, so this will be perfect for the demo.
So let’s go ahead and let’s run that command analyze command.
Joff Thyer
Okay, here we go.
Derek Banks
Look and see what this SSH command does. That I found. Man, it is fast.
Zach Hill
How often do you guys utilize Google anymore? Just curious, do you. Do you mainly just put throw everything in like some type of LLM or are you guys just. It’s probably going between both.
Joff Thyer
I balance between both. Yeah.
Derek Banks
Yeah, I’d say 80% large language models, 20% Google.
Joff Thyer
So this was really cool. We. We actually surprised ourself with the level of specificity that the. The large language model came back with.
With the appropriate prompt, I mean.
Derek Banks
Yeah, exactly. So here’s my command. Ssh.exe username, host, a bunch of command switches. Basically this came from an ir. I wrote a blog post on this and it’s definitely is a backdoor, ssh, living off the land, back door.
And so it gives me, what the commands are, the breakdowns. I asked it to do that Give me some example uses. It tells me that there’s malicious use to it.
It a back door. Hey, I mean it is spot on. Right? But what if I to, make a quick change to make it more useful to a sock worker?
And let’s say I wanted to come down here to the bottom and I want to say provide a sigma rule for the, for detecting the command.
save that and let’s see. Let me clear my screen. Oh, that’s not the right command.
Sigma rule. It’s like the punch line right at 101.
Joff Thyer
Aha.
Derek Banks
Huh.
Joff Thyer
And there’s the sigma rule. Yep.
Derek Banks
And so. Oh yeah, so sometimes it doesn’t get the flags in here. Now can you copy and paste this into your sim and it be a rule, man? Probably not, but it is a good place to start to write a rule to detect that, ssh command.
So again, I still think we’re at kind of like the, the beginning or the infancy of using large language models, but that’s kind of a quick and dirty demo and obviously more to come in our class.
Joff Thyer
So we’re going to take the needle off the record. Thanks for watching everybody. Derek and I are very excited about what we’re putting together here for you. We really honestly would love for you to come to the class.
I think that a lot of the community, with the amount of effort and thought we’re putting into this, is going to get a lot out of it. I really do think this mission is a worthy mission.
And with that, I guess over to you, Zach.
Derek Banks
And one thing.
Joff Thyer
So go ahead.
Derek Banks
Class in December and then also in person at Wild West Hackfest in Denver.
Zach Hill
Yeah, that’s right. And I put a link in the chat for y’all, for the class, so you guys should be able to see all the options there for, for what’s coming up. so, yeah, if you guys are interested, definitely check out their class.
And thank you guys for being here to share your knowledge. It’s always fantastic having you guys on and, learning us something new, if you will. You guys always go in depth with this, with the AI, and I love that, man.
So thank you. you guys have a couple minutes for some questions?
Joff Thyer
Yeah, sure.
Derek Banks
Awesome.
Zach Hill
right there was. I think this one’s going to kind of relate directly back to your class. But they said there’s been a lot about the info. What about the SEC. How are we going to keep LLMs from leaking info they shouldn’t?
For example, there are already attacks that can directly extract Training texts, a forward facing chatbot that can leak the company data it’s based on could be a significant concern.
Joff Thyer
Yeah, that is well, offensive. large language model, penetration testing is emerging quickly as something that has to be done.
And adversarial AI. yeah, adversarial AI work is something we are going to talk about, we are engaging in. because that is definitely a concern and you’re right on target.
Right. Because we’re going to have a lot of corporations out there that download one of these base models and then they fine tune train on their own internal knowledge base and then they put it online as an interactive application for their customers.
And so they directly have an exposed attack surface.
Derek Banks
And so there’s a couple of things about that. So yes, there are model related attacks like for extracting data for sure at the moment on the big models, the big like Chad, OpenAI and anthropic.
And I think that they’re probably already doing those types of evaluations and so really I think it’s going to be custom implementations that. And we’re already doing some of those types of tests.
and I think it’s a field that is emerging. As far as this class there will be, we’ll be talking about it. We’re not going to be like the hands on part is not going to be adversarial AI.
That is a class that is likely coming in 2025.
Joff Thyer
Yeah, we’ll touch upon it though for sure.
Derek Banks
Yeah. But what are we talking about? For most folks probably on this call, I would say the biggest thing that you have to worry about at the moment is your internal company data leaking out through people using various large language models without any kind of regulation and shout out to all of the compliance and GRC folks.
Hi Kelly. I think that what I would first do is I would get a policy created in your company about AI.
Joff Thyer
Yes, 100%. We created a policy at Black Hills very quickly through a working group and all companies need a policy on artificial intelligence technology use as quickly as possible.
Derek Banks
Didn’t expect that from hackers, did you?
Zach Hill
Josh Hankins says don’t train or ask, don’t train AI on DOS commands and we’ll be safe forever.
Joff Thyer
So so yeah, that whole obscurity is security argument. I mean it’s just not going to work right.
yeah, the genie’s out of the bottle. the question is can we exert enough governance Right. around it and can we for domain specific applications can we appropriately test the guardrails to make sure that it’s as secure as we can make it?
Zach Hill
Thank you. Sorry, I was getting a message about my kids from my wife so I had make sure I answered that.
Joff Thyer
Yeah.
Zach Hill
going through what level of knowledge do you expect a student of your AI for cybersecurity professionals two day course to have? I’d like to enroll but wanted to make sure I’m prepared to learn.
Joff Thyer
so what’s going to be very very helpful is we’ll do quite a lot of work in Jupyter notebook and in Python scripting. So if you have not learned how to script in Python I would 100% invest yourself in that.
That would be extremely helpful to you. and this is a total. I didn’t mean this to happen this way but I do have a course in Python coding.
so thanks, thanks for I’ll give you the 50 bill later Zach. but yeah but yeah if you want to take my on demand Python course I mean it’s a good way to get skilled up.
Derek Banks
and I would agree 100% because I, there was a point when I was doing the data science masters versus I was literally dreaming in Jupyter notebooks.
and so if you’re going to do data science and AI type hands on work I mean to get started there’s nothing that beats Jupyter and and Python.
There are other things out there but I just honestly I wouldn’t for information security specifically that overlap with Python. I agree with Joff 100%.
Joff Thyer
Right. good strong knowledge of command line fundamentals in Linux operating system stuff is also really good to have I would say. And then the other thing is you don’t have to be an expert in statistics but if you can sort of get yourself if there was any skill I would want you to bring is a little bit of, of familiarity with how a statistical regression works and how how a Bayesian network, phishing detection.
Just a fundamental understanding about how that works is, is the kind of stuff that would be nice to bring into the class if you could. That’s an optional it helps you to have to be thinking in that kind of mode.
I would say yeah, I.
Derek Banks
Agree people get wrapped around the axle of oh God math and so there’s no doubt about it, I’m not going to lie and sugarcoat it There’s a lot of math that’s involved with AI, but again like with other things in infosec, if you understand the concept and essentially like what is it doing?
You don’t necessarily need to be a math expert unless you’re actually doing like research and development in the data science space or you’re working for one of these like big companies where you’re working on new models.
Then yeah, you probably need a math degree. Right. but if you have a, an understanding or like a high school statistics kind of stuff, I think you’ll be in good shape.
Like basically, I think right now the only equation. Well no, that’s not true. the only equation you probably really need to know is like the slope of a line. If, you look at the linear or regression equation and you haven’t taken stats or calculus in a while, it might scare the crap out of you.
But slope of a line. High school algebra?
Joff Thyer
Yeah, high school algebra kind of stuff like max, mean median, basic statistics is good, it’s good to bring into, into the, into the class as a knowledge set and then think about.
The other thing I would encourage people to do is think about your day to day domain specific tasks that you’re trying to solve and how you might map those over to artificial intelligence.
Because those are the conversations we would want to engage in with you. because I think the very fundamental problem that everybody is having is what I would call the square peg, round hole kind of scenario where people want to hammer their problem into like an LLM but, but they don’t have a methodology or, or maybe they’re trying to approach it the wrong way or maybe it’s just not the right tool for the job.
and one of the things we’re going to do in the class is introduce you to actually some task, specific data science that is not large language models.
And for some problems some task specific data science is better suited to solve the problem.
Derek Banks
Yeah, exactly.
Zach Hill
Thank you guys. I have a couple questions here left. but if anybody else has any other questions for you. Is there a good way to get a hold of you guys, reach out in any way?
Joff Thyer
Yeah, you can find us on LinkedIn for sure. check that out. you can probably find me on X Twitter, although that’s a floating batch of flaming dumpster fire right now.
So I may not be on there for a while. LinkedIn is probably the best spot to find me right now.
Derek Banks
yeah, I’m, I, I Don’t social media a lot because I’m old and curmudgeonly I guess, and has nothing to do with the current state of the world or anything. I just, I don’t know, have other things to do.
You can try Discord. If you don’t get an answer, maybe poke somebody on Discord for me. You can always email me. My email is Derek @Black HillInfoSec.com but again, I might miss it.
So if I, if you do try and get in touch with me and I don’t give you an answer in a day or so, please poke at me.
Joff Thyer
Yeah, and, and what I would say was too, if you’re communicating with us, kind of, we’re in the last sprint of developing this class and if you have a domain specific suggestion, help us help you, we’d happy to take, but we’re happy to take suggestions.
Zach Hill
We’re gonna close it out here with this question and before I answer, we answer this question, I also wanna let everybody know we are gonna be doing an ama, session after this. So if you have the Zoom application installed on your device at the bottom of your screen you’ll see breakout rooms here in a second and we’ll open up that AMA room here, as soon as we finish up this question.
Everybody’s welcome to join. It’s a very open format. Any questions you have regarding Starbase security, certifications, your journey resum, we’re here to help you.
So with that said, we’re going to ask this last question and then we’ll kick that room off. So Mark G actually asks a really good question. That is what if I’m 60 plus years old and have never been good with math or programming?
Sounds like a class is a waste of time for me. What do you guys think?
Joff Thyer
what’s your, what’s your goal? I would answer the question with a question. What is your actual goal? Where do you want to go? and think about that.
I’m not a spring chicken either. and I think that the secret, if you want. It’s not even a secret.
The key to survival in this industry is to surf technological waves. And I have made a career of doing exactly that.
So even if you don’t have a specific goal and you just want to surf the waves. Wave, by God, my friend, surf the wave. that’s a philosophy I live by.
Derek Banks
Yeah. So I would say, yeah, exactly. I echo what Josh says, like what do you want to get out of it. Did you get anything out of, like, the history section of the class? I mean, I mean, if you did, then, I mean, I’m sure there’ll be other things that you get out of the class, but I’m definitely not going to, lie and say there’s not going to be coding in the class.
However, the notebooks that we give you basically will run themselves. Right. Like it’s, basically for demo demonstration purposes. Will you, like, as part of the lab, have to do some coding?
Oh, yeah, sure. But if you don’t, I still feel like that you could go through the notebook and get the examples and actually like, learn from it. Right. So like Joss says, it’s up to you.
If you’re like, hey, I never want to code, then maybe not. Right. Because I think that this is really geared toward folks who want to like, dive in, to AI. And so, yeah, I think the fact that you’re here at the, at the webcast, means you’re interested.
Right. So I would say, say, yeah, I, if, if it doesn’t scare you to learn about it, I, Again, I don’t think that. I think you’d find that for most things like AI, like you, you don’t really have to get into the math.
You just kind of have to understand, like, what it’s doing.
Joff Thyer
Right. Conceptually. Yep. and that’s all we’re going to do, by the way. We are not just be clear in the class. We are not going to give you a theoretical mathematics or statistical degree in two days.
Right? Right.
Derek Banks
I mean, we’re not teaching linear algebra.
Joff Thyer
Like, we’re not teaching linear algebra. We may have some knowledge of that, but this is a two day AI class. Right. So it’s going to be applied technology.
Derek Banks
We want you to come away with it, with something like you can take back to like, work and like, make a difference. Right. And if the only thing you take out of that or even this talk, is that we need to have, an AI policy because of stuff is scary, then I mean, that’s a, that’s enough for me.
Joff Thyer
That’s, that’s a win in and of itself. Yeah.
Zach Hill
Awesome, guys. Thank you so much for being here and, sharing your knowledge with us. It’s always, always appreciated. I know everybody has loved, what you guys had to share and, looks like a lot of people be joining your class as well.
So, now at this time we’re going to get started in our ama, but come back next week. We’re going to have Tim Fowler here joining us. Us.
to talk about, well, cyber security and space hacking, space systems. So stay tuned for that. It’s going to be a lot of fun. but until next time, we’ll see y’all later. And, we’ll see some of you in our ama.
Take care, everybody.