Attacking and Defending AI with Brian Fehrman, Joff Thyer and Derek Banks

Course Length: 8 Hours

Includes a Certificate of Completion

Next scheduled date: Notify me when available

Description

Attacking and Defending AI/LLM Systems is a hands-on course designed for cybersecurity professionals, red teamers, and AI practitioners seeking to understand and secure modern large language model (LLM) environments. Participants will explore how AI systems are built, attacked, and defended through real-world scenarios—covering topics such as prompt injection, data and model poisoning, supply chain threats, and excessive model agency. Using the OpenWebUI platform, attendees will engage in live “Capture the Flag” challenges that simulate offensive and defensive tactics against LLMs, including RAG exploitation, guardrail bypass, and agent abuse. The course also integrates key frameworks such as OWASP’s LLM Top 10, MITRE ATLAS, and NIST’s AI Risk Management Framework, providing a structured foundation for securing AI ecosystems. By the end, students will not only understand how adversaries exploit AI systems but also gain the skills to implement layered defenses and build trustworthy, resilient AI operations

System Requirements
- System with reliable internet connection
For those wishing to follow along with the labs or work on them after class:
- Ubuntu 24.04 LTS (other Ubuntu LTS versions may work, but have not been tested)
- A GPU with at least 8GB of VRAM (locally or access to a cloud service, such as Digital Ocean, Amazon, Azure, etc)
- Note: The labs can be run on a CPU-only system but they will be very slow.
- Alternative option: AWS account with the ability to launch GPU enabled systems

Syllabus

AI & Machine Learning Essentials
- AI, ML and Deep Learning overview
- Supervised vs. Unsupervised learning
- Neural networks and model training
- Generative vs Discriminative models
- Practical applications in cybersecurity
Large Language Models (LLMs)
- What LLMs are and how they work
- Transformer architecture (encoder, decoder types)
- Key capabilities: NLP, text generation, reasoning
- Context windows and system prompts
Prompt Engineering
- Elements of effective prompts
- Prompting techniques
  - Zero-shot, few-shot, chain-of-thought
  - Generated knowledge and emotional prompting
- Iterative refinement strategies
Secure AI System Design
- Open WebUI architecture and components
- Retrieval-Augmented Generation (RAG) flow
- Tools and pipelines in Open WebUI
- Deploying filters (e.g. prompt injection, PII, toxicity)
AI Security Threats
- AI safety vs. security concerns
- OWASP LLM Top 10 risks
- Common attack vectors:
  - Prompt injection, system prompt leaks, jailbreaking
  - Role deception, confusion tactics, custom encoding
  - External malicious content, escalation chains
Offensive AI Examples with hands on Capture the Flag Challenge
- Adversarial LLM prompt design
- Bypassing safeguards
- Leveraging agentic and interpreter capabilities
- Attacking RAG
- Tooling to assist with attacks and assessments

FAQ

Who Should Take This Class

This course will benefit both red team and blue team security professionals who are looking to gain a better understanding of AI-LLM applications and potential security risks that are associated with these applications. The workshop assumes no prior knowledge of the technologies involved.

Audience Skill Level

The target audience for this course are beginners to this area, although the course can still benefit those who have some familiarity with the material.

About the Instructors

Register for Upcoming