Getting Started with AI Security Testing: Resources for Pentesters and AppSec Professionals
I’m thrilled to have recently completed the AI Red Teamer path on HackTheBox and earned the badge 🧠 🔐
But more importantly, I want to use this opportunity to share useful resources with the community — things that helped me and might help you navigate the growing field of AI security testing.
AI is rapidly becoming part of our daily lives, and companies are trying to adopt new technologies wherever possible. Naturally, pentesters and AppSec engineers are increasingly tasked with testing AI-powered products. But AI systems aren’t your typical web apps, APIs, or mobile apps — we don’t yet have a huge number of well-established guides or standardized environments for testing them.
That’s why I’ve collected some of the most valuable resources to help build your strategy, get hands-on practice, and perform proper security testing of applications with AI features.
🛡️ OWASP Top 10 for Large Language Model Applications
This is arguably the most essential reference when it comes to testing AI and LLMs. You probably already know OWASP, and their LLM Top 10 is a fantastic starting point.
It provides up-to-date information on risk types and mitigations for LLMs, including:
- Prompt Injection
- Bias and Toxicity
- Data Leakage
- Data Poisoning
- Hallucinations / Confabulations
- Agentic Vulnerabilities
- Supply Chain Risks
Another must-read is the GenAI Red Teaming Guide, which covers:
- Key Risks to Consider
- Steps for Vulnerability Testing
- Differences Between Traditional and AI Testing
- Best Practices
Also, check out genai.owasp.org for video content, initiatives, and the roadmap.
🔐 Google SAIF: Secure AI Framework
Alongside OWASP, another must-know initiative is Google’s Secure AI Framework (SAIF). It’s designed to help organizations approach AI system security in a more holistic and scalable way.
SAIF outlines a security-first approach to AI, covering six key principles:
- Secure the AI Supply Chain
- Secure the AI Development Lifecycle
- Secure the Deployment Environment
- Secure the AI Inputs and Outputs
- Monitor and Audit AI Systems
- Adapt Controls to Emerging Risks
It’s more strategic than hands-on, but still a great resource if you're building security programs around AI or integrating security into the ML lifecycle.
🧪 PortSwigger: Web LLM Attacks
Time to move from theory to practice. After going through the OWASP materials, reinforce your knowledge with hands-on labs from PortSwigger's Web LLM attacks section.
Here are four great labs for practical exercises:
- Exploiting LLM APIs with Excessive Agency
- Exploiting Vulnerabilities in LLM APIs
- Indirect Prompt Injection
- Exploiting Insecure Output Handling in LLMs
🎓 HTB Academy: AI Red Teamer
This course requires cubes (not free), but even the lowest-tier subscription for a month is enough. It offers deep insights into how AI models work, how they're built, algorithms, testing, and even how to create your own tools and models.
The most relevant sections for pentesters:
Other modules like Fundamentals of AI and Applications of AI in InfoSec are also packed with practical and theoretical knowledge.
🧩 HTB Labs: AI and ML Exploitation Track
A solid set of 9 challenges focused on AI and ML Exploitation.
Not all of them are focused on prompt injection — some require scripting to extract flags from models — but challenges like Prometheon are entirely centered around prompt injection and are especially relevant.
Whether you're just starting with AI security or already deep into testing, these resources will help you build your skills and stay ahead of the curve. If you’ve found other useful tools or labs — I'd love to hear about them!