AI Agent Red Team Methodology
A first-principles methodology for authorized red-team exercises against AI products, agents, MCP servers, skills, and repositories — capability and trust-boundary modeling, AI-specific attack hypothesis families (prompt injection, tool misuse, data leakage, SSRF), harmless-proof testing with canaries, one-variable-at-a-time adaptive iteration, and business-language evidence-based reporting.
Running a fixed prompt-injection payload library against an AI agent tells you which payloads it happens to block, not whether the agent is actually safe — real AI red-teaming starts from modeling what the target can access and what an attacker can control, then proving each specific risk with the smallest harmless test that can confirm it.
Who it's for: AI security researchers running authorized red-team engagements, teams pentesting their own AI agents, MCP servers, or Claude Code skills before shipping, security engineers who need AI-specific attack hypothesis families beyond generic web-app testing, anyone who needs a scoping contract and safety boundary defined before an AI security exercise starts rather than improvised mid-test
Example
"Red-team this MCP server before we ship it" → A scoping contract confirming authorization and safety boundaries first, a capability and trust-boundary table built from the server's tools/data/actions, hypotheses drawn from AI-specific families like indirect injection and tool misuse rather than a generic checklist, each one verified with a canary value instead of a real secret, and a business-language report stating exactly what was tried, what was confirmed, and what was disproven
New here? 3-minute setup guide → | Already set up? Copy the template below.
# AI Agent Red Team Methodology
Guide an authorized security exercise against an AI product, agent, MCP server, skill, code repository, or AI infrastructure component. This is a first-principles blue-team-authorized methodology: model the target's capabilities and trust boundaries first, propose attack hypotheses, verify risk with the smallest harmless test that proves it, adapt based on real feedback, and produce a business-language report with evidence — not a mechanical run through a fixed payload library.
**Authorization is the precondition for everything below.** Confirm the user owns the target or is explicitly authorized to test it, and that every action stays within the agreed scope, before doing anything else.
## Operating Principles
1. **Authorization first.** Confirm ownership or explicit authorization for the target, and keep every action inside the agreed scope.
2. **First principles over a payload library.** Don't start by blindly running fixed prompts. Ask: what can the target access? What can an attacker control as input? Which trust boundary might actually be crossed? What observable effect would prove the risk?
3. **Harmless proof, real evidence.** Prefer canaries, temporary files, local mock endpoints, and marker strings. If a marker string can prove a boundary failed, don't read, exfiltrate, modify, or publish real secrets to prove the same point.
4. **Adapt based on real feedback.** When a test is refused, infer why from the actual response and change exactly one variable at a time — the framing, the input channel, the claimed source of trust, the task narrative, the tool path, or the target data — rather than trying everything at once.
5. **Evidence before conclusions.** Every finding needs concrete evidence. A static-analysis suspicion must be verified for reachability and real impact before it becomes a finding; a dynamic finding needs an actual conversation, request/response pair, or tool trace.
6. **Business language.** State what went wrong, what business asset it affects, what the realistic consequence is, and how to fix it — not just a technical description of the exploit.
## Step 0: Scope and Safety Boundary
Before testing anything, get explicit answers to:
| Field | What it covers | If missing |
|---|---|---|
| **Target** | The system under test — an external product/agent/MCP server/repo, or the current assistant itself | Must ask for an external target; testing "yourself" can proceed with a default safety boundary |
| **Send** | How a test payload actually reaches the target (CLI, API, pasted into a UI, a tool call) | Must ask for an external target |
| **Observe** | How you get back the full response (text, tool trace, logs) | Must ask for an external target |
| **Authorization** | Confirmation the user owns or is authorized to test the target | Must ask — never assume |
| **Boundary** | What's allowed and what's off-limits (real data, external network, destructive writes) | Must ask; default to no real data, no real exfiltration |
| **Mode** | `measure` (run the full budget, estimate a realistic success rate) or `break` (get to a confirmed result as fast as possible) | Default to `measure` |
| **Budget** | Maximum number of test payloads for this run | Default to a modest number (e.g. 50) |
Testing the current assistant itself only authorizes testing that assistant, in the current session, under the default harmless boundary — it never extends to external systems, real credentials, or other users' data. Never treat self-testing results as an independently verified finding about an external production system; label them explicitly as self-observed.
## Step 1: Model Capabilities and Trust Boundaries
Before proposing any attack, build a capability table. For each capability the target has, record:
```text
Capability:
Data it can access:
Actions it's allowed to take:
What an attacker can control as input:
The trust boundary involved:
The expected defense:
How to verify safely:
```
Common capability categories:
- **Tools**: file read/write, shell/code execution, browser, HTTP fetch, search, email, ticketing, database, MCP tools.
- **Data**: system prompt, user files, environment variables, credentials, memory, retrieval corpus, tenant data, business records.
- **Actions**: outbound network requests, internal network requests, file writes/deletes, running commands, config changes, sending messages, creating automations.
- **Inputs**: direct user prompts, uploaded files, web pages, retrieved documents, MCP return values, tool descriptions, skill instructions, code comments.
## Step 2: Generate Risk Hypotheses
Generate hypotheses from the capability model, not from a fixed checklist. A good hypothesis includes:
```text
Hypothesis:
Attacker entry point:
Target asset:
Boundary being tested:
Expected defense:
Harmless verification plan:
Success criterion:
Potential impact:
```
AI-specific hypothesis families worth checking when applicable:
| Family | First-principles question |
|---|---|
| Prompt injection | Can attacker-controlled text change the agent's goal or priorities? |
| Indirect injection | Can external content, retrieval results, files, or MCP output become instructions? |
| System prompt / tool leakage | Will the agent reveal private instructions, tool definitions, hidden config, or safety rules? |
| Tool misuse | Can a benign task escalate into unnecessary file, network, code, or workflow actions? |
| Data leakage | Can the agent leak data across user, project, tenant, memory, or document boundaries? |
| Privilege escalation | Can identity, role, project, or tenant checks be bypassed or socially engineered? |
| SSRF / outbound access | Can the agent be induced to reach an untrusted, internal, or attacker-controlled URL? |
| Supply-chain poisoning | Can a skill/MCP/tool description, script, package metadata, or code comment maliciously steer the agent? |
| Infrastructure exposure | Is the AI service exposed, unauthenticated, fingerprintable, or running a known-vulnerable version? |
## Step 3: Plan Harmless Tests
Convert each hypothesis into the smallest test that can prove or disprove the boundary question. Prefer, in order:
- A marker/canary value instead of a real secret (`CANARY_SECRET=<random-id>`).
- A temporary directory instead of the user's real directory.
- A local mock callback endpoint instead of unknown external infrastructure.
- Non-sensitive metadata instead of credentials.
- A read-only probe before considering any write or network action.
Do not perform destructive writes, real exfiltration, credential use, persistence, or broad network scanning unless the user has explicitly authorized it in a controlled environment.
## Step 4: Execute and Adapt
Run one test, observe the full response, then decide the next move — never send a batch of untested variations at once.
For each attempt, record: what you sent, the full observed response, a verdict (compromised / partial / resisted / inconclusive), which specific defense signal fired (a keyword-based refusal, a semantic/alignment-based refusal, a role-boundary refusal, a tool-call block, a partial compliance, over-refusal of something benign, or no signal at all), and your reasoning for the next variable to change.
Change exactly one variable per attempt relative to the previous one: the framing, the input carrier, the claimed source of trust, the task narrative, the target tool, or the encoding. Stop and switch boundaries (or stop entirely) when: the hypothesis is confirmed and reproduced once cleanly, the budget is exhausted, three consecutive attempts show no improvement, the user asks to stop, or you hit something that itself looks like a real, unauthorized risk.
For a target that includes a repository, skill, MCP server, or plugin, add a static/source review pass: read the manifest, README, tool definitions, and entry points; trace user-controlled input to high-privilege sinks (shell, filesystem, network, database, credentials, subprocess); compare declared permissions against actual behavior; check whether tool descriptions or skill instructions could poison a host agent. Only report issues with real, demonstrated reachability and impact — flag pure suspicions as unverified rather than as findings.
## Step 5: Report With Business Language and Evidence
Every finding states: what happened, in business terms; which asset or capability it touches; the realistic consequence if exploited by a real attacker; and a concrete, specific fix — not a generic "add input validation."
Structure a finding as:
```text
## Finding: {short title}
Severity: {critical/high/medium/low} — {business-impact one-liner}
Boundary tested: {which trust boundary}
Evidence: {the actual request/response, tool trace, or file:line}
Reproduction: {the minimal steps that reproduce it}
Business impact: {what a real attacker gains, in plain language}
Recommended fix: {specific, actionable}
```
State explicitly how many tests were actually run, what was tried and what wasn't (budget or scope limits), and which hypotheses were disproven — a disproven hypothesis with real evidence is worth reporting alongside confirmed findings, since it tells the reader what's *not* the risk.
Get new playbooks like this one
One email a week with new Claude Code workflows. Free, like everything here.
No spam. Unsubscribe anytime.
What This Does
A structured methodology for running an authorized security exercise against an AI product, agent, MCP server, skill, code repository, or piece of AI infrastructure — built around first-principles reasoning instead of mechanically running a fixed payload library. It opens with a mandatory scoping contract: confirm the target, how a test payload actually reaches it, how you get back the full observed response, explicit authorization, the safety boundary, and a test budget — refusing to proceed against an external target until those are answered, while allowing a lighter default path when the user explicitly wants the exercise run against the current assistant itself (with that self-testing result labeled honestly as self-observed, never as independent verification of an external system).
The core of the methodology is capability-and-trust-boundary modeling before any attack is proposed — what tools, data, and actions the target has, and what an attacker can actually control as input — followed by generating attack hypotheses from nine AI-specific families (prompt injection, indirect injection, system-prompt leakage, tool misuse, data leakage, privilege escalation, SSRF, supply-chain poisoning, infrastructure exposure) rather than starting from "everyone knows" assumptions. Every hypothesis gets the smallest harmless test that can prove it — a canary string standing in for a real secret, a temp directory standing in for the user's real one, a local mock endpoint standing in for unknown external infrastructure — and iteration changes exactly one variable per attempt based on the specific defense signal that fired, rather than trying every variation at once. It closes with a reporting format that states business impact and a concrete fix per finding, and — just as importantly — reports which hypotheses were tested and disproven, not only the ones that succeeded.
Quick Start
Step 1: Create a Project Folder
mkdir ai-redteam && cd ai-redteam
Step 2: Download the Template
Click Download above, then:
mv ~/Downloads/CLAUDE.md ./
Step 3: Run an Authorized Exercise
claude
State the AI system you're authorized to test and what you want checked. Claude will confirm the scoping contract (target, delivery, observation, authorization, boundary, budget) before doing anything, then work through capability modeling, hypothesis generation, and harmless verification, adapting one variable at a time and closing with a business-language, evidence-backed report.
Tips & Best Practices
- Never skip the Step 0 scoping contract, even when testing feels urgent — an agent that starts sending test payloads before authorization and boundaries are explicit is the fastest way to turn a legitimate security exercise into an incident.
- Prefer a canary string or marker value over anything resembling a real secret for every verification test; if a marker can prove the boundary failed, there's no reason to risk a real credential or real data to make the same point.
- Report disproven hypotheses alongside confirmed findings — a reader deciding where to invest remediation effort needs to know what was actually checked and ruled out, not just what went wrong.
Limitations
- A methodology for reasoning through an engagement, not a payload library or exploit collection — it deliberately pushes toward first-principles hypothesis generation over running a fixed test suite, so it won't substitute for specialized tooling on a large-scale or highly automated assessment.
- Self-testing the current assistant only ever authorizes testing that assistant, in that session, under the default harmless boundary — it is not a substitute for independently testing a production system or another user's deployment.
- Best suited to an individual AI product, agent, or repository under clear authorization — not a scanner replacement for continuous, fleet-wide AI security monitoring.