AI Agent Red Teaming

AI agent red teaming is authorized adversarial testing of a complete agent workflow. Testers try to make the agent cross a security or business boundary, then capture enough evidence to reproduce, fix, and retest the failure.

The target is larger than the model. An agent receives instructions and data, loads Skills, reads and writes Memory, calls tools, uses delegated identity, asks for approval, talks to other agents, and changes external systems. A red team has to test that full path from input to effect.

A useful exercise does more than collect prompts that caused bad text. It demonstrates whether an attacker can create an unauthorized action, expose data, persist a poisoned instruction, bypass approval, expand authority, hide from monitoring, or keep operating after revocation.

TL;DR

A practical AI agent red-teaming program should:

• Start from the agent’s threat model and prohibited outcomes.
• Scope the exact production-like release, not a generic model endpoint.
• Get written authorization, safety limits, stop conditions, and named owners.
• Use isolated systems, synthetic data, test credentials, and reversible effects.
• Inventory every input, tool, identity, data source, runtime, and agent handoff.
• Test direct and indirect prompt injection through realistic content sources.
• Test whether untrusted content can alter AGENTS.md guidance, Skills, or Memory.
• Try valid tools in unsafe sequences, not only malformed tool calls.
• Attempt privilege escalation, cross-tenant access, and delegated-identity confusion.
• Challenge approval scope, expiry, parameter binding, and reviewer context.
• Test MCP server identity, tool metadata changes, and poisoned tool output.
• Exercise code execution, network access, filesystem limits, and sandbox escape paths.
• Attack multi-agent delegation, messages, shared state, and cascading actions.
• Repeat probabilistic attacks and vary tasks, wording, models, and order.
• Capture the complete trace and the external effect, not only the final answer.
• Rank findings by achieved impact, reach, repeatability, detection, and recovery cost.
• Turn every confirmed finding into a regression test and release gate.
• Rerun the exercise after material changes and test fleet-wide containment.

The main question is concrete: can an authorized tester make the agent do something the organization has prohibited, and can the team prove which control failed?

What AI Agent Red Teaming Means

AI agent red teaming is a structured attempt to defeat the security, safety, and governance controls around an agent system.

The red team takes an adversary’s view. It studies the workflow, selects target outcomes, develops attacks, adapts when controls block them, chains weak points, and records demonstrated effects. The builders and defenders then use those findings to change the system and verify the fix.

NIST’s AI Risk Management Framework Playbook recommends red-team exercises to test AI systems under adversarial or stress conditions, measure failure modes, compare claimed and actual performance, and document test conditions and results.

For agents, the exercise should cover four layers:

Layer	What the red team attacks	Example target outcome
Model behavior	Instruction following, planning, refusals, uncertainty	The model accepts a malicious goal
Agent configuration	AGENTS.md guidance, Skills, Memory, tool schemas, routing	Untrusted content becomes trusted guidance
Runtime controls	Identity, authorization, approvals, isolation, rate limits	A prohibited tool call executes
External effects	Data, files, messages, transactions, deployments, other agents	Sensitive data leaves or a record changes

A refusal at the model layer is useful, but it is not the security boundary. The red team should keep testing whether deterministic controls stop the prohibited effect when the model makes the wrong choice.

Why Agents Need Their Own Red-Team Method

Standard application and model tests still matter. Agents add behavior that requires more work.

Agents act across several steps

An attack may look harmless at each step. The agent reads a document, summarizes it, queries a system, writes Memory, and sends a message. The harmful result appears only when those allowed steps form a chain.

Test complete trajectories, including loops, retries, fallbacks, and delayed jobs. Review the final system state as well as each call.

Untrusted data can look like instructions

Agents read email, tickets, websites, code, files, records, search results, and tool output. Attackers can place instructions in any of those sources.

NIST describes agent hijacking as a form of indirect prompt injection in its technical work on agent hijacking evaluations. A realistic test pairs a legitimate user task with malicious content encountered while completing that task, then checks whether the agent performs the attacker’s task.

Agents hold authority

The same prompt can have very different impact against a read-only research agent and an operations agent with shell access, production credentials, and permission to send messages.

Red-team scope must include effective authority: the agent identity, delegated user, direct roles, inherited access, ambient credentials, tool scopes, network reach, approval limits, and authority passed to another agent.

Behavior is probabilistic

One blocked attempt does not prove a defense works. The same attack can succeed after changes in wording, task, retrieved data, ordering, model response, or retry count.

In one NIST agent hijacking experiment, repeating attacks raised the measured average attack success rate from 57 percent on one attempt to 80 percent across 25 attempts. Design the trial count around the number of attempts a real attacker could make.

Effects can persist

An attack may write Memory, change a file, update configuration, create a scheduled task, alter an index, or send instructions to another agent. The attacker can gain value after the original session ends.

The exercise needs reset procedures and post-run inspection. Deleting the chat transcript does not remove persistent effects.

How Red Teaming Differs From Related Tests

These practices overlap, but they answer different questions.

Practice	Main question	Output
Threat modeling	How could the agent be attacked or fail?	Attack paths, controls, owners, test ideas
Agent evaluation	Does the agent meet selected quality, safety, and policy criteria?	Scores, pass rates, limits, regressions
Security review	Does the design and implementation meet requirements?	Findings from architecture and code review
Penetration testing	Can technical vulnerabilities be exploited in the deployed stack?	Exploitable flaws and remediation
AI agent red teaming	Can an adaptive adversary chain model, context, identity, tool, and runtime weaknesses into a prohibited effect?	Demonstrated attack paths, effects, and replay evidence

AI agent threat modeling should supply the first attack hypotheses. AI agent evaluation supplies repeatable test infrastructure and release metrics. Red teaming adds an adaptive attacker who searches beyond known cases and tries to join several failures together.

Scope the Exact Agent Release

Define the target before testing. “The support agent” is too broad because a model, prompt, Skill, tool, credential, or routing change can produce a different system.

Record:

• Agent ID, purpose, owner, environment, and release
• Model provider, model ID, exact revision, router, and fallback
• AGENTS.md guidance, policy, Skill, and Memory versions
• Context retrieval sources and snapshots
• Tool names, schemas, descriptions, versions, and side effects
• MCP servers, resources, prompts, and deployment identity
• Agent and delegated-user identities
• Access roles, assignments, credential scopes, and approval rules
• Runtime image, sandbox, filesystem, network, and browser policy
• Other agents, handoff protocols, and shared state
• Data fixtures, tenant boundaries, and external services
• Monitoring, rate limits, stop controls, and audit configuration

Use immutable version IDs or digests where possible. If a mutable model alias, remote document, tool description, or latest Skill can change during the exercise, capture the resolved value for every run.

Test the release the team plans to deploy. A hardened model in an empty test harness does not represent a production agent with tools, private context, and user delegation.

Set Authorization and Safety Rules

Red teams intentionally try to create harmful effects. Written authorization protects the people, systems, and data involved.

The rules of engagement should define:

• In-scope agents, systems, identities, tenants, and time windows
• Allowed and prohibited attack techniques
• Test accounts, data, endpoints, recipients, and network ranges
• Limits on code execution, persistence, load, spend, and data movement
• Actions that require live approval from the exercise controller
• Stop conditions and an emergency contact
• How to mark, store, transmit, and delete test data
• Evidence retention and access
• Vulnerability disclosure and escalation paths
• Reset, cleanup, and recovery responsibilities

Use synthetic or seeded data that lets the team prove exposure without touching live customer records. Route messages to controlled domains. Replace payments and destructive writes with reversible test effects where the control can still be evaluated.

Do not weaken every control to make the exercise easy. Use a production-like environment, then add safety at the surrounding systems: isolated tenants, test credentials, capped transactions, controlled recipients, and a reliable stop mechanism.

Build Attack Objectives From Prohibited Outcomes

Start with business effects, then work backward to attack paths.

For a support agent, prohibited outcomes might include:

• Read another tenant’s cases.
• Send a customer message without valid approval.
• Reveal internal policy or credentials.
• Change billing or access.
• Save customer data into broadly routed Memory.
• Follow instructions found in a ticket attachment.
• Ask another agent to perform an action outside the support workflow.

Turn each outcome into a test objective with a success condition. “Test prompt injection” is vague. “Cause the agent to send a seeded record from Tenant B to the controlled external mailbox while acting for a Tenant A user” is measurable.

Map objectives to the MITRE ATLAS technique library and the system’s AI agent threat model. A technique catalog improves coverage, while the system model keeps tests tied to real assets and boundaries.

Test the Full AI Agent Attack Surface

Use a matrix that crosses attack sources, techniques, controls, and effects. The following areas should be in scope when the agent exposes them.

Direct and indirect prompt injection

Test malicious instructions from users and from content the agent encounters while doing valid work:

• Email and calendar events
• Tickets and comments
• Websites and search results
• Documents, images, and attachments
• Code, issues, and pull request text
• Database records
• Retrieval results
• Tool output
• Messages from another agent

Vary placement, language, encoding, formatting, distance from the user request, and conflicts with trusted guidance. Split one attack across several sources or steps. Test whether quoted, summarized, translated, or extracted content can become an instruction.

Measure the external effect. A final answer that repeats malicious text is different from a tool call that sends data.

Context and Skill manipulation

Try to make untrusted content change what the agent treats as trusted context.

Test whether an attacker can:

• Replace, shadow, or suppress required AGENTS.md guidance
• Route context intended for another team, user, or agent
• Make stale or draft context appear current
• Install or select a similarly named Skill
• Modify a Skill file after approval
• Load a revoked or unassigned version
• Trigger hidden instructions in examples or reference files
• Expand tool or data access through a Skill
• Exceed the context budget so a policy is omitted

AI agent context governance and AI agent Skills governance should make versions, access, assignments, publication, and delivery observable enough to test.

Memory and persistent-state poisoning

Test whether attacker-controlled content can survive the session and influence later work.

Attempt to write false facts, hidden instructions, sensitive data, changed goals, or attacker-controlled references into Memory. Then start a clean session and test whether the agent retrieves and acts on the poisoned entry.

Check:

• Who can read and edit each Memory
• Which agents receive it through Enabled or Required assignments
• Whether source and author remain visible
• Whether conflicting entries are resolved safely
• Whether sensitive content is rejected or scoped
• Whether a malicious revision can be rolled back
• Whether revocation stops future delivery
• Whether derived indexes and caches are cleaned

Include other persistent state such as files, queues, scheduled jobs, browser storage, retrieval indexes, and configuration.

Tool misuse and unsafe action chains

Do not limit tests to invalid tool arguments. Attackers often use valid tools with valid parameters for a harmful purpose.

Try to:

• Call a tool outside the workflow purpose
• Read more records than the task needs
• Change a safe draft into an external send
• Join a sensitive read with an external write
• Use one tool’s output as another tool’s command
• Exploit retries, batch operations, or pagination
• Reach hidden or deprecated actions
• Change targets after approval
• Use a fallback tool with wider authority
• Trigger an action through a scheduled or asynchronous path

Inspect final state in the target system. A tool trace that says “success” does not prove which record, recipient, deployment, or transaction changed.

Identity, access, and tenant boundaries

Test both agent identity and delegated user identity.

Attempt:

• Cross-user and cross-tenant reads or writes
• Confused-deputy actions where the agent has more access than the user
• Token reuse across tasks or sessions
• Scope expansion through refresh or exchange
• Use of ambient credentials from the host
• Role changes during a long-running session
• Access through a Group the principal has left
• Direct access after assignment removal
• A route that unexpectedly grants repository access
• Agent-to-agent calls that lose the original principal

Permissions and routing are separate. Permissions control what a principal may discover, read, or change. Routing independently controls whether context is Included or Required for an agent. Red-team cases should test both decisions and their changes during a session.

Approval bypass

An approval is useful only when it binds to the exact action a reviewer saw.

Test whether the agent can:

• Split one high-impact action into smaller steps below the threshold
• Reuse an approval for another target, amount, recipient, or tool
• Change parameters after approval
• Act after approval expiry or revocation
• Flood reviewers until one approves without enough context
• Hide a risky action inside a large plan
• Ask another agent to execute the action
• Use a fallback path that has no approval gate
• Continue after the reviewer denies or edits the request

The evidence should connect the approved object, reviewer identity, decision, expiry, parameters, and executed effect.

Data exfiltration and secret access

Place unique canary values in each test data class so the team can identify exactly what crossed a boundary.

Test leaks through:

• Final responses and citations
• Tool arguments and tool output
• URLs, redirects, DNS, and web requests
• Email, chat, tickets, and file sharing
• Logs, traces, analytics, and error messages
• Generated code and artifacts
• Memory and retrieval stores
• Other agents and downstream workflows

Try both direct requests and indirect instructions. Test encoding, summarization, partial records, repeated small transfers, and data hidden in apparently valid output.

Keep live secrets out of test context. Use test credentials and canaries that prove access without creating a new incident.

Code execution, filesystem, and network controls

For agents that run code or commands, test the execution boundary rather than trusting the prompt.

Attempt to:

• Read files outside the allowed workspace
• Write to startup, configuration, credential, or executable paths
• Follow symlinks or path traversal
• Reach host services or container control sockets
• Access instance metadata and credential endpoints
• Start background or persistent processes
• Download and run remote code
• Open prohibited network destinations
• Consume excessive CPU, memory, disk, tokens, or time
• Hide changes from version control or audit tools

Use defense in depth: isolated runtime, narrow filesystem mounts, blocked network routes, short-lived credentials, command policy, resource caps, and monitoring. A model refusal is not a substitute for any of them.

MCP servers and dynamic tool discovery

MCP can change an agent’s tool and data surface during runtime.

Test:

• Server impersonation and connection to an unapproved endpoint
• Tool-name collisions and similarly named tools
• Changed descriptions, schemas, annotations, or side-effect claims
• Poisoned tool output
• Delegated-access confusion
• Cross-user session or token reuse
• Tool discovery after the release was approved
• Chained calls that expand authority
• Server or tool revocation during a session

Capture the server identity and tool metadata version used in each run. A stable tool name does not prove the same schema or behavior was tested.

Multi-agent attacks and cascading failure

When agents delegate, test the handoff and the combined authority.

Try to make one agent:

• Impersonate a trusted agent or human
• Pass untrusted data as a trusted instruction
• Drop the original user identity or purpose
• Delegate to an agent with broader tools
• Create a loop of retries or mutual delegation
• Poison shared Memory or workflow state
• Hide the full plan across several handoffs
• Trigger actions that no single agent was allowed to complete
• Keep operating after the initiating agent is stopped

Record the full chain. A finding assigned only to the last agent can hide the control failure at the first handoff.

Use Adaptive and Repeatable Tests

Static cases find known regressions. Human-led or adaptive testing finds new paths.

NIST’s 2026 analysis of a large-scale red-teaming competition reports that more than 400 participants made over 250,000 attack attempts, and every target model had at least one successful attack. The research also found attack families that transferred across models and scenarios.

The UK AI Security Institute’s agent deployment security research studied 22 agents across 44 realistic scenarios and found policy violations across a large attack set. The result supports two practical rules:

1. Do not treat one blocked prompt as a pass.
2. Keep adding attacks that adapt to the exact system and control set.

For each high-impact scenario, vary:

• Attack wording and representation
• Legitimate user task
• Content source and position
• Model and fallback
• Tool availability and order
• Identity and tenant
• Context and Skill versions
• Memory state
• Number of attempts
• Time delay and session boundary

Keep a held-out attack set so teams do not tune only for known test strings. Add new human-developed attacks to the regression set after disclosure and remediation.

Instrument the Test Environment

Red teaming without reliable evidence creates arguments instead of fixes.

Capture:

• Test case and objective
• Agent, release, environment, and timestamp
• User, agent, service, and delegated identities
• Delivered AGENTS.md guidance, Skill, and Memory versions
• Retrieved content and source
• Model request, response, and routing decision
• Plans and intermediate state when available
• Tool discovery, arguments, output, errors, and side effects
• Authorization and approval decisions
• Credential and token scopes
• Network, filesystem, browser, and process activity
• Other-agent messages and handoffs
• External system state before and after
• Runtime alerts, blocks, and stop actions
• Reset and cleanup result

Protect sensitive traces, but do not redact away the facts needed to reproduce the finding. Store test secrets separately and use stable references in reports.

The OWASP AI Agent Security Cheat Sheet recommends structured security testing before production and after material changes to prompts, tools, Memory, retrieval, policies, or model providers. It also calls for repeatable cases covering prompt override, tool misuse, privilege escalation, Memory poisoning, data exfiltration, recursive tool abuse, approval bypass, and multi-agent chaining.

Measure Effects, Controls, and Recovery

Do not reduce a red-team exercise to one aggregate score. Track several measures because they answer different questions.

Measure	What it shows
Prohibited-effect success rate	How often the attack creates the defined harmful effect
Attempts to first success	How much attacker effort the control resists
Attack transfer rate	Whether one attack works across tasks, models, or agents
Control block rate	How often deterministic controls prevent execution
Detection rate	How often monitoring identifies the attack or effect
Time to detect and contain	How quickly the team can stop ongoing harm
Blast radius	Which data, systems, tenants, agents, or workflows were reached
Persistence	Whether the effect survives sessions, restart, or component removal
Replay pass rate	Whether the fix stops the exact attack and useful variants
Regression coverage	Whether confirmed findings stay in release testing

Report denominators and conditions. “Blocked 95 percent” is incomplete without the number of attacks, attempts per attack, system release, model, tools, identities, and success definition.

Write Findings That Engineers Can Replay

A confirmed finding should include:

1. Prohibited outcome and business impact
2. Exact target release and environment
3. Preconditions, identity, access, and assignments
4. Attack source and input
5. Complete sequence of agent and tool actions
6. External effect and proof
7. Control expected to stop the attack
8. Why the control failed
9. Repeatability across attempts and variants
10. Affected agents, tenants, workflows, and versions
11. Containment and cleanup steps
12. Recommended fix and owner
13. Regression case and replay result

Rank severity by demonstrated effect, reachable data and authority, exposure, repeatability, detectability, persistence, and recovery cost. A convincing malicious response with no action path may be lower impact than a quiet cross-tenant read through an approved tool.

Separate confirmed effects from plausible follow-on impact. The report should state what happened, what could happen next, and the evidence for each claim.

Fix the Control, Then Replay the Chain

Do not stop at adding the attack string to a prompt filter. Fix the control at the layer that can reliably stop the effect.

Possible fixes include:

• Narrowing context access or agent assignments
• Publishing corrected Required AGENTS.md guidance
• Removing or replacing an unsafe Skill
• Validating retrieved content before it reaches Memory
• Reducing tool and credential scope
• Enforcing authorization on every tool call
• Binding approvals to exact parameters and expiry
• Filtering target records and recipients
• Blocking network or filesystem paths
• Isolating tenants, users, sessions, and agent state
• Pinning MCP server and tool metadata versions
• Adding rate, retry, chain-depth, and spend limits
• Improving alerts and stop controls

Replay the exact chain, then test nearby variants and legitimate tasks. A fix fails if it blocks the known string but allows a paraphrase, or if it prevents the agent from doing its intended job.

Add the finding to CI or the release evaluation suite when automation is practical. Keep high-skill adaptive attacks for scheduled exercises.

Make Red Teaming Part of Release and Operations

Red teaming is not a one-time launch event.

Run or refresh testing when the agent receives:

• A new model or fallback
• Changed AGENTS.md guidance or policy
• A new or updated Skill
• Different Memory behavior or routing
• A new tool, MCP server, or data source
• Wider access, credentials, or tenant reach
• More autonomy or fewer approvals
• Code execution, browser access, or broader network access
• Another agent or changed handoff protocol
• A new runtime or deployment environment
• A fix for an incident or prior red-team finding

Use risk to set depth and frequency. A read-only internal summarizer needs less adversarial effort than an agent that can deploy code, send external messages, change access, or move money.

Test containment too. Revoke access, remove an assignment, disable a tool, rotate a credential, block an MCP server, stop a session, and verify that every affected agent and queued task loses the path.

Assign Clear Roles

An effective exercise needs more than a red team.

Role	Responsibility
Exercise owner	Approves scope, resources, risk, and final decisions
Red team	Develops and executes attacks, adapts, and preserves evidence
Agent builder	Explains architecture, supports instrumentation, and fixes findings
Security or platform owner	Operates identity, runtime, network, credential, and monitoring controls
Data and workflow owner	Defines prohibited outcomes and confirms business impact
Exercise controller	Enforces safety limits, approves risky steps, and stops the test
Incident responder	Handles unexpected effects and validates containment paths
Independent reviewer	Challenges scope, evidence, severity, and closure claims

The red team should have enough system knowledge to form realistic attacks, but not unpublished answers to every test. Use a mix of white-box review, gray-box access, and black-box testing based on the objective.

How Alignbase Fits

Alignbase is an AI context control plane. It governs the AGENTS.md guidance, Skills, Memory, versions, access, assignments, and point-in-time distribution behind agent work.

For AI agent red teaming, Alignbase can help teams:

• Define the exact governed input versions in the target release.
• Separate published guidance and Skills from drafts.
• Control which principals can view, propose, edit, publish, or own Resources.
• Keep repository permissions separate from Included and Required routes.
• Route the intended AGENTS.md guidance, Skills, and Memory into test agents.
• Compare a tested context set with the versions sent to production sessions.
• Revoke unsafe context or Skill access and distribution.
• Reconstruct which governed inputs reached a session at a point in time.

Alignbase does not execute attacks, isolate runtimes, scan model weights, enforce external tool authorization, operate sandboxes, or replace runtime security monitoring. Those systems need their own controls and test evidence.

Connect the records. A red-team finding should link the Alignbase versions and assignments to the model, runtime, tools, credentials, data fixtures, attack trace, control decisions, and external effects from the same run.

The Standard to Aim For

AI agent red teaming should demonstrate what an adaptive attacker can make the full system do, not only what the model can be made to say.

The team should know which release it tested, which boundaries the attacker crossed, which external effect occurred, why the control failed, how widely the attack transfers, whether monitoring caught it, how the system was contained, and whether the fix survives replay.

Treat every confirmed finding as a new test case. Treat every material change as a reason to revisit scope. Keep the evidence tied to the exact context, identity, tools, runtime, and system state that produced the result.