AI Agent Threat Modeling

AI agent threat modeling is a structured way to find how an agent could be manipulated, misuse authority, expose data, corrupt persistent state, or cause another harmful effect before the system reaches production.

The model maps the complete agent workflow, not only the language model. It covers users, identity, delegated authority, instructions, retrieved context, Skills, Memory, tools, credentials, code execution, data flows, approvals, other agents, monitoring, and the systems where actions land.

The output should be practical: named threats, attack paths, controls, owners, test cases, residual risk, and a go or no-go decision.

TL;DR

A useful AI agent threat model should:

• Define the agent’s intended outcome and prohibited outcomes.
• Draw the full workflow and every external dependency.
• Mark trust boundaries around users, context, Memory, tools, data, runtimes, and agents.
• Record the agent’s effective authority, including delegated and ambient access.
• List assets that need confidentiality, integrity, availability, or accountability.
• Model malicious users, compromised content, unsafe agent behavior, and control failure.
• Trace attack chains through several otherwise valid steps.
• Include persistent effects such as poisoned Memory and changed configuration.
• Rank threats by impact, feasibility, exposure, autonomy, and reversibility.
• Assign preventive, detective, and response controls to named owners.
• Turn each high-risk threat into a repeatable security test.
• Record residual risk and the person who accepted it.
• Update the model when capabilities, context, integrations, or operating conditions change.

The test is direct: if an attacker controls one input, one integration, one identity, or one agent in the workflow, can the team explain what the attacker could reach and which control would stop each harmful effect?

What AI Agent Threat Modeling Means

Threat modeling is a focused risk assessment for a system. NIST’s draft Guide to Data-Centric System Threat Modeling describes threat modeling as a form of risk assessment that models attack and defense for a selected entity such as data, an application, a host, a system, or an environment.

For an AI agent, the selected entity should be the full path from request to effect:

1. A user, event, schedule, or agent starts work.
2. The agent receives identity, policy, instructions, and task context.
3. It retrieves more data or loads a Skill.
4. A model proposes a plan or action.
5. The agent calls tools, runs code, delegates work, or writes state.
6. Runtime controls allow, narrow, approve, or block the action.
7. A target system creates an effect.
8. The result returns to the agent, logs, Memory, or another system.

Each step can add a threat. Each connection can cross a trust boundary. Each persistent write can affect later sessions.

How Threat Modeling Differs From Related Work

Threat modeling supports a broader security program, but it has its own output.

Practice	Main question	Main output
Threat modeling	How could this system be attacked or fail in a harmful way?	System map, threats, attack paths, controls, tests
Risk management	Which risks matter across the agent portfolio, and how will the organization handle them?	Inventory, risk tiers, treatments, owners, acceptance
Security review	Does the current design and implementation meet security requirements?	Findings and required fixes
Red teaming	Can an adversarial tester break the deployed or test system?	Demonstrated attack paths and evidence
Agent evaluation	Does the agent perform safely and reliably under selected cases?	Test results, scores, regressions, limits
Incident response	How will the team contain, investigate, recover, and learn after failure?	Response plan, evidence, recovery actions

AI agent risk management decides how much review and control a workflow needs. Threat modeling explains where that workflow can break. AI agent evaluation then tests behavior and controls against cases taken from the threat model.

Start With Scope and Prohibited Outcomes

Write one sentence that defines the intended outcome. Keep it specific enough to bound the review.

For example:

The support agent reads one customer’s support history and current policy, drafts a reply, and sends it only after an authorized support lead approves the exact message.

Then list prohibited outcomes:

• Reading another tenant’s data
• Sending a reply without approval
• Changing billing or access
• Exposing internal policy or credentials
• Saving customer data into shared Memory
• Following instructions found inside customer content
• Calling tools outside the support workflow

These statements make the threat model testable. “Secure the support agent” does not define what failure means.

Record what is out of scope too. If identity provider security is covered by a separate review, link that review and state the assumption. Do not silently omit a dependency because another team owns it.

Draw the Agent System and Data Flows

Build a diagram that shows components, connections, data, and effects. A simple box-and-arrow diagram is enough when every box and arrow has a clear label.

Include:

• Users, service accounts, schedules, and event sources
• Agent application and orchestrator
• Models and model gateways
• System prompts and policy sources
• Retrieved documents and search indexes
• Skills and tool definitions
• Short-term session state and persistent Memory
• MCP servers, APIs, browsers, shells, and code runners
• Databases, file stores, message queues, and external services
• Credential brokers and identity providers
• Human approval systems
• Other agents and handoff protocols
• Logs, traces, audit stores, and stop controls

Label each flow with the data type, direction, protocol, identity, and expected protection. “Agent to database” is too vague. “Support agent reads customer-visible case rows through a read-only API using a five-minute task token” gives reviewers something to test.

Draw the effect path as well as the request path. A tool call may return success before an email is delivered, a job runs, or another system consumes the event.

Mark Trust Boundaries

A trust boundary is where identity, ownership, privilege, data handling, or control changes.

Common agent trust boundaries sit between:

• A user and the agent
• Trusted policy and user-controlled text
• Retrieved content and system instructions
• The model and the tool executor
• One tenant and another tenant
• The agent and an MCP server
• A sandbox and its host
• A private network and the internet
• One agent and another agent
• Session state and persistent Memory
• The organization and an external model or data service
• A proposed action and a human approval

For each crossing, ask:

1. How is the sender authenticated?
2. How is authorization checked?
3. Which data may cross?
4. How is integrity checked?
5. Can content on the less-trusted side become instructions?
6. What gets logged?
7. What happens when the control is unavailable?

Do not treat model output as trusted just because the model received a system prompt. Generated plans, tool arguments, code, and summaries need validation at the boundary where they could create an effect.

Inventory Assets and Authority

Assets are the things a threat could expose, change, destroy, misuse, or make unavailable.

Agent assets often include:

• Customer, employee, financial, or health data
• Source code, infrastructure, and deployment systems
• Credentials, tokens, signing keys, and sessions
• Policy, system prompts, Skills, and tool schemas
• Memory, retrieval indexes, and workflow state
• Business records and transactions
• External communications and company identity
• Agent availability, compute, tokens, and spend
• Audit evidence and approval records
• Human time and decision capacity

Next, inventory authority. Record what the agent can actually do, not what its prompt says it should do.

Capture:

• Agent identity
• User or service principal behind the task
• Direct permissions
• Delegated permissions
• Ambient credentials inherited from the runtime
• Tool actions and resource scope
• Data classes and tenants
• Network destinations
• Approval limits
• Spend and rate limits
• Authority passed to subagents

The effective authority may be broader than any one policy record. A coding agent with a read-only repository token can still gain write authority if its shell inherits a developer’s credential helper.

Model the Full Agent Attack Surface

The OWASP Agentic Threats Navigator groups agent attack surfaces around reasoning, Memory, tools, identity, human oversight, and multi-agent interactions. Use those areas as prompts, then adapt them to the actual workflow.

Entry points and goals

List every source that can influence the task:

• User messages
• Emails, tickets, documents, and web pages
• API responses and tool output
• Events and queue messages
• Code comments and repository files
• Retrieved context
• Other agents
• Saved Memory

Ask whether an attacker can change the agent’s goal, add hidden instructions, split an instruction across sources, or make untrusted text look like trusted policy.

Context, Skills, and policy

Map who can create, edit, approve, publish, route, and receive agent inputs. Check for stale versions, wrong tags, overbroad distribution, source confusion, and unauthorized Skill changes.

Threats include a policy that never reaches the agent, a temporary exception that stays active, a Skill that adds a new tool, or retrieved content that overrides team guidance.

Memory and persistent state

Memory makes one session’s input affect later work. Model who can write it, which sources can become Memory, how revisions work, who shares the Memory, and how long entries remain.

The OWASP Top 10 for Agentic Applications treats Memory and context poisoning as a distinct agentic risk because corrupted stored or retrieved information can change later reasoning and actions.

Tools and actions

For each tool, list actions, parameters, target resources, side effects, identity, rate limits, and approval rules. Model valid tool use for an unsafe purpose as well as malformed calls.

An agent may stay inside its permissions and still delete the wrong record, send private data to an allowed recipient, or combine several allowed reads into a prohibited export.

Code and runtime

Map generated code, shells, interpreters, package installation, file access, environment variables, network access, child processes, and sandbox escape paths.

Check what a task inherits from the host. A sandbox with the developer’s home directory, browser profile, or cloud credentials mounted inside has a weak boundary.

Identity and delegation

Track the human, agent, workload, and service identities at every step. Ask whether the system can confuse who requested the work, whose permissions apply, which agent acted, or whether approval covers the exact action.

Data and output

Trace sensitive data into model context, tool arguments, files, logs, Memory, external services, and final output. Model both direct disclosure and transformations such as summaries, encodings, screenshots, or generated artifacts.

Multi-agent communication

Record agent discovery, authentication, message integrity, replay protection, delegation scope, shared state, and failure propagation. OWASP’s Multi-Agentic System Threat Modeling Guide notes that multiple autonomous agents add attack surfaces beyond a single-agent workflow.

Human oversight and operations

Model approval fatigue, missing context for reviewers, forged approval state, emergency access, delayed detection, incomplete logs, and stop controls that fail under load.

Write Threats as Concrete Misuse Cases

Use a consistent sentence form:

An actor uses a condition to perform an action, which causes an impact despite or because of a control gap.

Examples:

• A customer places hidden instructions in a support ticket, causing the agent to include another customer’s data because retrieved content and system policy are not separated.
• A compromised MCP server changes a tool description, causing the agent to send data to a new destination because tool versions are not pinned or reviewed.
• A user approves a draft, then the agent changes the recipient before send because approval is not bound to the final arguments.
• A subagent uses its own broad service identity instead of the delegated task grant, causing a cross-tenant read because authority grows at the handoff.
• A tool timeout triggers a retry after the first request succeeded, causing a duplicate payment because the workflow lacks an idempotency key and effect check.

Concrete misuse cases expose missing assumptions. A label such as “prompt injection” does not say which input is controlled, which action follows, or which asset is harmed.

Trace Multi-Step Attack Paths

Agent failures often need a sequence of valid steps.

Consider this path:

1. An attacker adds instructions to a document the agent is allowed to read.
2. The document asks the agent to retrieve a larger data set for “verification.”
3. The agent uses an approved search tool.
4. It encodes the result inside a support attachment.
5. It sends the attachment through an approved messaging tool.

Each call may pass a narrow allowlist. The sequence violates the intended data flow.

For every high-impact asset, work backward from the harmful outcome:

• Which final action creates the effect?
• Which identity or credential can perform it?
• Which tool or runtime exposes that action?
• Which inputs can influence its arguments?
• Which prior reads supply the data?
• Which controls inspect the sequence?
• Which signal would trigger a stop?

This method finds confused-deputy paths, permission combinations, approval gaps, and harmful action chains that isolated tool reviews miss.

Cover Agent-Specific Threat Categories

Use a threat list as a completeness check after modeling the actual system.

Threat category	Question to ask
Goal hijacking	Can untrusted content replace or bend the assigned goal?
Context injection	Can data become instructions or outrank trusted policy?
Memory poisoning	Can one session corrupt persistent or shared working recall?
Tool misuse	Can an approved tool be used for an unsafe purpose or sequence?
Identity abuse	Can the agent inherit, confuse, or escalate authority?
Unsafe code execution	Can generated code reach the host, secrets, network, or other tenants?
Data exfiltration	Can sensitive data cross an unapproved boundary directly or indirectly?
Supply chain compromise	Can a model, Skill, tool, package, or connector change without review?
Insecure handoff	Can another agent spoof, replay, widen, or lose delegation scope?
Cascading failure	Can one bad result spread through agents, Memory, tools, or workflows?
Resource exhaustion	Can loops, fan-out, retries, or large context consume time or money?
Evidence tampering	Can the agent omit, alter, flood, or bypass audit records?

The list does not replace system mapping. It helps reviewers notice categories they may have missed.

Rank Threats by Real Exposure

Score threats consistently so owners can fix the most important paths first.

Useful factors include:

• Impact on confidentiality, integrity, availability, safety, money, or reputation
• Feasibility of controlling the required input or identity
• Exposure of the entry point
• Agent autonomy
• Effective permission and data scope
• Persistence across sessions
• Number of users or tenants affected
• Detectability
• Reversibility
• Existing control strength

A read-only public research agent and an autonomous production operator may share the same threat label, but their risk differs because authority and impact differ.

Document uncertainty. If the team does not know whether a tool follows redirects or which credential a code runner inherits, mark that as an open finding, assign an owner, and block high-risk deployment until it is tested.

Map Controls to Threats and Evidence

Every material threat needs controls that prevent, detect, and respond.

Control type	Purpose	Example
Prevent	Stop the harmful path before effect	Bind approval to exact tool arguments
Detect	Identify an attempted or completed path	Alert on large reads followed by external writes
Respond	Limit damage and restore safe state	Revoke task grants and stop the session

For each control, record:

• Threats addressed
• Enforcement point
• Owner
• Configuration or policy version
• Expected evidence
• Failure mode
• Test case
• Review date

Prompts can be one preventive control, but important boundaries also need deterministic enforcement. AI agent runtime security should apply decisions at tool gateways, data services, credential brokers, network proxies, sandboxes, and context or Memory services the model cannot bypass.

Use defense in depth for high-impact paths. A data export may require scoped retrieval, field filtering, recipient policy, exact approval, output checks, rate limits, and an audit record. One control failure should not expose the full asset.

Turn Threats Into Security Tests

Each high or medium threat should produce at least one test.

A useful test defines:

• Preconditions
• Attacker-controlled input
• Agent identity and authority
• Expected control decision
• Expected user-visible result
• Expected system effect
• Expected log and alert
• Cleanup or rollback

Test examples include:

• Put an injected instruction in every supported input type.
• Try to use an approved tool with an unapproved resource or purpose.
• Change tool arguments after approval.
• Replay an agent-to-agent message.
• Write untrusted content into persistent Memory.
• Return a poisoned tool result that asks for another action.
• Attempt cross-tenant access through retrieval and direct APIs.
• Cause a timeout after a target system creates an effect.
• Exhaust tool, token, time, or subagent budgets.
• Revoke a credential or policy during a long-running session.
• Disable a decision service and confirm the documented failure mode.
• Stop the agent and verify that in-flight and queued work cannot continue.

Run tests at individual boundaries and through the whole workflow. A tool gateway may block a direct request while a shell, browser, or subagent offers another route.

Decide on Residual Risk and Deployment

Controls reduce risk, but they do not remove every threat.

For each unresolved path, record:

• Remaining likelihood and impact
• Missing or partial control
• Affected assets and users
• Monitoring and response plan
• Compensating control
• Exception owner
• Expiration or review date
• Person authorized to accept the risk

The threat model should support a deployment decision:

• Proceed
• Proceed with limits
• Run only in a test environment
• Require human review for all effects
• Block until named controls pass
• Do not deploy

NIST’s AI Risk Management Framework Core says the Map function should establish system context, identify risks across components, and inform an initial decision about whether an AI system should proceed. Threat modeling gives security and engineering teams a concrete artifact for that decision.

Keep the Threat Model Current

An agent threat model goes stale when the system changes.

Review it when the agent gains:

• A new tool or tool action
• A new data source or data class
• A new model, prompt, Skill, or context source
• Persistent or shared Memory
• A new credential or identity flow
• A new user group or tenant
• More autonomy or a higher approval limit
• Code execution or broader network access
• Another agent or handoff protocol
• A new runtime, environment, or external service

Update it after incidents, repeated policy denials, control failures, major evaluation findings, and changes in attacker methods.

Store the diagram, threats, controls, decisions, and tests with versions. Link the version to deployment and audit records so a reviewer can tell which threat model applied to a given run.

A Practical Threat Model Template

Use this structure for each agent workflow:

Section	Contents
System	Name, owner, version, environment, and review date
Outcome	Intended users, business goal, and prohibited outcomes
Architecture	Components, dependencies, data flows, effects, and trust boundaries
Assets	Data, credentials, systems, policy, state, availability, and evidence
Authority	Agent, user, workload, tool, data, network, and delegation scope
Entry points	Every source that can influence the workflow
Threats	Actor, condition, action, impact, and control gap
Attack paths	Multi-step routes to high-impact outcomes
Controls	Prevention, detection, response, owners, and evidence
Tests	Inputs, expected decisions, effects, logs, and cleanup
Residual risk	Open findings, exceptions, acceptance, and expiry
Decision	Proceed, limit, block, or retire

Avoid a generic threat model copied across every agent. Reuse the structure and threat library, then map the real context, authority, data, tools, and effects for each workflow.

How Alignbase Fits

Alignbase is an AI context control plane. It manages the context, policies, Skills, Memory, permissions, versions, and distribution that shape agent work.

Those records help threat modeling answer:

• Which inputs are trusted policy, and which are task data?
• Who can create, edit, approve, publish, and receive each context asset?
• Which Skills and versions can the agent use?
• Which Memory is assigned, who may write it, and which revision applied?
• Which policy and context reached the agent before an action?
• How can a control change reach every relevant agent?

Alignbase does not replace a threat-modeling method, runtime security layer, identity provider, sandbox, or test harness. It supplies governed agent-input facts and point-in-time evidence that make the model more accurate and the resulting controls easier to maintain.

Connect the threat model to current context and policy versions, then connect runtime decisions and test results back to the same record. That gives reviewers a trace from identified threat to written rule, distributed input, enforced control, and observed result.

The Standard to Aim For

A strong AI agent threat model explains the full path from untrusted input to business effect. It identifies who or what can influence each step, which authority the agent can use, where trust changes, what can persist, which controls apply, and how the team will test those controls.

The model should change engineering decisions. It should remove unnecessary tools, narrow credentials, split trusted policy from untrusted content, constrain Memory, bind approvals to exact actions, add runtime checks, improve logs, and block deployments with unresolved high-impact paths.

If the document only lists common threats, it is a checklist. A threat model connects those threats to this agent, this architecture, this authority, and this effect.