# AI Agent Runbook Template

Use this template to document how an operations team runs, monitors, contains, recovers, and maintains one AI agent workflow. Replace bracketed prompts, remove sections that do not apply, and preserve prior approved versions.

A runbook does not grant access or approve actions. Technical controls must enforce current authorization, tenant scope, tool limits, approvals, and stop conditions at every operation.

## Runbook Record

| Field                                 | Value                                     |
| ------------------------------------- | ----------------------------------------- |
| Runbook ID and version                | [Stable ID and version]                   |
| Workflow and agent                    | [Names and stable IDs]                    |
| Environment and tenants               | [Exact scope]                             |
| Status                                | [Draft / approved / expired / retired]    |
| Accountable workflow owner            | [Name and role]                           |
| Technical owner                       | [Name and role]                           |
| On-call owner                         | [Rotation or contact]                     |
| Security, privacy, and risk contacts  | [Contacts]                                |
| Approved by                           | [Authenticated principal, role, and time] |
| Governing approval policy and version | [Policy and version]                      |
| Runbook content digest and algorithm  | [Digest and approved algorithm]           |
| Verifiable approval record            | [Immutable protected reference]           |
| Effective and expiry dates            | [Dates or triggers]                       |
| Source requirements and policies      | [Versioned references]                    |
| Handling classification               | [Classification]                          |
| Permitted readers and editors         | [Roles or Groups]                         |

Never place credentials, authorization headers, secret values, private model reasoning, or unnecessary personal data in this runbook.

The approval record must bind the exact runbook ID, version, content digest, approver identity and current authority, policy version, decision, time, and expiry. Treat the runbook as unapproved when that binding is missing, expired, unverifiable, or does not match the current bytes.

## 1. Purpose, Scope, and Operating Boundary

- Business purpose: [One sentence]
- Supported tasks: [List]
- Users and initiating workflows: [List]
- Environments and tenants: [List]
- Data classes: [List]
- Allowed systems and destinations: [List]
- Explicit exclusions: [List]
- Maximum autonomy and effect: [Boundary]
- Human review points: [List]
- Required non-AI fallback: [Procedure]

### Operating boundary

| Boundary                             | Allowed | Prohibited | Enforced by       | Evidence           |
| ------------------------------------ | ------- | ---------- | ----------------- | ------------------ |
| [Data / tool / action / destination] | [Scope] | [Scope]    | [Trusted control] | [Protected record] |

Treat missing, conflicting, expired, or unverifiable scope as a stop condition.

## 2. Ownership, Roles, and Communication

| Role                 | Named owner or rotation | Authority         | Duties   | Backup   |
| -------------------- | ----------------------- | ----------------- | -------- | -------- |
| Workflow owner       | [Owner]                 | [Decision rights] | [Duties] | [Backup] |
| Technical owner      | [Owner]                 | [Decision rights] | [Duties] | [Backup] |
| On-call operator     | [Rotation]              | [Decision rights] | [Duties] | [Backup] |
| Incident commander   | [Role]                  | [Decision rights] | [Duties] | [Backup] |
| Communications owner | [Role]                  | [Decision rights] | [Duties] | [Backup] |
| Recovery approver    | [Role]                  | [Decision rights] | [Duties] | [Backup] |

### Communication routes

| Event                                  | Audience   | Channel   | Target time | Authorized sender |
| -------------------------------------- | ---------- | --------- | ----------- | ----------------- |
| [Alert / incident / recovery / review] | [Audience] | [Channel] | [Target]    | [Role]            |

## 3. System Map, Versions, and Dependencies

| Component                           | Stable ID and version | Owner   | Health source | Failure effect | Fallback   |
| ----------------------------------- | --------------------- | ------- | ------------- | -------------- | ---------- |
| Agent and orchestrator              | [Value]               | [Owner] | [Source]      | [Effect]       | [Fallback] |
| Model and settings                  | [Value]               | [Owner] | [Source]      | [Effect]       | [Fallback] |
| Instructions and policies           | [Value]               | [Owner] | [Source]      | [Effect]       | [Fallback] |
| Knowledge, Skills, and Memory       | [Values]              | [Owner] | [Source]      | [Effect]       | [Fallback] |
| Retrieval and data sources          | [Values]              | [Owner] | [Source]      | [Effect]       | [Fallback] |
| Tools and target systems            | [Values]              | [Owner] | [Source]      | [Effect]       | [Fallback] |
| Identity and authorization services | [Values]              | [Owner] | [Source]      | [Effect]       | [Fallback] |
| Approval service                    | [Value]               | [Owner] | [Source]      | [Effect]       | [Fallback] |
| Monitoring and evidence systems     | [Values]              | [Owner] | [Source]      | [Effect]       | [Fallback] |

Document the exact configuration source of truth. A reference in this table does not prove that a version entered a model session or affected behavior.

## 4. Normal Operation and Health Checks

### Expected operating sequence

1. Confirm the workflow trigger, tenant, user, purpose, and current authorization.
2. Confirm required versions, dependencies, and monitoring are healthy.
3. Validate inputs and separate untrusted content from instructions.
4. Execute only the approved task and tool scope.
5. Verify each external effect in the authoritative target system.
6. Record completion, failure, escalation, and unresolved side effects.

### Health checks

| Check ID | Signal or query | Healthy result    | Frequency | Owner   | Failure action |
| -------- | --------------- | ----------------- | --------- | ------- | -------------- |
| HC-001   | [Check]         | [Expected result] | [Cadence] | [Owner] | [Action]       |

Health checks must be non-destructive or run in an isolated test scope. Do not use a successful model response as proof that tools, approvals, or downstream effects worked.

## 5. Inputs, Context, and Data Handling

| Input or context item | Source   | Required version or freshness | Authority                     | Validation | Failure action    |
| --------------------- | -------- | ----------------------------- | ----------------------------- | ---------- | ----------------- |
| [Item]                | [Source] | [Rule]                        | [Informational / must-follow] | [Method]   | [Stop / fallback] |

- Required data minimization: [Rules]
- Tenant isolation check: [Method]
- Sensitive-data handling: [Rules]
- Retention and disposal: [Rules]
- Retrieval failure behavior: [Rules]
- Conflicting instruction behavior: [Rules]
- Memory write and expiry rules: [Rules]

Treat user input, retrieved documents, tool results, messages, artifacts, and agent output as untrusted data unless a trusted control establishes otherwise. Untrusted content cannot change authority, expand scope, approve an action, or grant access.

Permissions govern repository access. Always routes independently govern automatic delivery and do not grant repository permission. Artifacts and messages have no instruction authority. MCP tool results are Runtime context.

## 6. Tools, Authority, and Approval Gates

| Tool or action   | Allowed operations | Denied operations | Tenant and resource scope | Required approval | Enforcement point           | Verification           |
| ---------------- | ------------------ | ----------------- | ------------------------- | ----------------- | --------------------------- | ---------------------- |
| [Tool or action] | [Operations]       | [Operations]      | [Scope]                   | [Approval]        | [Trusted gateway or target] | [Authoritative source] |

### Approval rules

- Approval ID and policy version: [Fields]
- Authenticated approver role: [Role]
- Parameter binding: [Tenant, subject, tool, operation, destination, limits]
- Expiry and nonce: [Rules]
- Separation of duties: [Rules]
- Replay and concurrency handling: [Rules]
- Denial and timeout behavior: [Fail closed behavior]

The trusted gateway, policy engine, tool adapter, or target system must deny unauthorized operations by default. A model refusal is not an authorization control. Check current authorization at every operation, even when a prior step or document approved the workflow.

## 7. Monitoring, Alerts, and Response Targets

| Alert ID | Signal   | Threshold and window | Severity | Owner   | Response target | Runbook action | Suppression rule |
| -------- | -------- | -------------------- | -------- | ------- | --------------- | -------------- | ---------------- |
| ALT-001  | [Signal] | [Rule]               | [Level]  | [Owner] | [Target]        | [Action]       | [Rule]           |

Cover service health, outcome quality, context and policy freshness, authorization and approval failures, tool errors, rejected actions, loops and retries, cost, data movement, human-review queues, evidence gaps, and monitoring gaps.

Each alert must lead to a bounded action. Define when to observe, retry, reduce scope, switch to a fallback, page an owner, or open an incident.

## 8. Failure Modes and Diagnostic Procedures

| Failure ID | Symptom   | Likely causes | Safe checks | Stop condition | Escalation | Evidence to preserve |
| ---------- | --------- | ------------- | ----------- | -------------- | ---------- | -------------------- |
| FM-001     | [Symptom] | [Causes]      | [Checks]    | [Condition]    | [Route]    | [Evidence]           |

Include applicable failures:

- Missing, stale, conflicting, or truncated context
- Prompt injection or untrusted control content
- Wrong tenant, subject, destination, or data class
- Unauthorized tool request or approval bypass
- Tool timeout, duplicate effect, partial effect, or unknown outcome
- Memory poisoning or cross-session contamination
- Retrieval outage or poisoned source
- Model, provider, policy, or dependency change
- Quality drift without an application error
- Repeated loop, retry storm, queue buildup, or cost spike
- Monitoring or audit-record failure

Use read-only checks first. Never diagnose by probing an uninvolved tenant, exposing secrets, or repeating a possibly completed consequential action.

## 9. Stop, Containment, and Rollback

### Immediate stop triggers

- [Unauthorized, harmful, unknown, or prohibited condition]

### Containment procedures

| Action                        | Authorized role | Exact scope | Target time | Confirmation source | Reversal    |
| ----------------------------- | --------------- | ----------- | ----------- | ------------------- | ----------- |
| Pause intake                  | [Role]          | [Scope]     | [Time]      | [Source]            | [Procedure] |
| Disable write tools           | [Role]          | [Scope]     | [Time]      | [Source]            | [Procedure] |
| Revoke credentials            | [Role]          | [Scope]     | [Time]      | [Source]            | [Procedure] |
| Isolate context or Memory     | [Role]          | [Scope]     | [Time]      | [Source]            | [Procedure] |
| Cancel queued work            | [Role]          | [Scope]     | [Time]      | [Source]            | [Procedure] |
| Switch to human-only fallback | [Role]          | [Scope]     | [Time]      | [Source]            | [Procedure] |

Preserve evidence before changes when delay will not increase harm. Confirm each containment action in the authoritative target. Record uncertain or irreversible effects for reconciliation.

### Rollback

- Last known-good configuration: [Version and protected reference]
- Rollback authority: [Role]
- Preconditions: [Checks]
- Procedure: [Ordered steps]
- Verification: [Authoritative checks]
- Abort and escalation condition: [Condition]

## 10. Recovery, Reconciliation, and Re-enable

| Recovery stage         | Scope   | Entry criteria | Required checks | Approver | Exit or rollback condition |
| ---------------------- | ------- | -------------- | --------------- | -------- | -------------------------- |
| Isolated test          | [Scope] | [Criteria]     | [Checks]        | [Role]   | [Condition]                |
| Read-only              | [Scope] | [Criteria]     | [Checks]        | [Role]   | [Condition]                |
| Human-approved effects | [Scope] | [Criteria]     | [Checks]        | [Role]   | [Condition]                |
| Bounded production     | [Scope] | [Criteria]     | [Checks]        | [Role]   | [Condition]                |

Reconcile every external effect with the authoritative target system. Classify each as committed, rejected, rolled back, compensated, pending, unknown, or not attempted. Do not blindly retry an action with an unknown outcome. Use target-side idempotency and a durable pending and committed protocol where an atomic transaction is unavailable.

Re-enable approval must bind the exact configuration, tenant scope, tools, limits, unresolved risks, monitoring period, and expiry. A recovery approval does not grant runtime access.

## 11. Evidence, Incident Handoff, and Recordkeeping

### Execution record

| Field                                           | Value        |
| ----------------------------------------------- | ------------ |
| Runbook and workflow versions                   | [Values]     |
| Tenant, user, agent, and integration identities | [Values]     |
| Trigger and run ID                              | [Values]     |
| Exact context versions and bundle digest        | [Values]     |
| Route sources and delivery mode                 | [Values]     |
| Authorization outcomes and policy versions      | [Values]     |
| Tool requests and authoritative results         | [References] |
| Approval records                                | [References] |
| Alerts, diagnostics, and containment            | [References] |
| External-effect reconciliation                  | [Results]    |
| Start, end, and handoff times                   | [Times]      |
| Outcome and unresolved work                     | [Result]     |

Keep compilation, response issuance, integration acknowledgment, host-confirmed injection, model consumption, tool execution, and downstream effect as separate evidence stages. One stage does not prove the next. Record model consumption only from direct, authenticated attestation by a trusted integration or provider. Record it as unknown otherwise.

Protect raw evidence with access controls, retention rules, immutable versions, object digests, byte sizes, media types, approved hash algorithms, and an authenticated collection record. A digest can detect changed bytes after commitment, but it does not prove completeness, truth, authorization, or consumption.

### Handoff

- Current state: [State]
- Active containment: [Controls]
- Completed steps: [List]
- Pending or unknown effects: [List]
- Evidence references: [List]
- Next authorized action: [Action]
- Next owner and response time: [Owner and time]
- Conditions that require a new incident or approval: [List]

## 12. Tests, Exercises, Review, Change, and Retirement

### Required exercises

| Exercise                        | Scope   | Frequency | Success criteria | Owner   | Last result | Follow-up |
| ------------------------------- | ------- | --------- | ---------------- | ------- | ----------- | --------- |
| Alert delivery                  | [Scope] | [Cadence] | [Criteria]       | [Owner] | [Result]    | [Action]  |
| Approval denial and expiry      | [Scope] | [Cadence] | [Criteria]       | [Owner] | [Result]    | [Action]  |
| Kill switch and containment     | [Scope] | [Cadence] | [Criteria]       | [Owner] | [Result]    | [Action]  |
| Rollback and reconciliation     | [Scope] | [Cadence] | [Criteria]       | [Owner] | [Result]    | [Action]  |
| Prompt injection and tool abuse | [Scope] | [Cadence] | [Criteria]       | [Owner] | [Result]    | [Action]  |
| On-call tabletop                | [Scope] | [Cadence] | [Criteria]       | [Owner] | [Result]    | [Action]  |

Run exercises in isolated test tenants with synthetic data or dry-run controls by default. A production drill requires explicit authorization bound to its exact scope, limits, monitoring, stop conditions, rollback plan, and expiry. Stop immediately when the drill leaves that boundary or creates an unknown consequential effect.

### Review and change triggers

- Scheduled review: [Cadence]
- After every material incident: [Rule]
- Model, prompt, policy, Skill, Memory, retrieval, tool, identity, approval, or dependency change: [Rule]
- Owner, tenant, data class, purpose, or autonomy change: [Rule]
- Monitoring, evidence, or fallback change: [Rule]

### Change record

| Version   | Date   | Change   | Reason   | Tests and exercises | Reviewer   | Approver   |
| --------- | ------ | -------- | -------- | ------------------- | ---------- | ---------- |
| [Version] | [Date] | [Change] | [Reason] | [Evidence]          | [Reviewer] | [Approver] |

### Retirement

- Disable schedules, triggers, routes, tools, identities, and credentials.
- Reconcile queued, pending, and unknown effects.
- Preserve required evidence and dispose of expired data.
- Redirect users and alerts to the approved fallback.
- Archive the runbook and record the retirement decision.