Prompt Injection vs Jailbreak Attacks on AI Agents

Prompt injection and jailbreak attacks on AI agents require different controls because they differ in adversary position and objective. Jailbreaks seek to bypass what a model will say or authorize, while prompt injection seeks to hijack what an agent does. Treating them as one guardrail problem leaves gaps in how content safety, instruction trust, and tool execution work together.

Why prompt injection and jailbreaks are different problems, not one guardrail problem

The shared symptom is that the model does something it should not. That resemblance obscures two different failures of control. In a jailbreak, the adversary is the user, directly crafting input to bypass safety alignment and obtain policy-violating content. In indirect prompt injection, the adversary is outside the conversation and plants instructions in material the agent retrieves while helping a legitimate user.

The difference determines where a defense can intervene. A refusal control addresses whether the model should comply with a request, but it does not establish whether a retrieved document has authority to make that request. Finding untrusted external content also does not tell the model whether a direct user request is safe. Controls designed for one problem can leave the other unaddressed.

How the architecture creates the exposure for both attacks

LLM applications intermix instructions and data in a single context window without cryptographically enforced separation between them. The model resolves conflicts through learned heuristics, with no hard constraints enforcing those decisions. This allows attackers to misuse the model’s authority in both kinds of attack. The model must interpret text while also deciding which parts of that text have authority over its behavior.

The two attacks exploit that ambiguity through different paths. A jailbreak engineers user-side input to overcome the model’s learned refusal behavior. Prompt injection engineers external data to override the system prompt or the legitimate user’s intent, promoting content into instructions. A control at the user-input boundary therefore does not cover an attack arriving through retrieved material.

What jailbreaks actually target in an agentic context

Jailbreaks target safety alignment, the learned constraints on what a model will output. In a chatbot, the classic objective is a response the model is trained to refuse, including harmful instructions or other policy-violating content. The harmful artifact is the answer itself. Once tools are connected, however, the same bypass can authorize file deletion, credential exfiltration, or an API call to an external system. The consequence changes from producing content to directing execution.

Florida International University’s SoK: Rethinking Jailbreaking in the Era of Agentic AI finds that strong native alignment does not imply robustness against adversarial jailbreaks. Automated and optimization-based attacks can get past well-aligned models with carefully built prompts. Planning, memory, and tool interactions may remain unsafe even when the answer presented to the user is filtered. The adversary still operates from the user position, but that limited entry point does not limit the consequences once the model can act.

What prompt injection actually targets — and why the adversary position changes everything

Indirect prompt injection requires no adversarial access to the user prompt. Instead, instructions arrive through web pages, emails, documents, database records, API responses, or other external material. The attacker operates in the agent’s environment. Direct injection enters through user-controlled input, just as jailbreaking does, but indirect injection can reach the agent through anything it reads during an otherwise legitimate task.

The Model Context Protocol, now a dominant connector between agents and tools, extends that exposure through tool descriptions, server responses, and metadata. Each server supplies content that can reach the model and contain instructions it acts on. The attacker aims for goal hijacking, which need not involve getting prohibited text. This can mean compromising the agent’s own behavior, an attack on the agent, or using it to reach downstream databases, APIs, and infrastructure, an attack through the agent.

Gray Swan’s large-scale red-teaming competition identified concealment as a critical, underexplored dimension of this threat. A compromised agent may give believable reasons for actions the attacker directs while giving the user a normal-looking final response. Successful compromise need not announce itself at all. Because tool-execution logs are too verbose for effective human inspection in practice, neither a reassuring answer nor the mere availability of logs establishes that the workflow remained authorized.

Production evidence: what each attack class has actually achieved

Aim Security disclosed EchoLeak, CVE-2025-32711, in June 2025 as the first known prompt injection weaponized for concrete data exfiltration in a production AI system. A single crafted email enabled remote, unauthenticated exfiltration from Microsoft 365 Copilot. No user interaction was required for the attack.

EchoLeak crossed several defenses in sequence. It evaded Microsoft’s XPIA classifier, circumvented link redaction with reference-style Markdown, exploited automatically fetched images, and abused a Microsoft Teams proxy permitted by the content security policy. The attacker’s position remained entirely outside the conversation. The chain illustrates why separate protective features do not establish a secure boundary when adversarial content can exploit a path through them.

Cursor IDE’s CVE-2025-54135 and related vulnerabilities illustrate the transition from instruction influence to execution. Indirect injection made the agent create a malicious .cursor/mcp.json configuration without the user’s approval, which allowed remote code execution through that configuration. Microsoft’s research into agent frameworks documented the same threshold with Semantic Kernel CVE-2026-26030 and CVE-2026-25592, where influence over tool parameters becomes an execution primitive.

Other cases show objectives beyond that execution path. Salesforce Agentforce’s ForcedLeak, disclosed in September 2025 and rated CVSS 9.4, used hidden instructions in a web-to-lead form to send CRM data to an allowlisted but expired domain purchased by attackers. A Google Gemini Enterprise injection in a Jira task description triggered a silent memory wipe without confirmation and earned a Google AI VRP bug bounty. State corruption was the objective in that case. OpenClaw/ClawHavoc supply-chain poisoning in late January–February 2026 put malicious skills into the agent ecosystem and compromised thousands of developer machines, giving the attacker a place in the tool registry.

Devin AI provides a useful qualification to this separation. A researcher purchased access to the autonomous coding agent, then placed carefully crafted indirect injection payloads on external sites and repositories it visited. The agent exposed ports, leaked access tokens, and installed command-and-control malware. This shows how an adversary holding the user position can combine that access with environmental payloads to produce severe agentic consequences.

Why the attacks compound when injection and jailbreak objectives combine

An injection can introduce a new goal that requires an action the agent would normally refuse. First, external content gains authority over the task. Then the redirected goal pressures or bypasses alignment so that the agent can complete it, producing a jailbreak-equivalent outcome. These are distinct failures in one execution sequence. Collapsing them into a single category hides the point at which each boundary failed.

In multi-agent systems, that sequence can spread from one system to another. A compromised orchestrator can pass injected instructions to subordinate agents, creating a cascade through inter-agent trust. Input provenance controls establish where instructions originate, alignment and refusal controls constrain what the model will authorize, and tool-call enforcement constrains what reaches execution. Combining attack objectives makes those separate checks more necessary as distinct controls.

What the empirical robustness record shows about current defenses

Gray Swan’s competition found a weak relationship between capability and robustness, including a highly capable model that was also highly vulnerable. Universal attack strategies worked across 21 of 41 tested scenarios and more than one model family. Greater capability was not a dependable security signal. Transfer across scenarios also points to weaknesses beyond isolated prompts or individual model implementations.

The FIU SoK finds that jailbreak-defense effectiveness depends heavily on the model, attack, and agent component. Defenses can impose costs in over-refusal, utility, and latency. Counting how much attacks drop leaves out those costs. A defense may add little to what native alignment already does, or keep the final answer safe while leaving the steps before it open to attack. Tests must tell these outcomes apart and check the agent’s safety beyond the response it gives.

AgentDojo evaluates 97 user tasks and 629 security cases across banking, Slack, travel, and workspace domains, measuring benign utility, utility under attack, and attack success. Agent Security Bench finds that current defenses have limited success, with its highest average attack success rate still far above what would be acceptable in practice. MCP-SafetyBench tests 20 attack types across 5 domains and finds that host-side attacks succeed in a majority of attempts on average. The WASP benchmark characterizes web-agent security as largely “security by incompetence,” with defenses failing to stop attacks that fail because agents cannot complete them. Failure to execute is not a durable boundary.

The controls that fit each threat class, applied at the right layer

For defenders, indirect injection calls first for controls at the boundary between trusted instructions and untrusted content. Provenance must show where each of the tool-call arguments came from. A legitimate tool can become unsafe when its parameters derive from attacker-controlled values outside the user’s authorized task. The relevant execution record includes the selected tool, schema, argument values, execution result, and downstream consumers, because authority must remain traceable through the workflow.

Agent-Sentry demonstrates this approach by compiling a typed authorization blueprint before execution and enforcing it through a deterministic monitor. Its provenance-aware authorization distinguishes user-authorized values from untrusted environmental observations. On AgentDojo, it reduces attack success to near zero with only a small clean-utility drop. That result is specific to the evaluated setting. Input isolation and sandboxed agentic interfaces address a complementary need by limiting how far injected instructions can propagate when initial detection fails.

For direct jailbreaks, refusal classifiers and safety tuning address the correct boundary because the adversary controls user input. Their value should be measured as the marginal gain over native alignment, including utility and latency costs. Planning, memory, and tool interactions each need their own checks, while execution-aware defenses track intermediate state and the moves between components as well as the answer returned to the user.

Compounding attacks require enforcement at both instruction intake and tool-call parameters. A single classifier at either point leaves the other boundary dependent on model behavior. Human review remains a necessary backstop as well. Because concealment makes a quick look at the output unreliable, review should include approved-workflow verification to check that the expected tool-call sequences still finish correctly after a security change. This tests protection without assuming that a blocked attack also means the legitimate workflow still works.

What the distinction means for teams shipping agents into production

Teams should diagnose exposure by adversary position first, establishing whether each threat originates inside the user session or in the environment. The corresponding control layer must exist and face real attack attempts. The next diagnostic is the attacker’s objective. Content-policy bypass and goal hijacking need defenses where those goals take shape, as well as where the final response appears. This exposes gaps such as strong alignment without argument-level enforcement, or direct-input detection without provenance tracking for retrieved content.

Workflow verification should then establish whether approved tool-call sequences complete correctly and unexpected invocations are surfaced. A normal-looking answer is not sufficient evidence. MCP-connected systems need the same close checks of server-side content and metadata because these inputs are part of the agent’s attack surface. Production readiness rests on reproducible attack results and verified fixes that preserve approved workflows.