Benchmarking Prompt Injection Defenses in Production Agents

Benchmarking prompt injection defenses in production agents takes more than a test of whether a model rejects malicious instructions. It requires evidence that a specific control stops a proven attack in the workflow where it lands, while approved behavior still passes. That standard changes the unit of evaluation from an isolated prompt to the full workflow, including the trust boundaries an attack can cross.

Why prompt injection benchmarks built for chatbots fail when applied to agents

A standalone LLM and an agent expose different consequences of the same injection. In a standalone interaction, the attacker gets a bad output in an exchange that has a clear end. An agent can turn that instruction into data exfiltration, code execution, or lateral movement across tools. The failure can outlive the original exchange. A benchmark that observes only the response therefore misses the actions that follow it.

OWASP’s State of Agentic AI Security traces this structural risk through researcher Simon Willison’s “lethal trifecta” framing, which combines access to private data, exposure to untrusted content, and the ability to act externally. Agents encounter that content through user requests, retrieved documents, memory files, tool outputs, and inter-agent messages. The user-input channel is only one entrance. Indirect injection, the most common form in production, can come through material retrieved during an otherwise legitimate task without ever touching that channel. Persistent memory adds a further mismatch because a payload encountered in one run can influence later sessions beyond a single-session benchmark’s view.

What the production attack surface actually looks like across an agent's full workflow

The production surface spans user input, retrieved web and document content, persistent memory files, tool call arguments, API responses, inter-agent messages, and trusted supply-chain artifacts. These are entry points and propagation paths. Content entering through retrieval can influence a tool argument, while an agent’s output can become another agent’s input. An injection that reaches shared memory can also affect later sessions or redirect orchestrator decisions.

The LiteLLM incident in March 2026 illustrates the upstream supply-chain exposure. A backdoored package sat on PyPI for three hours, deploying a three-stage credential harvester and Kubernetes worm. The compromise preceded any runtime prompt check. In that incident, hackerbot-claw was the attacker’s upstream tool used to compromise Trivy, separate from the package payload. This distinction places the dependency compromise and its delivered payload at separate points in the chain.

Coding agents have the most densely packed risks because execution is part of their workflow. CVE-2026-22708 against Cursor allowed an attacker to poison the execution environment through shell built-ins such as export that bypassed the allowlist. Subsequently allowlisted commands such as git branch could then deliver arbitrary payloads. The allowlist actively aided the attack. The University of Washington’s Bad Memory study documents a different propagation path, finding that attack success and payload persistence vary substantially across systems, models, adversarial goals, and multi-session sequences.

How existing benchmarks cover this surface — and where they stop

AgentDojo, in the NeurIPS Datasets & Benchmarks track, provides dynamic prompt injection evaluation in agent environments, while another benchmark evaluates indirect prompt injection in tool-integrated agents. AgentDrift provides step-labeled injection-hijacked trajectories, and PIArena, at ACL 2026, provides a platform for prompt injection evaluation. AgentLAB addresses long-horizon attacks, and GuardianAgentBench is another active benchmark. These tests go beyond general instruction-following, but none follows the full cascading workflow of a production agent.

Attempt counts and persistence expose further limits. Anthropic’s Claude Opus 4.5 system card documents indirect prompt-injection attack success rising sharply with additional attempts, so a single run captures only part of exploitability. Bad Memory evaluates persistent-memory injection in Claude Code and OpenAI Codex across Claude Haiku 4.5, Claude Opus 4.7, GPT-5.2, and GPT-5.5 in a sandboxed synthetic workspace. Its findings vary a great deal with the sessions, systems, and adversarial goals. A cross-sectional result cannot show all that happens through those sequences.

Controlled environments nevertheless serve an essential purpose. They support reproducible scientific comparison, while adding full-workflow complexity can make results harder to interpret. That is a valid limit on benchmark design within which these tools remain useful. The gap calls for a complementary production evaluation layer that answers a narrower operational question about a particular agent, control, and workflow.

Why prompt-level guardrails cannot be the primary defense validation target

Prompt-level guardrails operate at the same abstraction layer as the instructions they attempt to reject. Parallax argues that, when the reasoning system is compromised, guardrails inside that system provide zero protection because the compromised component also interprets the rules. An external architectural boundary has a different role. Its enforcement does not depend on the reasoning system continuing to obey a safety instruction.

The Attacker Moves Second, a study of adaptive attacks against jailbreak and prompt-injection defenses, shows that indirect prompt-injection defenses can be systematically bypassed when an attacker knows the defense type and optimizes against it. Passing a fixed suite therefore establishes a result at a point in time, while its stability under adaptation remains unproven. Cursor’s allowlist failure adds a related warning about permission checks. A guardrail can participate in the exploit when its assumptions do not hold across the execution workflow.

General resistance to malicious text does not establish that an execution boundary will hold after reasoning has been redirected. Rejecting an attack also does not show that legitimate work can still get done. Validation must establish whether this control stops this proven attack without breaking this workflow.

What argument-level and architectural controls change about what can be measured

A tool invocation can be authorized while one of its arguments contains an unauthorized payload. Invocation-level monitoring treats the call as permitted and misses that distinction. Argument-level monitoring makes the boundary more precise. It allows evaluation to distinguish authorization to use a tool from authorization to submit particular data through it, producing an auditable outcome at the point where trust changes.

PACT reports 100% security on the three strongest models in full AgentDojo deployments across five models, while bringing utility back to a level well above CaMeL at the same security level. Parallax examines a different boundary through Cognitive-Executive Separation, Adversarial Validation with Graduated Determinism, Information Flow Control, and Reversible Execution. Its Assume-Compromise Evaluation bypasses the reasoning system entirely across 280 adversarial test cases in nine attack categories. The test checks whether the architecture holds when the agent is fully compromised, with no need for the model to avoid compromise.

This precision comes with costs and limits. Fine-grained provenance tracking increases storage cost, logging complexity, privacy exposure, and annotation burden. Coarser provenance is easier to collect but can miss the argument that crossed a trust boundary. A benchmark must therefore state the granularity at which its claims hold. The useful evidence is an auditable record of what was enforced, against which input, and at which boundary, rather than an aggregate score detached from those conditions.

The methodology shift: from "does the defense hold?" to "does this control stop this proven attack without breaking this workflow?"

For production evaluation, this evidence standard leads to three requirements. First, the attack must be proven by an injection payload succeeding against an isolated replica of the real agent in its actual workflow context before the defense is evaluated. A synthetic task does not show that the attack works in that workflow. If the attack did not succeed before the change, its failure afterward cannot show that the control stopped it.

Second, the control must be the smallest change that stops the attack. Overly broad restrictions can block the exploit while also breaking approved workflows and generating false positives that erode trust. Security and utility must be tested together. The benchmark should check that the attack no longer succeeds and that every workflow that passed before still passes after the control is applied.

Third, the test must use realistic attempt counts. The system card’s finding about repeated attempts means that one attempt does not represent the risk posed by an attacker who can keep trying. The attempt budget must accompany the result. ISACA’s September 2026 white paper supports integrating agents into the secure SDLC through threat modeling across user input, retrieval, memory, inference, tool calls, APIs, human approvals, output delivery, tenant boundaries, and privileged actions. Those components define the workflow context that the evaluation must exercise.

The resulting evidence forms a record of the verified change. The evidence record shows the baseline attack that succeeded, the control that was applied, the attack result after the control, and the approved-workflow results. Each step is open to inspection and reproduction. A release approver can then assess the causal claim that the change stopped the exploit without removing the behavior the agent was approved to perform.

How the multi-session and multi-agent dimensions change what workflow verification must cover

For memory-touching changes, workflow verification must extend beyond the session in which the control is applied. Bad Memory shows that payloads planted in trusted files such as CLAUDE.md, AGENTS.md, and knowledge-base documents can influence future sessions. Blocking a new injection is not sufficient. A payload already in memory may still act after the original injection session ends, so the test must tell apart keeping it out and stopping its effects later.

After changes to memory read or write paths, approved-workflow replay must therefore run across more than one session. The defense must hold across the session boundary. This adds a temporal requirement to the evidence record, because successful behavior in the first run says little about what a later run inherits. The replay must establish both that the attack is stopped and that approved behavior remains available through the tested sequence.

When one agent’s output becomes another’s input, goal hijacking can steer peer agents, poison shared memory, and redirect the orchestrator. The original recipient is not the only target. The OWASP AIUC-1 Crosswalk from May 2026 names inter-agent communication security as one of eight priority governance gaps where requirements need to cover more. Workflow verification must therefore test multi-agent sequences that reflect how the agents work and check the control again at every boundary where trust changes hands. A result at the first agent cannot stand in for evidence about the rest of the pipeline.

Calibrating what a defense score actually means — and what it cannot claim

A defense score is a claim with a defined scope. Validation against one injection surface and workflow set says nothing about surfaces or workflows that were not tested. The adaptive-attack study shows why a defense may fail when an attacker knows it and tunes attacks against it, so a benchmark result must be described as a snapshot that gives no guarantee.

The report should state the attack class, injection surface, attempt count, verified workflow set, and post-control workflow pass rate. It should also name the surfaces not exercised, including memory persistence, inter-agent boundaries, supply-chain inputs, or argument-layer provenance where applicable. Omissions must remain visible to the release approver. Otherwise, an aggregate score can make the tests seem to cover more than they did.

A reviewer must inspect and sign the evidence record before the change ships, using its reproducibility to assess whether the claim follows from the results. The full suite should run before every release as well as after a new CVE.

Reproducible evidence with named blind spots is more useful than a summary score with hidden assumptions.