Goal Hijacking and Memory Poisoning Mechanics in Production Agents

Goal hijacking and memory poisoning exploit the gap between an agent’s instructions and the data it reads. Their danger is not just a bad response, but a redirected decision path that can extend across tool calls and sessions. To secure production agents, controls must be in place where content gains authority and where that authority leads to action, and must be tested against how the attack works.

Why agents cannot separate instructions from data, and why that makes them structurally exploitable

Traditional software treats executable logic and user input differently. Language models receive both developer instructions and attacker-controlled content as natural-language context, without an architectural boundary that reliably enforces the distinction. Crafted input can therefore cross from evidence into instruction. OWASP ranked prompt injection as LLM01:2025 in its Top 10 for LLM Applications, reflecting a structural characteristic.

Agentic systems turn that ambiguity into operational exposure. An agent may hold credentials, invoke APIs, write to databases, and execute code, so the consequences scale with its autonomy and tool access. The model’s interpretation can determine which permitted action happens next. An attacker need not change the software to influence that choice. The relevant attack mechanics concern how this gap is exploited during planning, retrieval, memory storage, and interaction with tools.

How goal hijacking works: redirecting an agent's objective without touching its code

Goal hijacking changes what an agent believes it should achieve. Because the agent derives instructions dynamically from context, attacker-controlled material can redirect its objective while leaving the execution machinery intact. A compromised decision path can produce a response that appears benign. Each step may use valid credentials, interfaces that have been approved, and API calls that are allowed.

The distinction is between permission and purpose. Authentication establishes who the agent is, while authorization establishes what it may call. Neither establishes whether the reason for making a call still follows the user’s objective. API telemetry and access logs can therefore show calls that were allowed without showing how the agent’s reading of the context made them harmful. Goal hijacking can persist through a task chain even when no individual call violates an access rule.

A hijacked sub-agent can sway downstream agents, change shared state, or shape the decisions an orchestrator makes. The deeper the pipeline, the further the redirected objective can reach. Unlike a one-shot jailbreak, this is behavioral drift across a sequence of decisions.

How memory poisoning works: persisting malicious instructions across sessions

Goal hijacking redirects an objective within a session or task chain, while memory poisoning corrupts the stored material from which future decisions are made. The malicious instruction survives in the agent’s own memory and can return as trusted history. Session closure no longer ends the exposure.

The reflected-versus-stored XSS analogy clarifies this temporal difference. Within-session injection is like reflected XSS because its immediate effect ends with the interaction, while poisoned memory is like stored XSS because the payload stays available for future triggers. Consolidation can make that payload harder to distinguish from legitimate history. When an agent rewrites an external observation as a remembered user instruction or system fact, the transformation launders its provenance.

A support-ticket instruction illustrates the delayed consequence. If an agent stores an attacker’s direction that vendor invoices from Account X should route to external payment address Y, a legitimate invoice three weeks later can trigger the planted rule. This is a hypothetical payment scenario. AgentPoison, developed by Chen et al. in 2024, demonstrated the underlying mechanism by poisoning memory or a knowledge base to influence future reasoning and planning. It achieved an 82% retrieval attack success rate, with substantial end-to-end target impact rates.

The stages at which poisoning actually enters an agent: retrieval, consolidation, and tool outputs

An agent’s operating loop connects an LLM core, memory, tools, and the environment. Retrieval brings stored or external material into the context used for reasoning and planning. Poisoned material can enter that context as evidence that seems relevant, then shape the objective or which tool the agent chooses. The distinction between relevance and authority explains why successful retrieval can also deliver an attack.

A system that summarizes or compresses observations before persistence introduces a rewriting stage that attackers can target. An untrusted document, email, or tool result may be rewritten as user history or system fact. The original authority boundary is then lost. MemPoison describes attacks on selective extraction and rewriting, showing why memory security cannot be reduced to whether an attacker can write directly into a store.

Deng et al.’s 2026 survey lists tool outputs as a distinct injection vector because agents may take the values tools return as environmental ground truth. A compromised API or manipulated response can consequently supply instructions under the appearance of factual output. The survey’s component-aligned taxonomy distinguishes memory, tool, environment, and agent-interaction threats. These are separate entry points whose effects can propagate through the same reasoning process.

What silent, multi-stage compromise looks like in practice: documented cases

EchoLeak, CVE-2025-32711, demonstrated prompt-injection-driven exfiltration of sensitive organizational data through Microsoft 365 Copilot without user interaction. Its CVSS score was 9.3. The absence of user interaction distinguishes the case from attacks that depend on persuading someone to approve each harmful step. Imprompter, developed by Fu et al. in 2024, demonstrated another route to exfiltration through automatically generated, obfuscated adversarial prompts. Production agents including Mistral LeChat and ChatGLM were coerced into tool misuse for PII exfiltration at high precision, and the attack transferred from open-weight to closed-weight models without access to target weights.

ToolHijacker, developed by Shi et al., targeted the retrieval-and-selection pipeline without requiring model access. Its no-box attack achieved very high success rates against GPT-4o on the MetaTool benchmark. It bypassed prevention defenses StruQ and SecAlign. It also bypassed known-answer detection, DataSentinel, perplexity detection, and perplexity windowed detection. Separately, black-box red-teaming of 14 production conversational agents in the Spain Living Lab smart tourism ecosystem found a critical attack success rate of 55.6% in text-accessible interfaces.

Habler and Amy Chang documented the identification and remediation of a persistent memory compromise in Claude Code in 2026. Brodt et al.’s 2026 Promptware Kill Chain framework treats multi-step attacks as malware that runs in natural-language space. Its unit of analysis is the attack sequence.

Why existing detection methods miss these attacks and what evaluation frameworks reveal about the gap

A detector may examine an incoming message without seeing how it later affects consolidation, retrieval, or a consequential tool call. Session-level monitoring extends the view but still excludes cross-session persistence. With poisoned memory, the incident may have begun before the agent was deployed. An investigation confined to the observed harmful action can therefore miss the event that supplied its justification.

Permission checks leave a different blind spot. An API gateway sees a permitted call, while goal hijacking resides in the contextual choice to make it. Anomaly detection may flag unusual behavior, but a plausible chain of authorized actions can look normal. ToolHijacker’s results also show that having familiar prevention and detection mechanisms in place is no proof that they can withstand the attack.

Evaluation scope imposes a further limit on confidence. AgentDojo, developed by Debenedetti et al., evaluates prompt injection against tool-calling agents across a substantial range of tasks and security tests. It models attacks as data returned by tools during a single task trajectory. It provides no methodology for evaluating cross-session persistence. The OWASP framework likewise focuses primarily on within-session attacks rather than supplying a methodology for long-horizon or sleeper-attack evaluation.

A 2026 threat model-driven test framework organized threats through STRIDE and LINDDUN and contextualized results against the OWASP threat model for Agentic AI and the Model Context Protocol threat model. Its category-based grouping left threat persistence unaddressed. A lifecycle survey of long-term memory security concluded that defenses “cannot be retrofitted at retrieval or execution time alone”. It placed storage-time provenance, versioning, and policy-aware retention at the foundation of memory security.

Controls that address the actual failure mode: provenance at storage, enforcement at runtime

Storage controls must keep track of what gives remembered content its authority. Agent memory needs provenance, scope, integrity, expiration, authorization, and sensitivity classification alongside vector similarity. Those properties distinguish a relevant entry from one entitled to justify an action. Without them, retrieval cannot reliably separate poisoned memory from legitimate history. Versioning and policy-aware retention extend that distinction through the memory’s lifecycle.

Xu et al.’s 2026 Provenance-Preserving Memory Firewall proposes middleware that preserves platform-maintained provenance and authorizes tool calls by matching action risk to the authority of action-relevant memories. The action inherits the trust constraints of its justification, countering provenance laundering. Costa et al.’s 2025 information-flow control proposal extends the principle across the decision path. It tracks which untrusted sources influenced decisions and blocks high-risk consequential actions when that influence is present.

Runtime controls address the objective as execution unfolds. Meta’s open-source LlamaFirewall, released in 2025, includes an experimental AlignmentCheck component that compares an agent’s action sequence with the user’s stated objective. It reasons over the entire execution trace. Deviations can indicate covert prompt injection, misleading tool output, or goal hijacking. Attacks succeeded at a fraction of their undefended rate with it active.

These mechanisms govern different parts of the failure. Provenance constrains what remembered material may authorize, while intent verification examines whether the sequence still serves the approved purpose. Least-privilege tool enforcement complements both by limiting what a hijacked agent can accomplish, even when poisoning succeeds. No complete fix exists, and frontier models from OpenAI, Google, and Anthropic remain vulnerable after their best defenses are applied. Layering storage-time provenance, runtime intent verification, and least-privilege enforcement is therefore a minimum architecture.

What practitioners need to treat differently when securing agents against these specific attack classes

For practitioners, the operational shift is to govern context and memory as live attack surfaces with lifecycle requirements. Durable memory should be separated from ephemeral task context, with integrity controls applied to any store the agent can learn from. Authorization must also constrain both purpose and interface access. Binding actions to a constrained intent model and reviewing whether delegated permissions can serve unrelated objectives addresses the gap goal hijacking exploits. IAM and security teams consequently need to govern what an agent can become at runtime.

Monitoring must connect identity and permission to behavior. Relevant signals include mismatches across identity, data, model behavior, posture, and environment, particularly access to unrelated data, injected instructions taking effect, or drift from a known configuration. When compromise is detected, containment should treat it as a delegated privilege failure. The response should contain the agent before further API calls, revoke or narrow affected tool permissions, preserve call-chain logs, and examine whether the context store or instruction layer was modified.

A deployment’s risk assessment should establish whether the specific attack succeeds under tested conditions, rather than conclude that the agent is probably safe because a recognized defense is present. Evaluation must include the attack’s relevant time horizon and the workflow through which poisoned context becomes action. The work is not complete until the control has been tested against the real attack in the real workflow.