Human-in-the-Loop Controls for High-Stakes Agent Actions

An AI chatbot that gets manipulated by a hidden instruction produces bad text. An AI agent that gets manipulated by the same instruction executes a real action, using credentials it already holds. That difference is the entire subject of this piece: human-in-the-loop controls for high-stakes agent actions aren't a single switch a team flips on, but a layered decision about which actions need a person to sign off, what that person needs to know to decide well, and whether the checkpoint actually holds when something tries to get around it.

Why agent autonomy makes the pause-and-ask question unavoidable

For years, the worst-case output of a language model was a bad sentence. Agents change that calculus because they don't just answer, they act. An agent holding a set of credentials can send the email, move the file, update the database row, or push the deploy, and it does so with the same service-level permissions a trusted internal system would use. A single bad decision becomes a real-world event, often one that triggers other systems downstream.

This shift opens up risk categories that simply did not exist in the old request-response model, where a user asked a question and the model answered it. Fiddler's blog post on agent security names six distinct attack surface expansions that appear once a model gains tool access, persistent memory across sessions, and the ability to run decision loops without a person checking in at each step. Multi-agent delegation adds a further wrinkle: one agent's tainted output becomes another agent's trusted input, so a compromise doesn't stay contained to the system where it started.

None of this means autonomy is a mistake. The compounding is what raises the stakes, not any individual step in isolation, and that is why the question of when to pause and ask a human cannot be treated as optional or deferred to later. It has to be decided before the agent is given the keys.

What "high-stakes" means in an agent action taxonomy

Blocking every agent action pending human review, or approving all of them automatically, defeats the purpose of building an agent. An agent that asks permission for every API call isn't autonomous, and an agent that asks for none of it isn't governed. Two properties of an action actually warrant a pause, and they do most of the work: reversibility and blast radius.

Reversibility asks whether a wrong decision can be corrected after the fact. Blast radius asks a separate question: how many systems or people does this action touch, even if it could technically be reversed? A config change that's trivial to revert but affects every user of a shared service carries real risk regardless of how easy the rollback is.

Singapore's IMDA Model AI Governance Framework builds its trigger logic around four categories that map cleanly onto this reasoning: high-stakes actions and decisions, irreversible actions, outlier behavior, and user-defined boundaries. The framework doesn't replace the reversibility and blast-radius logic, it confirms that regulators are converging on the same two axes that security engineers would reach by just thinking through what can go wrong.

The gap between having this taxonomy and actually using it remains wide. Nearly two-thirds name security as the top barrier standing between their organization and scaling agentic AI further, which reflects how few teams have actually sat down and classified their agent's action space by reversibility and blast radius before deploying it.

How attacks exploit the absence of a human checkpoint

Most successful attacks on agents look like the agent doing what it was asked to do, except that the instruction came from somewhere it shouldn't have. The attacker doesn't need to break in; the attacker needs the agent to misread an instruction's source and treat it as legitimate. And that only works because no human looks at the resulting tool call before it fires.

The pattern follows three steps in nearly every documented case. First, an injected instruction arrives through content the agent will process as part of its normal job, a document it's asked to summarize, an email in its inbox, a web page it retrieves. Second, the model interprets that instruction as a legitimate task rather than as untrusted input riding along with legitimate data. Third, the agent executes a tool call using its own service-level credentials, credentials the attacker never needed to steal because the agent already had them. Fiddler's practitioner guide walks through this escalation pattern and makes a point that matters for anyone hoping input filtering alone will solve the problem: the injected content is often semantically valid text. There's nothing malformed about it for a sanitizer to catch.

The clearest real-world case is CVE-2025-32711, known as EchoLeak, rated 9.3 on the CVSS scale. Every individual component in that chain, the retrieval system, the model, the proxy, the client, behaved exactly as designed. The exfiltration happened because no single component check, and no human, ever looked at the full path the data took from inbox to exfiltration.

A second case shows the same mechanism in a different tool. That gave the attacker a path to remote code execution, rated 7.8 on the CVSS scale. Indirect prompt injection of this kind has moved well past the research stage. Attackers are actively seeding ordinary web pages with hidden instructions built to hijack browsing agents, coding assistants, and enterprise copilots that happen to read that page.

The same exploitation logic appears at the plugin layer. Hidden Unicode characters embedded in AI Agent Skills survive ordinary code review, since nothing about them looks suspicious to a human scanning the file, yet language models process them as live instructions. Malicious plugins built around this technique succeeded in the SafeClawArena benchmark regardless of which underlying model powered the agent, which tells you the vulnerability is in the architecture, not in any one model's training.

Multi-agent systems compound the exploitation further. In this type of attack, an adversarial agent intercepts and rewrites messages passed between other agents. A single compromised agent early in the chain poisons every agent downstream, each of which treats the tainted message as trustworthy input simply because it came from another agent.

The same underlying failure showed up in December 2025, when attackers embedded indirect prompt injection payloads inside a webpage being reviewed by an AI-based ad moderation system. The payload caused the agent to approve fraudulent advertisements it was built specifically to catch and reject. The attack didn't need to break the moderation model's judgment. It needed the approval action to never be gated on a human decision, and it wasn't.

Every one of these cases ends the same way: a tool call fires that a human checkpoint, placed at the right tier of the action taxonomy, would have caught before it executed.

Tool-Level Permissions Are Not a Substitute for a Checkpoint

The natural response to all of this is to say, we already scope our agent's permissions tightly, it can only call the tools we've explicitly allowed. That's a necessary step, but it doesn't close the gap the attacks above are built to exploit. A legitimate tool, called with a parameter that originated in attacker-controlled content, is still an unauthorized action. It's just one wearing the disguise of an authorized one.

The missing layer is provenance: tracking where the values flowing into a tool's arguments actually came from, not just whether the tool itself is on the approved list. A financial transfer tool might be perfectly safe to expose to an agent in general, but if the destination account number in a given call traces back to text parsed out of an untrusted email, the call itself is not safe, regardless of how well-scoped the tool's permissions are. Agent-Sentry makes this exact case: tool-use safety needs parameter-level lineage, tracking the sources of values that flow into tool arguments, because tool-level permission checks alone can't see where any individual argument came from.

Google DeepMind's CaMeL architecture builds this provenance tracking directly into its design. On the AgentDojo benchmark, built specifically to test agent security under adversarial conditions, CaMeL solved a substantial majority of tasks while providing provable security guarantees: the interpreter enforces the guarantee, not the model's behavior.

That distinction is the entire argument for why permission scoping isn't the finish line. Today, most agent deployments authorize tool calls using model alignment, a probabilistic property established during training, combined with post-hoc evaluation that reviews what happened after the fact. Neither of those gives a deterministic, enforced check at the moment an individual tool call is about to run. A checkpoint built at the architecture level, one that inspects where data came from before letting it reach a sensitive call, closes a gap that permission lists were never designed to cover.

What a Reviewer Needs to Decide Confidently

A checkpoint that puts a human in front of an approve button without giving that person the information to evaluate the request isn't oversight, it's a rubber stamp with a timestamp attached. When a reviewer can't actually assess what they're approving, the approval becomes reflexive, and the organization has handed that person legal and operational liability without giving them any real power to catch a problem.

A reviewer needs three specific things to make that decision with any confidence. The first is a precise statement of what the agent intends to do, expressed as the actual tool call and its exact parameters, not a natural-language summary the agent itself generated describing its own plan. A summary can be wrong in exactly the way the underlying reasoning was wrong. The second is the provenance of those parameters: whether any value in the call traces back to an untrusted source, such as an external document, an email, or a web page the agent retrieved during its run. The third is a clear answer to what reversing the action would require, and whether it's reversible.

The interface where this approval happens is a target that must be secured. Approval decisions should happen inside a signed-in dashboard, not through a one-click link embedded in an email, because an unauthenticated approve-from-email link is a direct invitation for abuse. Anyone who can send that email can trigger the approval, defeating the purpose of putting a human in the loop.

Two failure modes compound each other here. Fiddler's practitioner guide names insufficient logging and monitoring as one of three production-relevant threat categories, and the reasoning is straightforward: without continuous monitoring, data exfiltration looks identical to ordinary agent activity, and nobody notices until the damage is already done.

That record gives the reviewer evidence instead of a vague assurance that "the team looked at it."

How the checkpoint is enforced at runtime

A policy document stating that high-value transfers require human approval is worth exactly as much as the mechanism that enforces it. The test of a checkpoint is whether the system can be made to bypass it, not whether it exists on paper.

Real enforcement happens at the execution layer, not inside a system prompt asking the model nicely to pause before acting. A system prompt instruction is a suggestion the model can be talked out of, and attackers have shown repeatedly that they can talk a model out of almost anything given the right framing. CaMeL's custom interpreter checks data provenance before every tool call executes, and that check runs independent of whatever the model has been persuaded to believe it should do. The model's internal state, however manipulated, is irrelevant to whether the policy actually fires, because the policy sits in the interpreter, not in the model's reasoning.

Verifying that a checkpoint holds requires treating the checkpoint itself as something an attacker is actively trying to defeat. Red teaming for agents needs to specifically test whether the pause can be routed around through prompt injection, through manipulated tool arguments, or through a hijacked orchestrator, not simply confirm that the pause fires correctly when nothing is attacking it. NIST SP 800-218A defines AI red teaming as a structured testing effort to find flaws and vulnerabilities in an AI system, and for agents that work can't be a single pass/fail test. An LLM-based control might resist an attack nine times and fail on the tenth, so red teams need to run many iterations and report success rates statistically rather than treat one clean run as proof the checkpoint is sound.

Running that testing against an isolated twin environment, rather than against production, lets a team push a checkpoint hard without putting live systems in front of real attack traffic while the testing is underway. And none of this testing matters if there's no record of what happened while the agent ran. Continuous observability across every step an agent takes is what catches a bypassed checkpoint before the consequences compound, because without telemetry at each stage of a multi-step chain, a checkpoint that got skipped is invisible until the damage it was supposed to prevent has already occurred. Fiddler's practitioner guide names continuous observability as the single most effective control for catching threats that static guardrails miss once an agent is running in production.

A team that has written a policy naming which actions require human approval has done the easy part of this work. Proving with reproducible evidence that the checkpoint holds under active attack is what makes the control meaningful.