Runtime Telemetry Collection and Overhead Tradeoffs for AI Agents

Collecting runtime telemetry for AI agents means deciding separately what to collect from tool calls, reasoning traces, memory access, and inter-agent messages. Each reveals different threats and imposes different costs. Engineers must choose coverage against the agent’s access scope, manage latency, storage, and evaluation costs independently, and verify that enforcement changes preserve approved workflows.

Why AI agents cannot be monitored like traditional software

Traditional software observability captures the execution of known code paths. Agents instead generate behavior at runtime, shaped by context, tool outputs, memory, and model decisions that become knowable only during a run. As a result, the question of what to observe cannot be settled at design time in the same way it can for a deterministic service.

A prompt is a suggestion. Novel inputs or adversarial tool outputs can lead an agent to generalize differently from what its designer intended. Persistent memory adds another complication because gradual drift can survive across runs and remain undetected for extended periods. Stateless service monitoring has no direct equivalent to that failure mode. Runtime verification must therefore combine four jobs, namely defining allowed behavior, enforcing it, observing execution, and supporting recovery. Traditional APM tools were not designed to perform those jobs together.

The four signal types that make up agent telemetry, and why they are not interchangeable

Tool calls are the signal closest to action. Their records describe what the agent invoked, the arguments it supplied, and what came back, making them directly relevant to enforcement decisions. An invocation record alone is incomplete because an otherwise permitted call can contain injected argument values. Per-agent baselines, structured telemetry, and provenance tracking make these records useful for detection. Raw invocation counts do not provide that context.

Reasoning traces record the intermediate chain-of-thought or planning steps that the model goes through before it acts. They can support debugging and reveal goal drift, but detailed capture and storage become expensive at scale. Resolution changes their practical use and cost. Run-level collection gives a coarse audit, step-level collection helps with debugging, and token/span-level collection gives the most detailed view.

Memory access covers reads and writes to persistent stores, knowledge bases, context windows, and vector stores. Its security value comes from threats including memory and context poisoning, RAG poisoning, and knowledge base manipulation. These threats can last beyond a single exchange. Detection at this layer must therefore go beyond single-turn evaluation, and what gets collected must reflect how the agent keeps and retrieves information.

Inter-agent messages show what agents send each other in multi-agent and chained-agent systems, where session smuggling, AgentCard poisoning, and cascading compromise can spread from one agent to another. One corrupted agent can affect an entire workflow. A message that looks benign in isolation may become suspicious only when connected to user input, RAG content, tool outputs, model output, and conversation history. This makes cross-chain visibility a different requirement from inspecting individual messages. Cross-chain visibility resembles an intrusion detection system, while individual-message inspection resembles a packet filter.

How collection granularity changes what attacks become detectable

Run-level traces help with coarse audit, while step-level traces show the order of steps needed for debugging. Tool-call and parameter-level traces support safety enforcement, and token/span-level traces provide finer detail at higher cost. These are not interchangeable ways of recording the same evidence. An attack that operates between steps or within arguments can disappear when collection stops at a coarser boundary.

AgentCanary consumes the full agent trajectory. It evaluates Outcome Safety, Security Awareness, and Task Utility as separate dimensions, allowing the result of a run to be assessed independently from the agent’s recognition of danger and completion of its task. That scope is important for long-horizon attacks. Single-turn inspection cannot show links that span the trajectory, especially when compromised skills or persistent state shape what the agent does later.

Argument provenance is the sharpest illustration of this gap. Tool-using agents consume untrusted webpages, emails, files, and API outputs while issuing privileged tool calls. If a defense assigns trust to an entire invocation, mixed-trust workflows force a brittle choice because different arguments may have different origins. Provenance-Aware Capability Contracts (PACT) assign semantic roles to arguments, track value provenance across replanning steps, and enforce role-specific trust contracts on each argument’s origin. PACT reports perfect utility and 100% security on diagnostic suites under oracle provenance, a condition that limits what the result establishes beyond those suites.

A record of the tool name cannot substitute for information about where a parameter came from or what role it serves. The practical implication is that a team choosing invocation-only collection should name the threat classes it accepts as undetectable.

The real collection costs: latency, storage, and LLM-judge overhead are separate problems

LLM-as-judge evaluation can serve offline scoring, but per-check API calls and added delay make it structurally unsuitable for inline guardrails. Specialized detection models running in parallel offer a different architecture for that constraint. Raypher’s 2026 benchmarks recorded a median of 11 ms of added latency per tool invocation using fine-tuned model ensembles specialized for different attack patterns. That result shows sub-300ms detection without sending every check through a full frontier model, though the overhead will not be the same everywhere.

Storage responds to different engineering decisions than inline latency. Full-fidelity prompt logging costs a lot at production scale, while deduplication in OpenTelemetry-native pipelines can cut storage costs well below the cost of keeping every prompt in full. OpenTelemetry-native instrumentation also makes data portable, so teams can redirect telemetry without re-instrumenting as the observability vendor market consolidates. Portability separates instrumentation from the destination that stores the data, while collection costs remain.

Tracing tools represent LLM calls, tool calls, and retrievals as spans containing prompts, responses, latency, token counts, and cost. Connected spans form the trace of a run, the minimum unit for multi-step attack analysis. Storage optimization therefore has a different objective from reducing the time spent on an inline check. The former concerns retained evidence, while the latter concerns the delay before execution proceeds.

Evaluation-model spending is a third cost, separate from storage and delay. Each LLM-judge check has an API charge even when it runs outside the critical path. OpenAgentSafety found that judges often disagree on security and credential-related tasks and miss finer unsafe actions. Higher evaluation spending does not guarantee better security discrimination. Runtime enforcement also consumes CPU, but architecture-dependent CPU overhead is not the dominant difficulty compared with latency and evaluation-model cost.

What the threat record shows about which signals actually catch attacks in production

Cursor RCE via MCP prompt injection, CVE-2025-54135, operated through a file write whose tool name was not obviously forbidden. An indirect prompt injection caused the agent to create a malicious MCP configuration without user approval, triggering auto-execution and remote code execution before most logs could capture it. The relevant evidence was the configuration being written. Invocation-level permission checks could permit writing a file without establishing authorization to create that configuration.

Microsoft Copilot single-click exfiltration, CVE-2026-24307, exposed a related gap at the input boundary. The Reprompt attack enabled data exfiltration through URL parameter injection with zero user-entered prompts. A prompt-level scan had no victim-typed trigger to inspect. Together, these cases locate consequential behavior below the resolution of coarse prompt or invocation records.

The OpenClaw marketplace supply chain event in 2026 extended the problem across agents. A one-click RCE vulnerability, CVE-2026-25253, and the simultaneous but separate ClawHavoc supply chain campaign led to marketplace-wide contamination with malicious skills across many exposed instances. Inter-agent message telemetry was the signal needed to trace that propagation in real time, rather than treating each affected agent as an isolated execution.

The OWASP GenAI Q1 2026 incident log records the February 2026 OpenClaw inbox deletion, in which the agent ignored stop commands and deleted user emails. The log maps it to LLM06 excessive agency and LLM05 improper output handling. Memory-access or action-gating signals would have provided the detection point. When execution proceeds at machine speed, an alert read after completion leaves a transaction history to investigate rather than an action to prevent, so collection must occur before the action completes.

How to structure a deliberate collection decision across the four signal types

The decision process should begin with the agent’s access scope and the signals that would reveal abuse of that access. A coding agent with shell rights presents a different collection problem from a productivity agent with OAuth connectors or a custom-built agent with broad internal access. Using the same granularity for all of them hides those differences and leaves them unresolved. Teams should choose what to collect for each signal class based on the threats it can reveal, and weigh latency, storage, and evaluation costs separately.

The four-layer attack surface includes model and prompt, application, tool and MCP, and data. Agents that work across all four layers need at least invocation-plus-argument coverage for tool calls and span-level tracing for inter-agent messages. When agents have access to credentials, external APIs, or persistent memory, argument provenance is a required control. An invocation record cannot distinguish a benign retrieval-then-act sequence from an injected one. Full-chain detection must also inspect user input, RAG content, tool outputs, model output, and conversation history together when the threat unfolds across steps.

Policies should be machine-readable, versioned, testable, and auditable, with an enforcement layer that intercepts tool calls, file access, network requests, and data movement before completion. Retrospective logging is not equivalent to that interception. For teams whose policies change quickly in regulated environments, Ethical Hyper-Velocity (EHV) tackles the separate problem of governance latency. EHV relocates the Policy Enforcement Point into the inference pipeline through a Governance-Aware JIT Compiler, aiming under its stated assumptions to reduce the lag between policy updates and enforcement from days toward near-real-time.

The process can legitimately end with acknowledged blind spots. Cost-driven omissions of reasoning traces should be recorded together with the visibility they remove. This is a decision framework in which maximizing every signal is optional. It makes omissions available for mitigation and review instead of allowing them to persist as unnamed defaults. The resulting posture can then be reconsidered as the agent’s access scope changes.

Verifying that telemetry changes do not break the workflows they are meant to protect

Adding telemetry and tightening enforcement can break legitimate workflows. Runtime defenses can trade utility for safety, and improvements in aggregate security scores are uneven across models and configurations. Some configurations are essentially neutral or worse. An enforcement change therefore needs validation on the target model and workload. AgentCanary scores each dimension separately, so a gain in Outcome Safety can be seen alongside a drop in Task Utility.

Approved workflows should be re-run after every enforcement change to confirm that they still pass. The Verified Change Record model sets the release standard by requiring evidence that the change preserves approved behavior before a release approver signs off. Changes that cannot be reproduced and verified should not reach production. Human approval before production changes merge keeps the reviewer in control of the enforcement layer, making the consequences of each collection and policy decision accountable.